otel_exponential_histogram_subtract(curr: object, prev: object) -> object

Converts two adjacent CUMULATIVE OpenTelemetry exponential histogram datapoints of one series into the DELTA datapoint covering the interval between them, returning an object histogram. It is a plain scalar function rather than a window function: you obtain the predecessor yourself with lag inside window, which mirrors the shape of the native Snowflake exponential-histogram kernels.

The function cannot verify that subtracting is the right thing to do, because temporality is not carried inside the histogram object at all — the OpenTelemetry proto puts aggregation_temporality on the ExponentialHistogram metric rather than on its datapoints, and in the direct-write metric datastream it arrives in the sibling fields object. Pairing only genuinely cumulative, genuinely adjacent rows of one series is therefore the caller's responsibility.

Required call shape

filter int64(get_field(fields, "aggregation_temporality")) = 2
| make_col run: int64(get_field(complex_value, "start_time_unix_nano"))
| make_col prev: window(lag(complex_value, 1), group_by(metric, run), order_by(timestamp))
| make_col delta: otel_exponential_histogram_subtract(complex_value, prev)

Every part of that shape is load-bearing:

• Filter to cumulative rows. A delta row that gets subtracted again yields near-zero counts, which read as "no traffic" rather than as an error. Both temporalities routinely coexist under one metric name, because different services emit the same metric under different configurations.
• Group by the full series identity. group_by must list every column that identifies the series — the metric name and all of its tag columns. Omitting one mixes unrelated series and silently subtracts one from another.
• Group by the accumulation run. OpenTelemetry issues a new start_time_unix_nano when a process restarts, so putting it in the partition key makes a counter reset open a new partition, where lag is null and the delta is correctly the first observation itself. The function additionally compares the two datapoints' own start_time_unix_nano and treats a mismatch as a run boundary, so a reset stays correct even if you leave run out of the grouping — but keep it in anyway, because without it two overlapping runs of one series interleave by time and then every row looks like a boundary.
• Order ascending by time. The predecessor must be strictly earlier, because the delta describes the half-open interval (prev observation, curr observation]. Descending order is refused rather than producing a delta, and so is a duplicate row.

In a transform, put a bounded frame(back: ...) on the window so incremental execution widens its read window far enough to see the predecessor across slice boundaries. An ad-hoc query needs no frame: the query window is the frame. In either case the first row in scope for a series has no predecessor.

What the result contains

scale is the coarser of the two inputs' scales, with the finer histogram's buckets merged down to it. sum is a float64, emitted when both inputs carry one and omitted otherwise, since a difference is not derivable from a single side. start_time_unix_nano and time_unix_nano describe this delta's own interval — the predecessor's observation time and the current one — which is not the same thing as the input's start_time_unix_nano, which identifies the whole accumulation run. min and max are omitted, because an interval's minimum and maximum cannot be recovered by subtracting two cumulative snapshots.

When the result is null

The function is strict: it refuses rather than repairing, so that a broken input does not become a plausible-looking delta. The result is null when

• the two datapoints have different zero_threshold values, since their zero regions then cover different ranges. The specification describes reconciling a difference by taking the larger threshold and folding the lower buckets into the wider zero bucket, and states that rule for merging rather than subtracting — it does not forbid doing the same here, and several shipped implementations do reconcile a widening. This function simply does not, because reconciling means bucket-boundary arithmetic it has no other reason to perform; refusing keeps the pair's disagreement visible instead of resolving it silently.
• the predecessor's time_unix_nano is not strictly earlier than the current one's. Equal timestamps would make the interval (T, T] empty and an inverted pair would make it undefined; the specification treats both as the same condition and says receivers should eliminate such points, and it names the resulting hole a gap, which is legitimate.
• count or zero_count went backwards,
• any bucket went backwards, or the predecessor has a populated bucket that the current datapoint's range does not cover — even when count rose. A cumulative histogram's buckets are individually monotonic, so a decrease means the pair is not a valid cumulative sequence.

The result is also null when an input is malformed or outside the contract, rather than the function throwing and failing your whole query. That covers

• text that is not one well-formed JSON object — truncated, carrying trailing content, containing a duplicate key, nested more than 64 levels deep, or carrying a number with a leading +, which JSON does not permit even though most parsers accept it (an exponent sign, as in 1e+2, is a different position and is fine),
• a scale outside [-10, 20], the range the exponential-histogram bucket mappings support; outside it a bucket index has no defined meaning,
• a bucket_counts array longer than 70000 entries, a bucket offset that does not fit sint32, or two offsets whose combined index range would exceed 70000 buckets,
• a bucket count that is negative, fractional, or too large for int64 — bucket counts are unsigned 64-bit in the specification, so a negative one would reappear as a very large positive number in a consumer that reads them unsigned,
• two or more of the finer histogram's bucket counts that overflow int64 when they are merged onto one bucket of the coarser scale. Each count can be individually legal and their sum still not be, and a wrapped sum would emit a negative count,
• a max_size outside [0, 70000],
• a numeric field the function actually reads being something it cannot use: a non-finite value such as 1e999, a non-numeric string, a boolean, or an integer too large for int64.

That last point is limited to fields the function reads. An oversized integer in a field it does not consume passes through untouched, so a 20-digit id or an unsigned 64-bit counter carried alongside the histogram survives the no-predecessor path intact.

Because a scalar function has a single result channel, these refusals are indistinguishable from the ones above. A call site that coalesces the subtraction with the current datapoint will therefore substitute the current datapoint for genuinely corrupt input, and an input that would once have failed the query loudly now yields a null instead. That trade is deliberate: the alternative hands anyone able to influence the ingested data a way to fail an entire query with a single row.

The result is null, too, when a datapoint is not one this function can subtract at all:

• the two datapoints state different aggregation_temporality values. Temporality is not carried in the histogram object by the specification, but Observe's direct-write path does put it there, and when the two disagree subtracting them is meaningless rather than merely unverified,
• either datapoint sets FLAG_NO_RECORDED_VALUE, which the specification defines as "explicitly missing data" whose other properties "SHOULD be ignored" — counts that must be ignored cannot take part in a subtraction from either side,
• flags falls outside the unsigned 32-bit range,
• count or zero_count is negative, or is too large for a signed 64-bit integer in a field this function reads,
• time_unix_nano is zero, which the specification says consumers should reject, or one datapoint carries it and the other does not. Both omitting it is different and is accepted: the producer never claimed an interval, and the result then carries none either,
• the difference of the two sum values is not finite, or the interval's count works out to zero while its sum does not — an interval with no observations cannot have a sum.

Note what is deliberately not checked. The specification requires count to equal the sum of the bucket counts plus zero_count, and this function does not enforce it: if an input breaks that invariant the result inherits it. The line drawn is to refuse what this function would itself manufacture — a negative count out of the subtraction, a non-finite sum — and to pass through what the producer asserted about its own data.

If you would rather recover an unsignalled reset than lose the row, express that at the call site — coalesce the subtraction with the current datapoint — instead of having the function guess. This is the same division of labour as the canonical Go implementation, ExponentialBuckets.Diff in the collector-contrib cumulativetodelta processor, which reports a reset to its caller rather than deciding the policy itself.

A word of caution about that pattern on one specific input. A datapoint whose start_time_unix_nano equals its time_unix_nano is the specification's zero-duration reset marker, and this function returns null for it because there is no interval to attribute the counts to. The specification warns that losing such a point causes a consumer to overcount the next one — but coalescing it to the current datapoint is the opposite error and a larger one, because it attributes an entire accumulated counter to a single instant. If you use the coalesce form, filter that shape out first.

When there is no predecessor

With no predecessor at all, or a predecessor from a different accumulation run, the current datapoint is returned with its aggregation_temporality restamped to delta and everything else it carries — including any keys this function does not interpret — left in place. That is a different shape from the subtract path, which builds its result from a fixed set of fields and so drops anything it did not compute. It is already a valid delta over its own interval, and it keeps its min and max because those describe exactly that interval. start_time_unix_nano is then the run's own start, which is also this delta's interval start, so no special-casing is needed. A datapoint that is omitted entirely from a run boundary would lose real data, which is why it passes through rather than becoming null. A missing start_time_unix_nano is not treated as a mismatch: the specification allows omitting it when the start time is genuinely unknown.

The one exception is a datapoint whose start_time_unix_nano equals its time_unix_nano, which is null. The specification gives that form a specific meaning — a reset at an unknown start time, a point of zero duration indicating "that nothing is known about previously-reported points and that data may have been lost", whose rate contribution "is considered zero". There is no interval to attribute the accumulated counts to, so reporting them would place a whole counter at an instant.

Domain

This is a scalar function (calculates a single output value for a single input row).

Categories

• Semistructured

Examples

filter int64(get_field(fields, "aggregation_temporality")) = 2
| make_col run: int64(get_field(complex_value, "start_time_unix_nano"))
| make_col prev: window(lag(complex_value, 1), group_by(metric, run), order_by(timestamp))
| make_col delta: otel_exponential_histogram_subtract(complex_value, prev)

Turns a cumulative exponential histogram column into per-interval deltas. The filter keeps only cumulative rows, run puts the accumulation run in the partition key so a process restart starts a fresh partition, and prev carries the previous datapoint of the same series. Extend group_by with every tag column that identifies the series.