List of OPAL verbs

add_key (Add Candidate Key)

Type of operation: Metadata

Aliases: addkey(deprecated)

Description

Add a candidate key to the output. The candidate key describes a combination of columns that together identify a resource instance, and can be the target of a foreign key.

Is this verb streamable? Always.

Usage

add_key keyfield, ...

Examples

add_key cluster_uid, resource_uid

Adds a candidate key that says that cluster_uid plus resource_uid together uniquely identify the resource instance.

Arguments

Argument

Type

Required

Multiple

keyfield

fieldref

True

True

aggregate (Aggregate)

Type of operation: Aggregate, Metrics

Aliases: reaggregate(deprecated)

Description

Aggregates metrics across tag dimensions.

Is this verb streamable? Always.

Usage

aggregate [ groupby, ... ], groupOrAggregateFunction, ...

Examples

aggregate tx_bytes:sum(tx_bytes), group_by(podName, namespace, clusterUid)

Group the tx_bytes metric by ‘podName’, ‘namespace’ and ‘clusterUid’ on each time bin, calculating the sum of the values in each bin.

Arguments

Argument

Type

Required

Multiple

groupby

fieldref

False

True

groupOrAggregateFunction

expression

True

True

always (Filter where always)

Type of operation: Filter

Description

Select data for resources that matched the predicate at all times

Is this verb streamable? Never.

Usage

always predicate

Examples

always string(status_code) ~ /^2.*/

Select only resources where the ‘status_code’ column, converted to string, always started with ‘2’, at all points of the time window.

Arguments

Argument

Type

Required

Multiple

predicate

bool

True

False

colenum (Column Is Enum)

Type of operation: Metadata

Description

Mark columns as enumerations, or not, by name. Arguments are colname:bool where the bool value must be known at compile time. Columns that are enumerations are treated differently in GUI and visualization, such as using top-k summaries instead of histograms or sparklines.

Is this verb streamable? Always.

Usage

colenum col, ...

Examples

colenum cluster_uid:false, cluster_index:false, cluster_name: true

Marks the columns cluster_uid and cluster_index as scalar values, and marks the column cluster_name as an enumeration value.

Arguments

Argument

Type

Required

Multiple

col

expression

True

True

colimmutable (Column Is Immutable)

Type of operation: Metadata

Description

Mark resource columns as time immutable a.k.a. time-invariant, or not. Arguments are colname:bool where the bool value must be known at compile time. A time immutable column is a column that does not change for a given resource instance (as identified by the resource primary key). All key columns are implicitly immutable. Columns that are immutable can be stored and processed more efficiently. Beware: manually marking mutable columns as immutable can lead to wrong query results.

Is this verb streamable? Always.

Usage

colimmutable col, ...

Examples

colimmutable hostname:true, IP:false

Marks the column hostname as immutable, and the column IP as mutable.

Arguments

Argument

Type

Required

Multiple

col

expression

True

True

colshow (Show/Hide Columns)

Type of operation: Metadata

Description

Show or hide columns by name. Arguments are colname:bool where the bool value must be a literal true or false.

Is this verb streamable? Always.

Usage

colshow col, ...

Examples

colshow cluster_uid:false, cluster_index:false, cluster_name: true

Hides the columns cluster_uid and cluster_index, and shows the column cluster_name.

Arguments

Argument

Type

Required

Multiple

col

expression

True

True

dedup (Dedup)

Type of operation: Aggregate

Aliases: distinct

Description

dedup collapses all rows in an event dataset with identical values in specified columns and with identical timestamp to a single row. For the remaining columns, an arbitrary value from the collapsed rows is picked while preferring non-null values.

When no column names are given, dedup collapses rows with identical values in all the columns to a single row.

Is this verb streamable? Always.

Usage

dedup [ columnname, ... ]

Examples

dedup vf, message

Collapse the rows with identical values in vf and message columns and with identical timestamps to a single row.

dedup

Remove duplicate rows in the input dataset

Arguments

Argument

Type

Required

Multiple

columnname

expression

False

True

drop_col (Drop Column)

Type of operation: Projection

Aliases: coldrop(deprecated)

Description

Exclude one or more columns from the input dataset to the output dataset. Primary key and time columns may not be dropped.

Is this verb streamable? Always.

Usage

drop_col columnname, ...

Examples

drop_col debug_info, status_code

Exclude the columns ‘debug_info’ and ‘status_code’ from the data passed downstream.

Arguments

Argument

Type

Required

Multiple

columnname

fieldref

True

True

droptime (Drop Time)

Type of operation: Metadata

Description

Clear the ‘valid from’ (and the ‘valid to’), turning the output rows from the current query window into non-temporal rows. The output of this verb is not streamable.

Is this verb streamable? Never.

Usage

droptime

Examples

droptime

Drops the ‘valid from’ and ‘valid to’ designations of any such columns in the input dataset.

ever (Filter where ever)

Type of operation: Filter

Description

Select data for resources that at some point matched the predicate

Is this verb streamable? Never.

Usage

ever predicate

Examples

ever string(status_code) ~ /^5.*/

Select only resources where the ‘status_code’ column, converted to string, starts with ‘5’, at any point of the time window.

Arguments

Argument

Type

Required

Multiple

predicate

bool

True

False

exists (Exists)

Type of operation: Join

Description

Return the rows from the default dataset that have a match anywhere in the query time window. (Untemporal semijoin)

Is this verb streamable? Never.

Usage

exists predicate, ...

Examples

exists sensor_id=@right.sensor_id

Semijoin the default dataset with the ‘right’ dataset, returning rows from ‘default’ where there exists a key match at any point in time within the query window.

Arguments

Argument

Type

Required

Multiple

predicate

bool

True

True

extract_regex (Extract using RegEx)

Type of operation: Projection

Aliases: colregex(deprecated)

Description

Add one or more columns by matching capture names in a regular expression against a given source expression. Regex extractions create string columns. Named capture groups are an extension to POSIX extended regular expressions. If the column already exists, and the regular expression finds nothing, the previous value is preserved. See also: ‘make_col’.

Is this verb streamable? Always.

Usage

extract_regex path, regex

Examples

extract_regex message, /status=(?P<statuscode>\d+)/

Create the column ‘statuscode’ by matching for status=numbers in the field ‘message’.

extract_regex inputcol, /(?P<sensor>[^|]*)\|count:(?P<counts>[^|]*)\|env:(?P<env>[^|]*)/

Given an input column value like: “studio-aqi|count:654 201 28 0 0 0|env:3 4 4a”, generate three output columns: “sensor” with the value “studio-aqi”, “counts” with the value “654 201 0 0 0”, and “env” with the value “3 4 4a”.

Arguments

Argument

Type

Required

Multiple

path

expression

True

False

regex

regex

True

False

filter (Filter)

Type of operation: Filter

Description

Exclude rows from the input dataset that do not match the given predicate expression.

Is this verb streamable? Always.

Usage

filter predicate

Examples

filter string(status_code) ~ /^5.*/

Keep only rows where the ‘status_code’ column, converted to string, starts with ‘5’.

Arguments

Argument

Type

Required

Multiple

predicate

bool

True

False

flatten (Flatten)

Type of operation: Misc, Join

Description

Given an input of object or array type, recursively flatten all child elements into ‘_c_NAME_path’ and ‘_c_NAME_value’ columns, generating null values for intermediate object/array values. The default is to not suggest column types (‘suggesttypes’ = ‘false’.) See also flatten_single.

Is this verb streamable? Always.

Usage

flatten pathexpression, [ suggesttypes ]

Examples

flatten foo

Produce new columns that contain every possible path and its corresponding value, with null values for intermediate key paths so the full tree is returned. Column ‘foo’ will be removed.

flatten foo, true

Produce new columns that contain every possible path and its corresponding value. It will also attempt to determine the value’s type, creating a third column, ‘_c_foo_type’, containing the name of the identified type. Column ‘foo’ will be removed.

Arguments

Argument

Type

Required

Multiple

pathexpression

fieldref

True

False

suggesttypes

bool

False

False

flatten_all (Flatten All)

Type of operation: Misc, Join

Aliases: flattenall(deprecated)

Description

Given an input of object or array type, recursively flatten all child elements into ‘_c_NAME_path’ and ‘_c_NAME_value’ columns, including intermediate object/array values. (This is expensive – consider flatten_leaves instead.) The default is to not suggest column types (‘suggesttypes’ = ‘false’.)

Is this verb streamable? Always.

Usage

flatten_all pathexpression, [ suggesttypes ]

Examples

flatten_all foo

Produce new columns that contain every possible path and its corresponding value. Column ‘foo’ will be removed.

flatten_all foo, true

Produce new columns that contain every possible path and its corresponding value. It will also attempt to determine the value’s type, creating a third column, ‘_c_foo_type’, containing the name of the identified type. Column ‘foo’ will be removed.

Arguments

Argument

Type

Required

Multiple

pathexpression

fieldref

True

False

suggesttypes

bool

False

False

flatten_leaves (Flatten Leaves)

Type of operation: Misc, Join

Aliases: flattenleaves(deprecated)

Description

Given an input of object or array type, recursively flatten all child elements into ‘_c_NAME_path’ and ‘_c_NAME_value’ columns, returning only leaf values. The default is to not suggest column types (‘suggesttypes’ = ‘false’.) See also flatten_single.

Is this verb streamable? Always.

Usage

flatten_leaves pathexpression, [ suggesttypes ]

Examples

flatten_leaves foo

Produce new columns that contain every leaf path and its corresponding value. Column ‘foo’ will be removed.

flatten_leaves foo, true

Produce new columns that contain every leaf path and its corresponding value. It will also attempt to determine the value’s type, creating a third column, ‘_c_foo_type’, containing the name of the identified type. Column ‘foo’ will be removed.

Arguments

Argument

Type

Required

Multiple

pathexpression

fieldref

True

False

suggesttypes

bool

False

False

flatten_single (Flatten Single)

Type of operation: Misc, Join

Aliases: flattensingle(deprecated)

Description

Given an input of object or array type, flatten the first level of child elements into ‘_c_NAME_path’ and ‘_c_NAME_value’ columns. The default is to not suggest column types (‘suggesttypes’ = ‘false’.)

Is this verb streamable? Always.

Usage

flatten_single pathexpression, [ suggesttypes ]

Examples

flatten_single foo

Produce new columns that contain the path and values of the top level of keys in foo. Column ‘foo’ will be removed.

flatten_single foo, true

Produce new columns that contain the path and values of the top level of keys in foo. It will also attempt to determine the value’s type, creating a third column, ‘_c_foo_type’, containing the name of the identified type. Column ‘foo’ will be removed.

Arguments

Argument

Type

Required

Multiple

pathexpression

fieldref

True

False

suggesttypes

bool

False

False

follow (Follow)

Type of operation: Join

Description

Return the rows from the additional joined dataset that have a match anywhere in the query time window. (Untemporal semijoin)

Is this verb streamable? Never.

Usage

follow predicate, ...

Examples

follow sensor_id=@right.sensor_id

Semijoin the default dataset with the ‘right’ dataset, returning rows from ‘right’ where there exists a key match at any point in time within the query window.

Arguments

Argument

Type

Required

Multiple

predicate

bool

True

True

fulljoin (Outer Join)

Type of operation: Join

Description

Temporal full join, adding new columns in the output dataset.

Is this verb streamable? Always.

Usage

fulljoin predicate, ..., [ columnbinding, ... ]

Examples

fulljoin host_uid=@host.uid, hostname:@host.name

Temporal full join with dataset ‘host’, and extract the ‘name’ column from that ‘host’ table, calling the new column ‘hostname’ in the output.

Arguments

Argument

Type

Required

Multiple

predicate

bool

True

True

columnbinding

expression

False

True

interface (Interface)

Type of operation: Metadata

Description

Map fields of this dataset to a pre-defined interface.

Is this verb streamable? Always.

Usage

interface interfaceName, fieldBinding, ...

Examples

interface "notification", kind:myKindStr, description:logText, importance:sevInt

Make this dataset implement the ‘notification’ interface, binding the existing column ‘myKindStr’ to the ‘kind’ interfaceName, the existing column ‘logText’ to the ‘description’ interfaceName, and the existing column ‘sevInt’ to the ‘importance’ interfaceName.

interface "metric", metric:metricNameColumn, value:metricValueColumn

Make this dataset implement the ‘metric’ interface. Bind the existing column containing metric names (‘metricNameColumn’) to the ‘metric’ interfaceName, and the existing column containing ‘float64’ metric values (‘metricValueColumn’) to the ‘value’ interfaceName.

Arguments

Argument

Type

Required

Multiple

interfaceName

string

True

False

fieldBinding

expression

True

True

join (Inner Join)

Type of operation: Join

Description

Temporal inner join, adding new columns in the output dataset.

Is this verb streamable? Always.

Usage

join predicate, ..., [ columnbinding, ... ]

Examples

join host_uid=@host.uid, hostname:@host.name

Temporal inner join with dataset ‘host’, and extract the ‘name’ column from that ‘host’ table, calling the new column ‘hostname’ in the output.

Arguments

Argument

Type

Required

Multiple

predicate

bool

True

True

columnbinding

expression

False

True

leftjoin (Left Join)

Type of operation: Join

Description

Temporal left join, adding new columns in the output dataset.

Is this verb streamable? Always.

Usage

leftjoin predicate, ..., [ columnbinding, ... ]

Examples

leftjoin host_uid=@host.uid, hostname:@host.name

Temporal left join with dataset ‘host’, and extract the ‘name’ column from that ‘host’ table, calling the new column ‘hostname’ in the output.

Arguments

Argument

Type

Required

Multiple

predicate

bool

True

True

columnbinding

expression

False

True

lookaround_join (Lookaround Join)

Type of operation: Join

Description

Lookaround join is a type of inner join. Where this differs from “join” is that a row from the input dataset is matched only with rows from the other dataset that are within the specified timeframe around the input row. This join can only be used to join two event datasets.

The first argument “frame” specifies the timeframe. The “predicate” and “columnbinding” arguments work identically to that of a regular “join.”

Is this verb streamable? Always.

Usage

lookaround_join frame, predicate, ..., [ columnbinding, ... ]

Examples

lookaround_join frame_exact(back: 2s, ahead: 2s), ip=@host.ip, location:@hostdata.location

For every row in the input dataset, fetch the rows from “host” that are exactly within 2s of the input row and matching the IP address. Extract the location field from the “hostdata” dataset. Consider using frame() instead of frame_exact() where possible, for faster execution.

Arguments

Argument

Type

Required

Multiple

frame

frame

True

False

predicate

bool

True

True

columnbinding

expression

False

True

lookup (Look-up)

Type of operation: Join

Description

Find matching rows in a resource, making new columns in the output dataset.

Is this verb streamable? Always.

Usage

lookup foreignkeyequalitypredicate, ..., columnbinding, ...

Examples

lookup host_uid=@host.uid, hostname:@host.name

Look up the ‘host_uid’ value as the ‘uid’ column in the input table named ‘host’, and extract the ‘name’ column from that ‘host’ table, calling the new column ‘hostname’ in the output.

Arguments

Argument

Type

Required

Multiple

foreignkeyequalitypredicate

bool

True

True

columnbinding

expression

True

True

make_col (Make Columns)

Type of operation: Projection, Aggregate

Aliases: colmake(deprecated)

Description

Add one or more new columns from the input dataset to the output dataset. See also: ‘extract_regex’.

Is this verb streamable? Always.

Usage

make_col columnbinding, ...

Examples

make_col message:string(data.payload.message), ok:string(data.payload.ok)

Create the columns ‘message’ and ‘ok’ by coercing various data column object fields to strings.

Arguments

Argument

Type

Required

Multiple

columnbinding

expression

True

True

make_event (Turn Resource into Events)

Type of operation: Metadata

Aliases: changelog(deprecated)

Description

Given a resource, make_event will demote it to a series of update events that would create that resource.

Is this verb streamable? Always.

Usage

make_event

Examples

make_event

Given an input resource dataset, will un-mark the “valid to” timestamp field, and make the output dataset be an event dataset.

make_resource (Make Resource)

Type of operation: Metadata

Aliases: makeresource(deprecated)

Description

Convert an event table to a resource with the specified primary key. Collapse adjacent events that contain the same primary key value and use the first time such event is observed as ‘valid_from’. The ‘valid_to’ of the row is determined by the minimal of the following three things: 1) the timestamp of the next distinct event; 2) ‘valid_from’ + the expression optionally specified in validfor(); 3) ‘valid_from’ + the ‘expiry’ option value. If an ‘expiry’ value is not provided, the default is 24 hours (86,400 seconds.)

Is this verb streamable? Always.

Usage

make_resource [ options ], columnbinding, ..., primarykey, [ validfor ]

Examples

make_resource options(expiry:duration_hr(1)), col1:col1, primary_key(pk1, pk2)

Produces a resource with the primary key (pk1, pk2) and column col1 with an expiry period of 1 hour.

make_resource col1:col1, primary_key(pk1, pk2), valid_for(duration(col2))

Produces a resource with the primary key (pk1, pk2) and column col1 with an expiry period determined by column col2.

Arguments

Argument

Type

Required

Multiple

options

options

False

False

columnbinding

expression

True

True

primarykey

primarykey

True

False

validfor

validfor

False

False

make_session (Make Session)

Type of operation: Metadata, Aggregate

Aliases: makesession(deprecated)

Description

Group events or intervals that are close to each other into sessions, and calculate aggregations over each session. Two events or intervals would be assigned to the same session if the time period between them is below the session expiry time (default to one day). Two overlapped intervals will always be mapped to the same session. The output’s ‘valid_from’ and ‘valid_to’ fields would mark the start and end time of the session. Note that applying make_session over existing sessions may result in unexpected results. For instance, calculating average again over an already averaged column will not result in the correct average overall.

Is this verb streamable? Always.

Usage

make_session [ options ], [ groupby, ... ], groupOrAggregateFunction, ...

Examples

make_session cnt:count(1), group_by(server_name)

Group input events or intervals into per server name sessions, and count the number of events or intervals in each session. Return a dataset with 4 columns ‘valid_from’, ‘valid_to’, ‘server_name’, and ‘cnt’.

make_session options(expiry:10m), cnt:count(1), group_by(server_name)

Similar to the above example, but expire each session after 10 minute’s inactivity (no new event falls into the session).

Arguments

Argument

Type

Required

Multiple

options

options

False

False

groupby

fieldref

False

True

groupOrAggregateFunction

expression

True

True

merge_event (Merge Event)

Type of operation: Join

Aliases: mergeevent(deprecated)

Description

Merge an event (or point) table with the current resource.

Is this verb streamable? Always.

Usage

merge_event [ options ], pkequalitypredicate, ..., columnbinding, ...

Examples

merge_event options(expiry:duration_hr(1)), host_uid=@cpuload.host, cpu:@cpuload.load

Look up the ‘host_uid’ value as the ‘host’ column in the event table named ‘cpuload’, and extract the ‘load’ column from that ‘cpuload’ table, calling the new column ‘cpu’ in the output resource.

Arguments

Argument

Type

Required

Multiple

options

options

False

False

pkequalitypredicate

bool

True

True

columnbinding

expression

True

True

never (Filter where never)

Type of operation: Filter

Description

Select data for resources that at no point matched the predicate

Is this verb streamable? Never.

Usage

never predicate

Examples

never string(status_code) ~ /^5.*/

Select only resources where the ‘status_code’ column, converted to string, never started with ‘5’, at any point of the time window.

Arguments

Argument

Type

Required

Multiple

predicate

bool

True

False

pick_col (Pick Columns)

Type of operation: Projection

Aliases: colpick(deprecated)

Description

Exclude all columns except the specified columns from the input dataset to the output dataset. Primary key and time columns must be picked.

Is this verb streamable? Always.

Usage

pick_col columnbinding, ...

Examples

pick_col event_time:input_time, uid:data.request.sourceHost, status_code:int64(data.request.httpStatus), message:message, ok:int64(data.request.httpStatus) < 400

Re-shape the data to contain exactly the five columns ‘event_time’, ‘uid’, ‘status_code’, ‘message’, and ‘ok’.

Arguments

Argument

Type

Required

Multiple

columnbinding

expression

True

True

rename_col (Rename Columns)

Type of operation: Projection

Aliases: colrename(deprecated)

Description

Include all columns while renaming the specified columns from the input dataset to the output dataset. Argument structure is newname:oldname. Includes necessary primary key fields and time fields needed for downstream analysis.

Is this verb streamable? Always.

Usage

rename_col columnbinding, ...

Examples

rename_col event_time:input_time, uid:sourceHost, status_code:httpStatus

Renames the input columns to ‘event_time’, ‘uid’, ‘status_code’ while still retaining the rest of the columns in the table

Arguments

Argument

Type

Required

Multiple

columnbinding

expression

True

True

rollup (rollup)

Type of operation: Aggregate, Metrics

Aliases: align

Description

Rollup raw metrics into aligned metrics

Is this verb streamable? Sometimes.

Usage

rollup [ options ], metric, ...

Examples

rollup options(resolution:300s), requests:metric("requests_total")

Generates a column named “requests” holding “requests_total” metric and align them with 300s time bins.

rollup options(buckets:2000), failed_requests:metric("requests_total")

Generates a column named “failed_requests” holding “requests_total” metric and align them with 2000 uniform time bins in the query window.

rollup options(resolution:300s), failed_requests:metric("requests_total", filter:status_code >= 400 and status_code <= 599)

Generates a column named “failed_requests” holding “requests_total” metric where status_code is in [400, 599], and align them with 300s time bins.

rollup options(resolution:300s), failed_requests:metric("requests_total", type:cumulativeCounter, rollup:avg, aggregate:sum)

Generates a column named “failed_requests” holding “requests_total” metric and align them with 300s time bins with the provided method.

Arguments

Argument

Type

Required

Multiple

options

options

False

False

metric

expression

True

True

set_label (Set Label)

Type of operation: Metadata

Aliases: setlabel(deprecated)

Description

Declare the ‘label’ of the output to be the designated column. The column must contain strings.

Is this verb streamable? Always.

Usage

set_label name

Examples

set_label device_name

Sets ‘label’ of the output dataset as the ‘device_name’ column.

Arguments

Argument

Type

Required

Multiple

name

fieldref

True

False

set_metric (Set Metric)

Type of operation: Metrics, Metadata

Aliases: addmetric(deprecated)

Description

Register a metric, with its metadata defined in an options object. name should be an expected value in the metric name field.

Values for options metadata:

label, unit, and description are strings

type is one of: ‘cumulativeCounter’, ‘delta’, ‘gauge’

interval a duration representing the reporting interval of the metric, such as 1m, 15s

rollup is one of: ‘count’, ‘max’, ‘min’, ‘rate’, ‘sum’, ‘avg’

aggregate is one of: ‘any’, ‘any_not_null’, ‘avg’, ‘count’, ‘countdistinct’, ‘counddistinctexact’, ‘max’, ‘median’, ‘medianexact’, ‘min’, ‘stddev’, ‘sum’

unit is optional. See the Metrics documentation for more information.

Is this verb streamable? Always.

Usage

set_metric options, name

Examples

set_metric options(label:"Ingress Bytes", type:"cumulativeCounter", unit:"bytes", description:"Ingress reported from somewhere", rollup:"rate", aggregate:"sum", interval: 15s), "ingress_bytes"

Register the metric ‘ingress_bytes’ within this dataset. The dataset must already implement the “metric” interface.

set_metric options(label:"Temperature", type:"gauge", unit:"C", description:"Storage room B temperature", rollup:"avg", aggregate:"avg", interval: 5m), "temp"

Register the metric ‘temp’ within this dataset. The dataset must already implement the “metric” interface, for example with ‘interface “metric”, metric:sensor_type, value:value’

Arguments

Argument

Type

Required

Multiple

options

options

True

False

name

string

True

False

set_primary_key (Set Primary Key)

Type of operation: Metadata

Aliases: set_pk, setpk(deprecated)

Description

Declare the primary key of the output as consisting of one or more named columns. All rows with the same value in this column (or these columns) will be considered part of the same resource. This is a low-level function that will generate confusing results if not used as part of a larger context. It is recommended to instead use ‘make_resource’ or ‘merge_event’ or ‘timechart’ to go from event to resource, and ‘make_event’ to go from resource to event shape.

Is this verb streamable? Always.

Usage

set_primary_key columnname, ...

Examples

set_primary_key device_uid

Sets the primary key designation of the output dataset as the ‘device_uid’ field.

Arguments

Argument

Type

Required

Multiple

columnname

fieldref

True

True

set_valid_from (Set ‘Valid From’)

Type of operation: Metadata

Aliases: setvf(deprecated)

Description

Declare the ‘valid from’ of the output to be the named column. Beware changing time to a field that is too far off from the current timestamp field, because it may end up falling outside of the processing time window.

Is this verb streamable? Always.

Usage

set_valid_from [ options ], columnname

Examples

set_valid_from ts_col

Sets the ‘valid from’ designation of the output dataset as the ‘ts_col’ field.

set_valid_from options(max_time_diff:duration_hr(1)), ts_col

Sets the ‘valid from’ designation of the output dataset as the ‘ts_col’ field, and the maximum time difference between the original ‘valid from’ field and ‘ts_col’ is less than one hour.

Arguments

Argument

Type

Required

Multiple

options

options

False

False

columnname

fieldref

True

False

set_valid_to (Set ‘Valid To’)

Type of operation: Metadata

Aliases: setvt(deprecated)

Description

Declare the ‘valid to’ of the output to be the named column. Omitting the column name will clear the ‘valid to’, changing an interval input to a point-time output. This is a low-level function that will generate confusing results if not used as part of a larger context. It is recommended to instead use ‘make_resource’ or ‘merge_event’ or ‘timechart’ to go from event to resource, and ‘make_event’ to go from resource to event shape. If you absolutely need this: Beware changing time to a value that is too far off from the current timestamp field, because it may end up falling outside of the processing time window. Also, setting a “valid to” that’s before the “valid from” time will cause the datum to be filtered out by subsequent packing.

Is this verb streamable? Always.

Usage

set_valid_to [ options ], [ columnname ]

Examples

set_valid_to ts_col

Sets the ‘valid to’ designation of the output dataset as the ‘ts_col’ field.

set_valid_to options(max_time_diff:duration_hr(1)), ts_col

Sets the ‘valid to’ designation of the output dataset as the ‘ts_col’ field, and the maximum time difference between the original ‘valid to’ and ‘ts_col’ is less than one hour.

set_valid_to

Removes the ‘valid to’ designation from the output dataset

Arguments

Argument

Type

Required

Multiple

options

options

False

False

columnname

fieldref

False

False

statsby (Stats By)

Type of operation: Aggregate

Description

Calculate statistics of columns with aggregate functions, based on (optional) grouping columns. If you want a streamable version, consider timestats or timechart.

Is this verb streamable? Never.

Usage

statsby [ groupby, ... ], groupOrAggregateFunction, ...

Examples

statsby Count:count(1), group_by(server_name)

Group input data by server name, calculating a count of rows per server name, returning a dataset with the two columns ‘server_name’ and ‘Count’.

Arguments

Argument

Type

Required

Multiple

groupby

fieldref

False

True

groupOrAggregateFunction

expression

True

True

surrounding (Surrounding)

Type of operation: Join

Description

Rows from the “right” dataset that fall within the specified frame of at least one row in the default dataset are unioned with the input dataset. The shape of output would be as if the right dataset and left dataset were combined using the union verb.

The column bindings are applied only to the rows of the input dataset. Rows from the right dataset will have these new columns set to null in the output.

Is this verb streamable? Always.

Usage

surrounding frame, source, [ column bindings, ... ]

Examples

filter <panic> | surrounding frame(back: 2s, ahead: 2s), @logs, panic:true

After filtering to the rows matching “panic”, this pulls in all the rows from “logs” whose timestamp is within 2s of the filtered rows. In the output, rows from the input dataset will have “panic” field populated with true. Rows from the “logs” dataset will have this field set to null.

Arguments

Argument

Type

Required

Multiple

frame

frame

True

False

source

datasetref

True

False

column bindings

expression

False

True

timechart (Time Chart)

Type of operation: Aggregate

Aliases: bucketize

Description

Bin (in time) and aggregate point or interval table columns through time, based on (optional) grouping columns. An optional window frame can be specified to compute hopping window aggregation.

Is this verb streamable? Sometimes.

Usage

timechart [ options ], bin_duration, [ frame ], [ groupby, ... ], groupOrAggregateFunction, ...

Examples

timechart 1h, Count:count(1), group_by(server_name)

Group input point table by server name, calculating a count of rows through time per server name per hour, returning a dataset with the 5 columns ‘valid_from’, ‘valid_to’, ‘bucket’, ‘server_name’, and ‘Count’.

timechart 1h, frame(back:24h), Count:count(1), group_by(server_name)

Group input point table by server name, calculating a moving count of rows through time per server name per hour, with each count covering the 24 hour window ending at the hour.

timechart options(empty_bins:true), 1h, Count:count(1), group_by(server_name)

Similar to the first example, but generate a row with NULL value for each time bin in the query window with no matching input rows. Because of empty_bins, the query may run slowly, especially if the input data points are sparse.

Arguments

Argument

Type

Required

Multiple

options

options

False

False

bin_duration

duration

True

False

frame

frame

False

False

groupby

fieldref

False

True

groupOrAggregateFunction

expression

True

True

timestats (Time Stats)

Type of operation: Aggregate

Description

Aggregate resource columns at every point in time, based on (optional) grouping columns

Is this verb streamable? Always.

Usage

timestats [ groupby, ... ], groupOrAggregateFunction, ...

Examples

timestats Count:count(1), group_by(server_name)

Group input resource by server name, calculating a count of rows for each slice of time per server name, returning a dataset with the 4 columns ‘valid_from’, ‘valid_to’, ‘server_name’, and ‘Count’. As opposed to timechart, this calculates values that change at any point in time, whereas timechart calculates aggregates per fixed bucket.

Arguments

Argument

Type

Required

Multiple

groupby

fieldref

False

True

groupOrAggregateFunction

expression

True

True

topk (Topk)

Type of operation: Filter

Description

Selects all data for each of top k ranked groups. If no rank method is provided, a default one will be used. If no grouping is specified, the set of primary key columns will be used as the grouping.

Is this verb streamable? Never.

Usage

topk k, [ rank ], [ groupby ]

Examples

topk 100

Select the top 100 groups using the default rank method: the hash of the group identifiers (the set of primary key columns).

topk 100, group_by(clusterUid, namespace)

Similar to the first example, but explicitly specifying the grouping

topk 100, max(restartCount)

Similar to the first example, but using a custom rank method to find the groups with most restarts

topk 1, group_by()

This topk operates on empty grouping, where all rows belong to the same group, and hence all rows will be selected

Arguments

Argument

Type

Required

Multiple

k

int64

True

False

rank

expression

False

False

groupby

fieldref

False

False

union (Union Event Datasets)

Type of operation: Join

Description

Create a new event dataset, consisting of events from two or more datasets, where the datasets are mapped onto the shape of the main input through column name matching, and filling in with NULL for mismatched column names. The event time column does not need to be explicitly mapped. It is an error to map columns of different types to the same name.

Is this verb streamable? Always.

Usage

union dataset, ...

Examples

union @second, @third

Create a new dataset that is the union of the main input dataset, and the @second and @third datasets, where names that are not shared are given NULL values in the opposite dataset.

Arguments

Argument

Type

Required

Multiple

dataset

datasetref

True

True