Examples¶

Filtering¶

One of the most common OPAL operations is searching for data matching, or not matching, a condition. The filter verb accepts a predicate expression, a filter condition, and returns all matching events in the query time window. Additional verbs provide specialized matching conditions such as uniqueness, existence or non-existence, and top values.

Filter expressions¶

The simplest filter expressions use common arithmetic and logical operators, such as + and not. You may also use the equivalent function for those operators that have them.

Construct more complex conditions with POSIX extended regular expressions, full text search, and OPAL functions such as is_null().

Query every searchable text field in the event with the <...> text search operator.
```
filter <text search>
filter <word1 word2>
filter <some "words with spaces">
filter <some words or "other words" or word3>
```
<text> searches for the given value as literal text. Multiple space-delimited words are individual search terms, with and implied. To search for a phrase, enclose it in quotes. Note that or is special in the search syntax, it means “the thing on the left, or the thing on the right.” If you want to search for the word or, enclose it in quotes:
```
filter <"or">
```
Note

See below for how to search for text inside JSON with ~.
Filter on a specific field with ~

The ~ operator allows searching within the specified column, which may also be done with the OPAL search function. In other words, these two statements are equivalent:
```
// match values of log containing all three terms
filter log ~ <foo bar baz>
filter search(log, "foo", "bar", "baz")
```
To specify multiple columns to search:
```
filter message + error ~ <critical>
filter (message ~ <fatal>) or (error ~ <critical>)
```
The ~ operator also allows you to search for text inside JSON blobs, which are not standard searchable text:
```
// look for "fatal" and "error"
filter json_payload ~ <fatal error>
```
Regular expressions

For more complex pattern matching, combine filter with match_regex:
```
filter match_regex(log, /foo|bar|baz/)
```

Comparisons and logical expressions

filter temperature > 60 and temperature < 80
filter temperature < 30 or temperature > 100
filter hostname="www" or (hostname="api" and user="root")
filter not severity="DEBUG"
filter not log ~ /^DEBUG/
filter not <action completed successfully>

Unicode characters

There are several ways to use non-ASCII text with filter:
- Text containing Unicode characters may be typed or pasted into the OPAL console like any other text.
  
  Examples:
```
filter <हर दिन>
filter @."ввод" < 5

// These are equivalent
filter <"😀">
filter <\x{1F600}>
filter <"\x{1F600}">
```
- Unicode or special characters in a regular expression may be either a character or a hex value, but you must also specify the columns to search with ~:
  
  Examples:
```
filter message ~ /😀/
filter message ~ /\x{1F600}/

filter message ~ /\x{000d}\x{000a}/
filter message + name  ~ /\x{000d}\x{000a}/
filter (message ~ /\x{000d}\x{000a}/) or (name ~ /\x{000a}/)
```

Handling null values¶

In OPAL, null values always have a type. But they are not handled in the same way as a regular value. This is particularly important in comparisons.

This statement returns events with a severity not equal to DEBUG, but only for events that have a severity value:

filter not severity="DEBUG"

An event that does not have a severity (in other words: the value is null), will never match. Use is_null or if_null to explicitly include them:

// exclude "DEBUG" but include null
filter not severity="DEBUG" or is_null(severity)

// replace null with empty string, then check
filter if_null(severity, '') != "DEBUG"

For filter expressions using contains(), ensure what filter compares against (the result of the contains()) isn’t null:

// This filter expression suppresses null values,
// because contains(field_with_nulls, "string") returns null
filter not contains(severity, "DEBUG")

// These filter expressions include null values,
// because potential null values are handled
filter is_null(severity) or not contains(severity, "DEBUG")
filter not contains(if_null(severity, ""), "DEBUG")

For some comparisons, you may also compare with a null value of the appropriate type.

make_col positive_or_null:case(value > 0, value, true, int64_null())

Specialized filter verbs¶

In addition to filter, there are several additional verbs for different types of filter operations. See the OPAL filter verbs documentation for details. (Note that several of these verbs need a frame to be streamable.)

Fields¶

Change a field’s type¶

To change the type of an existing field, create a new field with the desired type. Use a new name to keep both, or replace the existing one by giving it the same name. This is useful when creating metrics, which require numeric fields to be float64.

Example:

make_col temperature:float64(temperature)

Extract from JSON¶

Reference properties in a JSON payload with either the dot or bracket operators:

make_col data:string(FIELDS.data), kind:string(FIELDS["name"])

Quote the string if the property name has special characters:

make_col userName:someField["user name"]
make_col userCity:someField."user city"
make_col requestStatus:someField.'request.status'

You may also combine methods:

// Sample data: {"fields": {"deviceStatus": {"timestamp": "2019-11-15T00:00:06.984Z"}}}
make_col timestamp1:fields.deviceStatus.timestamp
make_col timestamp2:fields["deviceStatus"]["timestamp"]
make_col timestamp3:fields.deviceStatus.["timestamp"]
make_col timestamp4:parsejson(string(fields.deviceStatus)).timestamp

Extract and modify values using replace_regex():

make_col state:replace_regex(string(FIELDS.device.date), /^.*([0-9]{4,4})-([0-9]{1,2})-([0-9]{1,2}).*$/, '\\3/\\2/\\1', 1)
make_col state:replace_regex(string(FIELDS.device.state), /ошибка/, "error", 0)
make_col state:replace_regex(string(FIELDS.device.manufacturer), /\x{2122}/, "TM", 0)

Extract with a regex¶

Use extract_regex to extract fields from a string.

extract_regex data, /(?P<deviceid>[^|]*)\|count:(?P<counts>[^|]*)\|env:(?P<env>[^|]*)/

Note

extract_regex allows named capture groups, unlike filter expressions.

Metrics¶

Registering with `set_metric`¶

set_metric registers a single metric. It accepts an options object containing details of its type, unit, how it should be aggregated, and other options.

set_metric options(label:"Temperature", type:"gauge", unit:"C", rollup:"avg", aggregate:"avg", interval:5m), "temperature"
set_metric options(label:"Power", description:"Power in watts", type:"gauge", rollup:"avg", aggregate:"avg"), "power"

The type of a metric determines how its values are interpreted.

Metric type	Description
cumulativeCounter	A monotonically increasing total over the life of the metric. A cumulativeCounter value is never negative.
delta	The difference between the current metric value and its previous value.
gauge	A measurement at a single point in time.

A metric’s rollup method determines how multiple data points for the same metric are summarized over time. A single value is created for multiple values in each rollup time window.

Rollup method	Description
avg	The average (arithmetic mean) of all values in the window.
count	The number of non-null values in the window.
max	The largest value.
min	The smallest value.
rate	The rate of change across the window, which may be negative for delta and gauge types. A negative rate for a cumulativeCounter is treated as a reset.
sum	The sum of all values in the window.

The aggregate type determines how values are aggregated across multiple metrics of the same type. For example, temperature metrics from multiple devices. Aggregate types correspond to the aggregate function of the same name.

Aggregate type	Description
any	An arbitrary value from the window, nondeterministically selected. Useful if you need a representative value, may be, but not guaranteed to be, faster to calculate than other methods.
any_not_null	Like `any`, but guaranteed to be not null.
avg	The average (arithmetic mean.)
count	The number of non-null values.
countdistinct	An estimate of the number of unique values in the window. Faster than countdistinctexact.
countdistinctexact	The number of unique values in the window, slower but more accurate than countdistinct.
max	The largest value in the window.
median	An approximation of the median value, faster than medianexact.
medianexact	The median value across the window.
min	The smallest value in the window.
stddev	The standard deviation across all values in the window.
sum	The sum of all values in the window.

Note

For more about units, see Introduction to Metrics.

Examples¶

Filtering¶

Filter expressions¶

Handling null values¶

Specialized filter verbs¶

Fields¶

Change a field’s type¶

Extract from JSON¶

Extract with a regex¶

Metrics¶

Registering with set_metric¶

Registering with `set_metric`¶