One of the most common OPAL operations is searching for data matching, or not matching, a condition. The filter verb accepts a filter expression, and returns all matching events in the query time window. Additional verbs provide specialized matching conditions such as uniqueness, existence or non-existence, and top values.

Filter expressions

Filter expressions consists of boolean expressions that can use any supported OPAL functions and operators, including special ~ search operator. Some examples of the simplest filter expressions include the following:

// keep only rows where "is_connected" field is "true"
filter is_connected

// only rows where "severity" field is not "info"
filter not severity = "info"

// another way to write it
filter severity != "DEBUG"

// only rows with temperature out of range
filter temperature < 97 or temperature > 99

~ operator can match one or all the fields, for example,

  // keep rows where "log" field contains "hello", case-insensitive
  filter log ~ hello
  // keep rows where any field contains "hello", case-insensitive
  filter * ~ hello

Opal uses hello as a search term. The search term consists of a sequence using the following parts:

  • letters, digits, or underscores. Any other symbols must be enquoted using single or double enquoted strings. You can add any characters inside the quotes, and you can slash-escape quote characters. These are case-sensitive. For example, "fig\"bar" matches fig"bar, but not Fig"bar

  • glob character *, which matches 0 or more characters of any type, including newlines. For example, fig*bar matches fig123bar and fIgBaR. * can also anchor text to the beginning or end of the string when used at the beginning or end of the search term. For example, fig* only matches strings beginning with fig. Leading newlines and spaces ignored

  • search term may optionally start with - to inverse the match: -foo matches any string which does not contain foo

Search terms always match case-insensitively.

Search term examples:


Multiple search terms placed inside <> must all match, regardless of the order. For example:

filter log ~ <fig bar baz>

// "log" field must include all 3 words in no particular order. This is equivalent to
filter search(log, "fig", "bar", "baz")

// this will match rows where log starts with `foo`, and don't contain `bar`
filter log ~ <foo* -bar>

// "or" and other boolean operators can be used between `~` expressions:
filter log ~ foo or log ~ bar

~ operator also accepts POSIX extended regular expressions and IPv4 CIDRs.

// mathing on a regular expression
filter log ~ /foo|bar/

// same as
filter match_regex(log, /foo|bar/)

// IP matching
filter ip ~

// can also use wild cards
filter ip ~ 192.168.*.*

// or even shorter. At least two segments with two dots are required
filter ip ~ 192.168.*

The left side of the ~ expression can be any field, which converted to string if necessary, a JSON payload, or *, which means that condition should be matched by at least one field.

// any field contains "error", case insensitive
filter * ~ error

// none of the fields contain "error"
filter * !~ error

// any field contains both "word1" and "word1"
filter * ~ <word1 word2>

// this can also be shortened to
filter <word1 word2>

Unicode characters

There are several ways to use non-ASCII text with filter:

  • Text containing Unicode characters may be typed or pasted into the OPAL console like any other text.


    filter <हर दिन>
    filter @."ввод" < 5
    // These are equivalent
    filter <"😀">
    filter <\x{1F600}>
    filter <"\x{1F600}">
  • Unicode or special characters in a regular expression may be either a character or a hex value, but you must also specify the columns to search with ~:


    filter message ~ /😀/
    filter message ~ /\x{1F600}/
    filter message ~ /\x{000d}\x{000a}/
    filter message + name  ~ /\x{000d}\x{000a}/
    filter (message ~ /\x{000d}\x{000a}/) or (name ~ /\x{000a}/)

Handling null values

In OPAL, null values always have a type, but not handled in the same way as a regular value. This is particularly important in comparisons.

This statement returns events with a severity not equal to DEBUG, but only for events that have a severity value:

filter not severity="DEBUG"

An event that does not have a severity (in other words: the value is null), will never match. Use is_null or if_null to explicitly include them:

// exclude "DEBUG" but include null
filter not severity="DEBUG" or is_null(severity)

// replace null with empty string, then check
filter if_null(severity, '') != "DEBUG"

For filter expressions using contains(), ensure what filter compares against (the result of the contains()) isn’t null:

// This filter expression suppresses null values,
// because contains(field_with_nulls, "string") returns null
filter not contains(severity, "DEBUG")

// These filter expressions include null values,
// because potential null values are handled
filter is_null(severity) or not contains(severity, "DEBUG")
filter does not contain (if_null(severity, ""), "DEBUG")

For some comparisons, you may also compare with a null value of the appropriate type.

make_col positive_or_null:case(value > 0, value, true, int64_null())

Specialized filter verbs

In addition to filter, OPAL uses several additional verbs for different types of filter operations. See the OPAL filter verbs documentation for details. (Note that several of these verbs need a frame to be streamable.)


Change a field type

To change the type of an existing field, create a new field with the desired type. Use a new name to keep both, or replace the existing one by giving it the same name. This is useful when creating metrics, which require numeric fields to be float64.


make_col temperature:float64(temperature)

Extract from JSON

Reference properties in a JSON payload with either the dot or bracket operators:

make_col data:string(FIELDS.data), kind:string(FIELDS["name"])

Quote the string if the property name has special characters:

make_col userName:someField["user name"]
make_col userCity:someField."user city"
make_col requestStatus:someField.'request.status'

You may also combine methods:

// Sample data: {"fields": {"deviceStatus": {"timestamp": "2019-11-15T00:00:06.984Z"}}}
make_col timestamp1:fields.deviceStatus.timestamp
make_col timestamp2:fields["deviceStatus"]["timestamp"]
make_col timestamp3:fields.deviceStatus.["timestamp"]
make_col timestamp4:parsejson(string(fields.deviceStatus)).timestamp

Extract and modify values using replace_regex():

make_col state:replace_regex(string(FIELDS.device.date), /^.*([0-9]{4,4})-([0-9]{1,2})-([0-9]{1,2}).*$/, '\\3/\\2/\\1', 1)
make_col state:replace_regex(string(FIELDS.device.state), /ошибка/, "error", 0)
make_col state:replace_regex(string(FIELDS.device.manufacturer), /\x{2122}/, "TM", 0)

Extract with a regex

Use extract_regex to extract fields from a string.

extract_regex data, /(?P<deviceid>[^|]*)\|count:(?P<counts>[^|]*)\|env:(?P<env>[^|]*)/


extract_regex allows named capture groups, unlike filter expressions.


Registering with set_metric

  • set_metric registers a single metric. It accepts an options object containing details of its type, unit, how it should be aggregated, and other options.

    set_metric options(label:"Temperature", type:"gauge", unit:"C", rollup:"avg", aggregate:"avg", interval:5m), "temperature"
    set_metric options(label:"Power", description:"Power in watts", type:"gauge", rollup:"avg", aggregate:"avg"), "power"
    • The type of a metric determines how its values are interpreted.

      Metric type



      A monotonically increasing total over the life of the metric. A cumulativeCounter value is never negative.


      The difference between the current metric value and its previous value.


      A measurement at a single point in time.

    • A metric rollup method determines how multiple data points for the same metric are summarized over time. A single value is created for multiple values in each rollup time window.

      Rollup method



      The average (arithmetic mean) of all values in the window.


      The number of non-null values in the window.


      The largest value.


      The smallest value.


      The rate of change across the window, which may be negative for delta and gauge types. A negative rate for a cumulativeCounter is treated as a reset.


      The sum of all values in the window.

    • The aggregate type determines how values are aggregated across multiple metrics of the same type. For example, temperature metrics from multiple devices. Aggregate types correspond to the aggregate function of the same name.

      Aggregate type



      An arbitrary value from the window, nondeterministically selected. Useful if you need a representative value, may be, but not guaranteed to be, faster to calculate than other methods.


      Like any, but guaranteed to be not null.


      The average (arithmetic mean.)


      The number of non-null values.


      An estimate of the number of unique values in the window. Faster than countdistinctexact.


      The number of unique values in the window, slower but more accurate than countdistinct.


      The largest value in the window.


      An approximation of the median value, faster than medianexact.


      The median value across the window.


      The smallest value in the window.


      The standard deviation across all values in the window.


      The sum of all values in the window.


    For more about units, see Introduction to Metrics.