Filter operations

One of the most common OPAL operations is searching for data matching, or not matching, a condition. The filter verb accepts a filter expression, and returns all matching events in the query time window. Additional verbs provide specialized matching conditions such as uniqueness, existence or non-existence, and top values.

Filter expressions

Filter expressions consist of Boolean expressions that can use any supported OPAL functions and operators, including the special ~ inexact search operator. Some examples of the simplest filter expressions include the following:

The ~ (tilde) operator supports inexact matching, and can match one or all of the fields. Some examples of inexact matches include:

Those OPAL examples use hello as a search term. Search terms can have any of the following:

• Letters, digits, or underscores
  • Any other symbols, such as spaces, quotes, colons, or dashes, must be quoted using single ' or double " quote strings. You can add any characters inside the quotes, and you can slash-escape quote characters
  • Quoted search term segments are case-sensitive. For example, "fig\"bar" matches fig"bar, but not Fig"bar
• Glob character *, which matches 0 or more characters of any type, including newlines.
  • For example, fig*bar matches fig123bar and fIgBaR
  • Glob * can also anchor text to the beginning or end of the string when used at the beginning or end of the search term. For example, fig* only matches strings beginning with fig
  • An * inside a quoted string will search for a literal asterisk character, such as "* list item".
  • You can use globs together with quoted strings, for example: "fig:"*":bar"
  • Multiple glob * characters can be used in a single search, such as: "fig:"*":bar:"*"baz"
• Leading newlines and spaces in the source data are ignored
• A search term may optionally start with - to invert the match: -foo matches any string which does not contain foo

Non-quoted search terms always match case-insensitively (though quoted search terms are case sensitive). For example:

More search term examples:

Multiple search terms can be combined using Boolean expressions. For example:

The tilde ~ operator also accepts POSIX extended regular expressions and IPv4 CIDRs.

The left side of the ~ expression can be any field, converted to a string if necessary, a JSON payload, or *, which means that condition should be matched by at least one field.

Unicode characters

There are several ways to use non-ASCII text with filter:

• Text containing Unicode characters may be typed or pasted into the OPAL console like any other text. For example:

• Unicode or special characters in a regular expression may be either a character or a hex value, but you must also specify the columns to search with ~. For example:

Handle null values

In OPAL, null values always have a type, but not handled in the same way as a regular value. This is particularly important in comparisons.

This statement returns events with a severity not equal to DEBUG, but only for events that have a severity value:

An event that does not have a severity (in other words: the value is null), will never match. Use is_null or if_null to explicitly include them:

For filter expressions using contains(), ensure what filter compares against (the result of the contains()) isn't null:

For some comparisons, you may also compare with a null value of the appropriate type.

Specialized filter verbs

In addition to filter, OPAL uses several additional verbs for different types of filter operations. See the OPAL filter verbs documentation for details. (Note that several of these verbs need a frame to be streamable.)

• always
• dedup
• ever
• never
• topk

Fields

Change a field type

To change the type of an existing field, create a new field with the desired type. Use a new name to keep both, or replace the existing one by giving it the same name. This is useful when creating metrics, which require numeric fields to be float64. For example:

Extract from JSON

Reference properties in a JSON payload with either the dot or bracket operators:

Quote the string if the property name has special characters:

You may also combine methods:

Extract and modify values using replace_regex():

Extract with a regex

Use extract_regex to extract fields from a string.

📘

Note

extract_regex allows named capture groups, unlike filter expressions.

Metrics

Register a metric with set_metric

Use set_metric to register a single metric. It accepts an options object containing details of its type, unit, how it should be aggregated, and other options.

The type of a metric determines how its values are interpreted.

Metric rollup methods

A metric rollup method determines how multiple data points for the same metric are summarized over time. A single value is created for multiple values in each rollup time window.

Metric aggregate types

The aggregate type determines how values are aggregated across multiple metrics of the same type. For example, temperature metrics from multiple devices. Aggregate types correspond to the aggregate function of the same name.

📘

Note

For more about units, see Collect and use metrics.

Links

Observe represents foreign keys through a concept called links. Links consist of two pieces: a mapping of columns in the current dataset to columns in a target dataset, and a name to refer to the link itself.

Create with set_link

Use set_link to create a link, and define the mapping between columns in the current and target Datasets.

This defines a link named ^Cluster between the current dataset and @"K8s Cluster", and maps the column clusterUid to the uid column of @"K8s Cluster".

Links may also be defined using a composite key, which can be composed of more than one column:

Four local columns, namely containerName, podName, clusterUid and namespace link to their counterparts in @Container. This is necessary because the @Container dataset defines the primary key in terms of those four columns.

Link labels

Because links are represented in the dataset by the key components, it's often not clear at a glance what value is pointed to by a link. For example, a cluster might have a primary key which is a UUID like 4ef39c4f-7685-11e8-9d40-02ab4d0e1e2e; while precise and unambiguous, it is not very useful for humans examining the data. Instead, Observe hides these key columns by default, and presents a column of type link, which is rendered using the label column of the target dataset.

For example, with the ^Cluster link defined above, although the local dataset stores only a clusterUid column containing the UUID, this is displayed as a column named ^Cluster with the value of @"K8s Cluster".name inlined. Observe will display the value on the matched row in the target dataset pulled from the column specified by set_label as the target dataset's label.

Sometimes, it is more helpful to filter by the value displayed for a link column rather than the local column values themselves. The label() function can be used on links to retrieve the label for the linked resource, which can then be used as a normal string value:

This filters out any rows with ^Clusters that do not have a label matching the regular expression /prod.*/i.

Join with links

Since links define a mapping from the current dataset to another input dataset, a simple equijoin between the two datasets can be written succinctly:

This is equivalent to an explicit equijoin between the source and target columns defined by the link:

This syntax mirrors the natural join syntax wherein the first argument is the dataset, and works for many Join Verbs, such as join, fulljoin, leftjoin and lookup.

Group by links

It's frequently useful to aggregate over all rows related to a linked resource, e.g., to summarize logs grouped by the container they come from:

The ... is the "unpack" operator. Conceptually, it acts as if you replaced the ^Container argument with a listing of each of the columns in the local dataset. So, the above is equivalent to:

which will bin all events in the current dataset by the container they come from, returning the count of events for each container.

Note that you can also group by the label of a link, though this has subtly different semantics.

The above groups bins by the label of the linked resource, for example, the string value presented to you in the UI. Depending on how the @Container dataset is defined, this may or may not give you the same results as grouping by ^Container.... This is because label values are not guaranteed to be unique. For example, if containers are simply named after the service they run, then group_by(label(^Container)) gives you bins that include that container's values across all clusters they appear in. Make sure you select the right grouping behavior for the query you want to perform.