filter

Type of operation: Filter

Description

Exclude rows from the input dataset that do not match the given predicate expression.

Types of accepted expressions:

• Boolean: returns True if the row matches or False if not. May include OPAL functions that return bool

• A field or a *, followed by ~ or !~ and a search expression. Matches a field, or all fields if * is used against the expression. Condition is inverted by !~. Search expression is one of:

  • A “search term”, which is a sequence of the following:

    • A single word consisting of letters, digits and underscores, matched case insensitively

    • A string enclosed in double or single quotes which can include arbitrary symbols and matches case-sensitively. Quote symbols can be included with escaping, for example "foo \" bar"

    • A glob *, which matches a sequence of any characters

    • Search term can optionally start with a - for negative matches

  • A sequence of search terms enclosed in <>, meaning all search terms should match regardless of their order

  • A regular expression enclosed in //

  • An IPv4 CIDR like 1.2.3.4/16 or 1.2.*.*

• A sequence of search terms enclosed in <>, a shorthand for * ~ <search terms>, meaning at least one field matches all search terms

Field values will be eagerly coerced to strings when necessary.

Note: to filter resources, use ever with a relative time range rather than filter. Resources track the state of multiple rows, which may not be easily matched by a filter expression. Filtering a subset of a resource’s underlying observations can have unexpected results.

Usage

filter predicate

Argument	Type	Required	Multiple
predicate	bool	Required	Only one

Accelerable

filter is always accelerable if the input is accelerable. A dataset that only uses accelerable verbs can be accelerated, making queries on the dataset respond faster.

Examples

filter string(status_code) ~ /^5.*/

Keep only rows where the status_code column, converted to string, starts with “5”.

filter not method="POST"

Keep only rows where the method column is not equal to the string “POST”

filter json_payload.name ~ /^TEST/

Keep only rows where property name of the JSON field json_payload matches the specified regular expression (begins with “TEST”).

filter count >= 5 and count <= 100

Keep only rows where the count column is between 5 and 100, inclusive.

filter contains(log, "ERROR")

Keep only rows where the log column contains the string “ERROR”. Note that the contains() function is case-sensitive.

filter 

Keep only rows where some column, converted to string if necessary, contains foo, ba*r, and "BA"z search terms, in any order.

filter 

Keep only rows where some column, converted to string if necessary, contains foo and bar, but not baz (case-insensitive).

filter log ~ error

Keep only rows where column log contains word error (case-insensitive).

filter * ~ -foo"BaR"baz

Keep only rows where none of the columns contain fooBaRbaz, where foo and bar are case-insensitive, and BaR is case-sensitive.

filter log ~ /^DEBUG/

Keep only rows where the field log matches the specified regular expression (begins with “DEBUG”).

filter json_payload.status = 

Keep only rows where property status of the JSON field json_payload contains the string “success” (case-insensitive).