filter

Type of operation: Filter

Description

Exclude rows from the input dataset that do not match the given predicate expression.

Types of accepted expressions:

  • Boolean: returns True if the row matches or False if not. May include OPAL functions that return bool, such as ipv4_address_in_network

  • Match: A field or a *, followed by =, !=, ~, or !~ and a search expression. Matches a field, or all fields if * is used against the expression. Condition is inverted by !~. Search expression is one of:

    • A field, for example sourceAccountID=destinationAccountID

    • A “search term”, which is matched case insensitively and is a sequence of the following:

      • A single word consisting of letters, digits and underscores

      • A string enclosed in double or single quotes which can include arbitrary symbols. Quote symbols can be included with escaping, for example "foo \" bar"

      • A glob *, which matches a sequence of any characters

      • Search term can optionally start with a - for negative matches

    • A sequence of search terms enclosed in <>, meaning all search terms should match regardless of their order (AND)

    • A regular expression enclosed in //

    • An IPv4 CIDR like 1.2.3.4/16 or 1.2.*.*

  • A sequence of search terms enclosed in <>, a shorthand for * ~ <search terms>, meaning at least one field matches all search terms (OR)

Field values will be eagerly coerced to strings when necessary.

Note: to filter resources, use ever with a relative time range rather than filter. Resources track the state of multiple rows, which may not be easily matched by a filter expression. Filtering a subset of a resource’s underlying observations can have unexpected results.

Usage

filter predicate

Argument

Type

Optional

Repeatable

Restrictions

predicate

bool

no

no

none

Accelerable

filter is always accelerable if the input is accelerable. A dataset that only uses accelerable verbs can be accelerated, making queries on the dataset respond faster.

Examples

filter string(status_code) ~ /^5.*/

Keep only rows where the status_code column, converted to string, starts with “5”.

filter not method="POST"

Keep only rows where the method column is not equal to the string “POST”

filter json_payload.name ~ /^TEST/

Keep only rows where property name of the JSON field json_payload matches the specified regular expression (begins with “TEST”).

filter count >= 5 and count <= 100

Keep only rows where the count column is between 5 and 100, inclusive.

filter contains(log, "ERROR")

Keep only rows where the log column contains the string “ERROR”. Note that the contains() function is case-sensitive.

filter <foo ba*r "BA"z>

Keep only rows where some column, converted to string if necessary, contains foo, ba*r, and "BA"z search terms, in any order.

filter <foo bar -baz>

Keep only rows where some column, converted to string if necessary, contains foo and bar, but not baz (case-insensitive).

filter log ~ error

Keep only rows where column log contains word error (case-insensitive).

filter * ~ -foo"/"baz

Keep only rows where none of the columns contain foo/baz.

filter log ~ /^DEBUG/

Keep only rows where the field log matches the specified regular expression (begins with “DEBUG”).

filter json_payload.status = <success>

Keep only rows where property status of the JSON field json_payload contains the string “success” (case-insensitive).