Customizing Alert Messages

Customizing Shared Actions

A Shared Action cannot be customized from a Monitor edit page. This is to prevent accidental edits to shared resources. To customize these actions, navigate to Monitors, View Shared Actions, and either Edit or Duplicate an Action.

Customization Options

Observe’s Monitoring system can use our extended Mustache formatting language to customize the data and metadata sent to destinations. The filtering and data reference options have been expanded for Monitors v2.

Control Options

There are three types of tags that can be used with Mustache:

Variable insertion

Variables are referenced with a single Mustache tag. At rendering, Mustache replaces a variable such as {{alert.severity.level}} with its value, such as Critical.

If test

An If test has an open and a close tag which can be tested for positive or negative matching.

  • Positive match: If the variable contains data, then insert everything between the tags; if not, nothing is inserted. Example:

{{#monitor.description}}<tr><th>Monitor Description</th></tr><tr><td>{{monitor.description}}</td></tr>{{/monitor.description}}

This will insert an HTML row with the Monitor Description if one is set. If the Monitor does not have a description, the row will not be included.

  • Negative match: If the variable does not contain data, then insert everything between the tags. For example:

{{^alarm.type.isAlarmEnded}}danger{{/alarm.type.isAlarmEnded}}"

This will emit the word “danger” if isAlarmEnded is false.

Transformation

A transformation has an open and a close tag. It will apply the transformation to anything between the tags. For example:

"url":"https://12345678901.observeinc.com/workspace/87654321/dashboard/User-Dashboard-12481632?param-user={{~urlqueryescape}}{{alert.valuesByName.User.value}}{{/urlqueryescape}}~userUid~{{alert.valuesByName.userUid.value}}"

This creates a properly escaped Observe dashboard link referring to a User resource. Without escaping, the spaces and punctuation in a person’s name would produce a non-working URL.

Iteration

The data in an alert can be structured in arrays, which can be looped over. For instance, a set of resources involved in an alert can be referenced with an array:

{{#alert.resources}}
    ,{
        "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "*<{{url}}|{{label}} : {{value}}>*"
            }
    }
{{/alert.resources}}

Each resource’s url, label, and value will be inserted into its own section.

The isFirst or isLast tests can be used for punctuation in lists:

[{{#alert.values}}{{^isFirst}}, {{/isFirst}}{{value}}{{/alert.values}}]

Each Alert Value will be presented in a comma separated list, such as [one, two, three].

Note

Array iteration functions at the alert level as well: the entire payload may be dumped without parsing or formatting by replacing the template with {{.}}. This is not recommended for most destinations, but can be useful in a Webhook.

Email Action - email addresses

The email notification action now supports the use of alert level variables in the Email addresses field. For example, your monitor contains a Group by column called service_owner that contains one or many email addresses, this can then be referenced in the Email addresses field, via the following syntax {{alert.valuesByName.service_owner.value}}. The values can be a single email address or, a semicolon or comma-separated list of email addresses.

Monitor Level Objects

  • {{name}}: The monitor name

  • {{description}}: The description saved with the monitor

  • {{id}}: The Observe object ID for this monitor

  • {{type}}: One of “Count”, “Promote”, or “Threshold”

  • {{icon}}: The icon saved with the monitor

  • {{url}}: The link to the monitor in the Observe console

Alert Level Objects

  • {{timestamp}}: The time the match was seen. Same as {{timestamp.rfc3339}}

  • {{start}}: The time the match condition started. Same as {{start.rfc3339}}

  • {{end}}: The time the match condition ended. This will only be set if isActive is false. Same as {{end.rfc3339}}

  • {{isActive}}: Indicates if the matching condition is still triggering.

  • {{url}}: A link to the Observe console for this alert.

  • {{id}}: The Observe object ID for this alert.

  • {{severity}}: The severity of the alert. Same as {{severity.level}}

  • {{type}}: The event type that created this notification. Same as {{type.friendly}}.

  • {{values}}: The values captured from the monitor match.

  • {{dataUrl}}: A link to the Observe console to explore the data and filtered context that generated this alert.

  • {{resources}}: An array of all the resources linked to this match.

  • {{resourcesByLink}}: The same resources as resources but you can index by the known resource type.

{{alert.start}}

This variable can be rendered directly, which prints the rfc3339 value or using one of the sub fields.

  • {{rfc3339}}: The RFC3339 Format (like 2024-06-01T12:34:56Z)

  • {{timePart}}: Just minutes and seconds (like 12:34 UTC)

  • {{datePart}}: Just the date without the year (like Jun 01)

  • {{epoch}}: The Unix seconds since epoch in UTC (like 1717244636)

{{alert.end}}

Same as {{alert.start}}

{{alert.timestamp}}

Same as {{alert.start}}

{{alert.severity}}

This variable can be rendered directly, which prints the same as level or using the sub fields below.

  • {{level}}: One of “Critical”, “Error”, “Informational”, or “Warning”

  • {{isCritical}}: Indicates if the level is critical

  • {{isError}}: Indicates if the level is error

  • {{isWarning}}: Indicates if the level is warning

  • {{isInformational}}: Indicates if the level is info

{{alert.type}}

This variable can be rendered directly, which prints the same as friendly or using the sub fields below.

  • {{eventType}}: One of “NewAlarm”, “Reminder”, “AlarmEnded”

  • {{friendly}}: A more user-friendly version of eventType

  • {{isNewAlarm}}: Indicates if the event type is NewAlarm

  • {{isReminder}}: Indicates if the event type is Reminder

  • {{isAlarmEnded}}: Indicates if the event type is AlarmEnded

{{alert.values}}

This variable can be enumerated as a section and contains all values captured from the monitor match. You can filter by the type, which allows you to, for example, only print only the group-by values. Not all columns of a monitor match are captured. Values that are used to link to resources, to define the grouping, are part of the aggregation (like count), or are part of a promote monitor are captured.

<!-- As an HTML table >
<table>
{{#alert.values}}
    <tr><td>{{name}}</td><td>{{value}}</td></tr>
{{/alert.values}}
{{^alert.values}}
    <tr><td span=2>No values captured with this alert.</td></tr>
{{/alert.values}}
</table>

<!-- As a list instead of a table >
<ul>
{{#alert.values}}
    <li>{{type.isGroupBy}}Grouped By {{name}} : {{value}}{{/type.isGroupBy}}<li>
{{/alert.values}}
{{^alert.values}}
    <li>No values captured with this alert.<li>
{{/alert.values}}
</ul>

  • {{type}}: The type contains sub fields you can use to test why the value was captured. A value can be multiple types.

  • {{name}}: A rendered name for the captured column value

  • {{value}}: The captured value in string form

  • {{isFirst}}: Indicates if this is the first in the iteration

  • {{isLast}}: Indicates if this is the last in the iteration

{{alert.values.type}}

This subtype that has “is*” flags for each of the types a value can be. These are not necessarily exclusive.

  • {{isGroupBy}}: The value is used in the grouping.

  • {{isAggregation}}: The value is part of the aggregation calculation.

  • {{isLinkSourceField}}: The value is used to link to a resource.

{{alert.valuesByName}}

You can access the {{alerts.values}} elements using their unique name. All of the fields exist except for the iteration fields isFirst and isLast.

{{#alert.valuesByName.Image}}
The crashlooping container's image is {{value}}
{{/alert.valuesByName.Image}}
{{^alert.valuesByName.Image}}
The crashlooping container's image is unknown.
{{/alert.valuesByName.Image}}

{{alert.resources}}

This variable can be enumerated as a section for each resource. Each resource contains:

  • {{label}}: The label or name of the resource

  • {{value}}: The label value dereferenced from the resource

  • {{url}}: A link in Observe to the resource explorer for this resource

  • {{isFirst}}: Indicates if this is the first in the iteration

  • {{isLast}}: Indicates if this is the last in the iteration

Data available in an Alert

Input data and type of monitor can complicate what’s available, but there are basic patterns.

Threshold monitor

The alert from a threshold monitor potentially has resources and group by values. Your resources and grouped values can be referenced specifically with {{alert.resources}} and {{alert.values}}, if they are present in the monitored data and monitor configuration.

Count monitor

The alert from a Count monitor is limited to the count and group by values, and the resource array is not populated. Your {{alert.values}} can be referenced, if they are present in the monitored data and monitor configuration.

Promote monitor

The alert from a Promote monitor typically has a great deal of data. The {{alert.resources}} and {{alert.values}} arrays are both likely to have data that can be referenced if present in the monitored data and monitor configuration.

Resource columns and value columns are likely to overlap in a Promote monitor alert, meaning that you may see duplicate entries in your alert. Template customization to select or eliminate specific fields can be useful.