Customizing alert messages

About alert templates

In addition to recipients, channel actions also define the contents of alert messages through a Mustache template.

You may replace the default message with a custom HTML email or JSON payload template. Some variables are nested inside sections containing multiple values. See Template syntax for more about accessing items inside a section.

Available variables are described in the tables below. Note that some types of information may be accessible in multiple ways. You may find it helpful to navigate this page with the Table of Contents in the right sidebar.

Channels and Channel Actions

Channel and Channel Action variables contain names and links to view or edit the Channel or Action:

Channel element

Variable name

Example

Name

channel.name

“On Call”

URL

channel.url

Link to the Edit Channel page for this channel

Channel Action element

Variable name

Example

Name

channelAction.name

“#ops-p2-alerts”

URL

channelAction.url

Link to the Edit Channel Action page for this channel action

Monitors

Common monitor variables

Monitor variables contain details of the monitor, such as the triggering condition and notification threshold. Every monitor includes the following basic elements:

Monitor element

Variable name

Example

ID

monitor.id

“1234567890”

Name

monitor.name

“Test monitor”

Description

monitor.notifyWhen

“Send a Separate Informational Notification when Any of the data triggers”

Condition to trigger on

monitor.triggerWhen

“Check all of the data where count of CONTAINER_ID is greater than 100 in the last 10m0s”

URL

monitor.url

Link to the Edit Monitor page for this monitor

The monitor.notifyWhenDetails variables contain information about how the notification was configured, such as whether it is a single or separate notification.

Monitor element

Variable name

Example

Condition details

monitor.notifyWhenDetails

Section containing details of why the monitor triggered notifications to be sent

Any or all groups

monitor.notifyWhenDetails.condition

“Any”

Importance

monitor.notifyWhenDetails.importance

“Informational”

Single or separate

monitor.notifyWhenDetails.merged

“Separate”

In addition to the text description in monitor.triggerWhen, the monitor.triggerWhenDetails variables contain more information about the triggering condition.

Monitor element

Variable name

Example

Condition details

monitor.triggerWhenDetails

Section containing the details of the triggering condition

Duration

monitor.triggerWhenDetails.duration

How long the condition has been occurring: “10m0s”

Field to check

monitor.triggerWhenDetails.field

“CONTAINER_ID”

Grouping type

monitor.triggerWhenDetails.grouping

“None”

Group by fields

monitor.triggerWhenDetails.groupingFields

The fields to group notifications by: “deviceId”

Kind

monitor.triggerWhenDetails.kind

Type of monitor: Count, Promote, Text Value (Facet), Threshold (Metric)

Variables by type of monitor

Most kinds of monitors have additional values for details specific to that type.

For Count monitors:

Monitor element

Variable name

Example

Threshold value

monitor.triggerWhenDetails.threshold

“is greater than 100”

For Text Value (Facet) monitors:

Monitor element

Variable name

Example

Continuity

monitor.triggerWhenDetails.continuity

“all the time”

Triggering text value

monitor.triggerWhenDetails.fieldValueEquality

The triggering value to check for in the specified field: “active”

Threshold (Metric) monitors:

Monitor element

Variable name

Example

Threshold value

monitor.triggerWhenDetails.threshold

“is greater than 1000”

Notifications

Common notification variables

Notification variables contain details of the notification itself, such as when the alert triggered and the resources or values that triggered it. Every monitor includes the following basic notification elements:

Notification element

Variable name

Example

Description

notification.description

Value from description field for Promotion monitors, text describing trigger for others: “count of CONTAINER_ID is greater than 100 in the last 10m0s”

Recipient email

notification.email

Email address, for email notifications: “user@example.com”

Kind

notification.kind

For Promotion monitors, value from the field configured as the Notification Kind. Same as notification.description for other types.

Triggered at

notification.startTime

“2021-01-29T01:33:46Z”

URL

notification.url

Link to this notification in Observe.

Has resources? T/F

notification.hasResources

If this notification includes triggering resources: true

Has values? T/F

notification.hasValues

If this notification includes triggering values: false

Variables for triggering resources

If the notification includes resources, their details are available in multiple formats. Choose the one most appropriate for how you want to display resources in your alert. (See Template syntax for more about working with sections.)

Both resourcesByLinkType and resourcesWithLinkType are ordered by the link name, also called its label. The label represents the foreign key relationship created by linking datasets. For example, a logGroup field linked to a Cloudwatch Log Group resource creates a linked column, in this example called Log Group in the event table. The values in that Log Group column are the individual name items, for example “/aws/lambda/ObserveCollection”.

Event Landing Page showing the Log Group column as a linked field containing labels. The open menu shows that it is linked to the Cloudwatch Log Group Resource Set. The Log Group column contains the individual resources.

The resourcesByLinkType section contains resources grouped by the link name, also called its label. For each label, there is a list of all the items of that type. This is useful when you are looking for a specific item you expect in the data, as you need to know the label.

Notification element

Variable name

Example

Resources by link type

notification.resourcesByLinkType

Section containing resources by type

Resource name

name

“Chris R. User”

Resource URL

url

Resource URL

The resourcesWithLinkType section also groups items by the label, each containing a list of instances of that type. This is more convenient for presenting resources arranged into sections.

Notification element

Variable name

Example

Resource link type

notification.resourcesWithLinkType

Section containing link types with their instances

Resource link type

linkType

“User”, “Container”

Instances of this type

instances

List of resources, with name and url

The notification.resources section contains an unordered list of resources. This is useful for showing a list of items in a webhook payload.

Notification element

Variable name

Example

All triggering resources

notification.resources

Section containing all triggering resources

Resource name

name

Resource name

Resource URL

url

Resource URL

Examples

Using resourcesByLinkType:

Access values for a link type in this section by appending its label to #notification.resourcesByLinkType. Then specify the desired item, available values are name and url. This example displays the name for each User resource.

Template:

{{#notification.hasResources}}
  {{#notification.resourcesByLinkType.User}}
    {{name}}
  {{/notification.resourcesByLinkType.User}}
{{/notification.hasResources}}

Sample data:

"resourcesByLinkType": {
  "Cluster": [
    {
      "name": "k8s.example.com",
      "url": "https://..."
    }
  ],
  "User": [
    {
      "name": "Chris R. User",
      "url": "https://..."
    }
  ]
}

Using resourcesWithLinkType:

This example displays a list of url and name values for each link type.

Template:

{{#notification.hasResources}}
  {{#notification.resourcesWithLinkType}}
    <tr>
      <th>{{linkType}}</th>
    </tr>
    {{#instances}}
      <tr>
        <td class="resource-value">
          <a href="{{url}}">{{name}}</a>
        </td>
      </tr>
    {{/instances}}
  {{/notification.resourcesWithLinkType}}
{{/notification.hasResources}}

Sample data:

"resourcesWithLinkType": [
  {
    "linkType": "Cluster",
    "instances": [
      {
        "name": "k8s.example.com",
        "url": "https://..."
      }
    ]
  },
  {
    "linkType": "User",
    "instances": [
      {
        "name": "Chris R. User",
        "url": "https://..."
      }
    ]
  }
]

Using notification.resources:

This example iterates over the list of items in the resources section and displays the value of name for each.

Template:

"Resources": "{{#notification.resources}}{{name}} {{/notification.resources}}"

Sample data:

"resources": [
  {
    "name": "k8s.example.com",
    "url": "https://..."
  },
  {
    "name": "Chris R. User",
    "url": "https://..."
  }
]

Variables for triggering values

Like resources, triggering values are also available in multiple ways. If notification.hasValues is true, these variables contain details of the values that triggered the notification.

The resourcesByFieldName section contains a list of name/value pairs for each resource. This is useful when you are looking for a specific item you expect in the data, as you need to know the the field name.

Notification element

Variable name

Example

Values by field name

notification.valuesByFieldName

Section containing name/value pairs

The resourcesWithFieldName section contains a list of resources grouped by the field name. This is helpful for presenting resources grouped by field.

Notification element

Variable name

Example

Values with field name

notification.valuesWithFieldName

Section containing field names and values as individual items

Field name

fieldName

“city”

List of values for this field

values

[“San Mateo”, “San Francisco”]

The notification.values section contains two lists, one with field names and another with values for those fields. The two lists are in the same order: the first field in the fields list has a corresponding value in the first item in the values list. This is useful for constructing a two dimensional table of the triggering values.

Notification element

Variable name

Example

All triggering values

notification.values

Section containing all triggering values

List of value fields

fields

List of fields

List of value rows

rows

List of rows, in the same order as fields list

Examples

Using valuesByFieldName:

Similar to resourcesByLinkType, access values in a notification.valuesByFieldName section by appending the desired label. Since it’s a list, also append {{.}} to get its members. This example displays a list of userId values.

Template:

{{#notification.hasValues}}
  {{#notification.valuesByFieldName.userId}}{{.}}
  {{/notification.valuesByFieldName.userId}}
{{/notification.hasValues}}

Sample data:

"valuesByFieldName": {
  "MonitorName": ["An error occurred"],
  "userId": ["123"]
}

Using valuesWithFieldName:

This HTML example displays a list of values for each field name as rows in a table.

Template:

{{#notification.hasValues}}
  {{#notification.valuesWithFieldName}}
    <tr>
      <th>{{fieldName}}</th>
    </tr>
    {{#values}}
      <tr>
        <td class="field-value">
          <span>{{.}}</span>
        </td>
      </tr>
    {{/values}}
  {{/notification.valuesWithFieldName}}
{{/notification.hasValues}}

Sample data:

"valuesWithFieldName": [
  {
    "fieldName": "userId",
    "values": ["123"]
  },
  {
    "fieldName": "MonitorName",
    "values": ["An error occurred"]
  }
]

Using notification.values:

This example displays the contents of the rows list.

Template:

    "Values": "{{#notification.values.rows}}{{.}} {{/notification.values.rows}}",

Sample data:

"values": {
  "fields": [
    "userId",
    "MonitorName"
  ],
  "rows": [
    [
      "123",
      "An error occurred"
    ]
  ]
}

Template syntax

Most variables are string values, these may be placed anywhere in your template. When an alert is triggered, Observe replaces the template variables with values from the current alert.

Add the desired variables inside double braces, like this:

<td class="title">{{monitor.name}} at {{notification.startTime}}</td>

To add a comment, put the text inside a {{! }} element.

Example webhook payload template:

{
  {{! Display some basic info }}
  "Notification": "{{notification.kind}}"
  "Monitor": "{{monitor.name}}",
  "Trigger at": "{{notification.startTime}}",
  "Importance": "{{monitor.notifyWhenDetails.importance}}",
  "Trigger Condition": "{{monitor.triggerWhen}}",
  "Notification": "{{monitor.notifyWhen}}",
  "Description": "{{notification.description}}",
  "Resources": "{{#notification.resources}}{{name}} {{/notification.resources}}",
  "Values": "{{#notification.values.rows}}{{.}} {{/notification.values.rows}}",
  "URL": "{{{notification.url}}}"
}

notification.resources and notification.values.rows contain lists of the resources or threshold values that triggered the notification.

In the HTML example below, {{#notification.hasResources}} begins a template section. If notification.hasResources is True, the contents of this section are evaluated, displaying the list of resources, with their name and url. If False, the section is skipped. {{/notification.hasResources}} indicates the end of the section. Variables inside a {{#sectionName}} {{/sectionName}} conditional section are accessed directly by name, without a section prefix.

Example email body template:

{{#notification.hasResources}}
  {{#notification.resourcesWithLinkType}}
    <tr>
      <th>{{linkType}}</th>
    </tr>
    {{#instances}}
      <tr>
        <td class="resource-value">
          <a href="{{url}}">{{name}}</a>
        </td>
      </tr>
    {{/instances}}
  {{/notification.resourcesWithLinkType}}
{{/notification.hasResources}}

For more about template syntax, see the Mustache documentation.

Examples

Slack

Slack alerts use Block Kit and the Incoming Webhook URL for your Slack app.

Slack Incoming Webhook guidelines:

  • A message may have up to 50 blocks.

  • A text item inside a section text field may contain up to 3000 characters.

  • Avoid fields blocks inside a section, as they may only contain 10 fields.

    • text items in field blocks are limited to 2000 characters.

For more information, see the Slack layout block reference.

Basic Slack alert body example
{
  "blocks": [
    {
      {{! header is a large bold font }}
      "type": "header",
      "text": {
        "type": "plain_text",
        "text":  "Test Slack Alert at {{notification.startTime}}"
      }
    },
    {
      {{! section is plain text, can use markdown }}
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Monitor:*\n{{monitor.name}}"
      }
    },
    {
      {{! this is a comment }}
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Kind:*\n{{notification.kind}}"
      }
    },
    {
      {{! divider is a narrow horizontal line }}
      "type": "divider",
    },
    {
      {{! actions is for interactive elements, like a button }}
      "type": "actions",
      "elements": [
        {
          "type": "button",
          "text": {
            "type": "plain_text",
            "emoji": true,
            "text": "View Notification"
          },
          "style": "primary",
          "url": "{{notification.url}}"
        }
      ]
    }
  ]
}
More complex Slack alert body example
{
  "blocks": [
    {
      "type": "header",
      "text": {
        "type": "plain_text",
        "text":  "{{notification.kind}} at {{notification.startTime}}"
      }
    },
    {{#notification.hasResources}}
      {{#notification.resourcesWithLinkType}}
        {
          "type": "section",
          "text": {
            "type": "mrkdwn",
            "text": "*{{linkType}}:*\n{{#instances}}<{{url}}|{{name}}>\n{{/instances}}"
          }
        },
      {{/notification.resourcesWithLinkType}}
    {{/notification.hasResources}}
    {{#notification.hasValues}}
      {{#notification.valuesWithFieldName}}
        {
          "type": "section",
          "text": {
            "type": "mrkdwn",
            "text": "*{{fieldName}}:*\n{{#values}}{{.}}\n{{/values}}"
          }
        },
      {{/notification.valuesWithFieldName}}
    {{/notification.hasValues}}
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Monitor:*\n{{monitor.name}}"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Description:*\n{{notification.description}}"
      }
    },
    {
      "type": "actions",
      "elements": [
        {
          "type": "button",
          "text": {
            "type": "plain_text",
            "emoji": true,
            "text": "View Notification"
          },
          "style": "primary",
          "url": "{{notification.url}}"
        }
      ]
    }
  ]
}

Microsoft Teams

Teams alerts use an Incoming Webhook. For more details, see the Microsoft Teams documentation: Create and send messages

Microsoft Teams alert body example
{
  "@type": "MessageCard",
  "@context": "http://schema.org/extensions",
  "themeColor": "0076D7",
  "summary": "{{monitor.name}} triggered at {{notification.startTime}}",
  "sections": [{
    "activityTitle": "{{monitor.name}} triggered at {{notification.startTime}}",
    "activitySubtitle": "{{notification.description}}",
    "activityImage": "https://teamsnodesample.azurewebsites.net/static/img/image5.png",
    "facts": [{
      "name": "Assigned to",
      "value": "Unassigned"
    }, {
      "name": "Started at",
      "value": "{{notification.startTime}}"
    }, {
      "name": "Importance",
      "value": "{{monitor.notifyWhenDetails.importance}}"
    }],
    "markdown": true
  }],
   "potentialAction": [{
    "@type": "OpenUri",
    "name": "View Monitor",
    "targets": [{
      "os": "default",
      "uri": "{{notification.url}}"
    }]
  }]
}