Alerting Example: Channels, Channel Actions, and Monitors

Here is an example that illustrates how channels and channel actions work with monitors to create alerts.

The goal:

You want to alert on too many errors in the TestWebApp application. Alerts should go to the owner of the application, a-developer@example.com, and a Slack channel, #test-web-app.

How?

Use a monitor to watch the number of errors and alert when it exceeds a threshold. You expect similar alertable conditions in the future, so set up a channel and actions in advance:

  • Create channel actions for each type of recipient (email or webhook.)
  • Create a channel containing the desired actions (recipients.)
  • Create a monitor whose notifications send alerts to this channel when the triggering condition occurs.

Monitor notifications may send alerts to multiple channels, and a channel may contain multiple actions. Channels allow you to configure alerts for a group of recipients without selecting them individually for each new monitor.

(You can also create channels on the fly. See below for more.)

Create Channel Actions

A channel action sends a single type of alert. For multiple types, or for different alert messages for different recipients, create a channel action for each.

To send an email to a-developer@example.com:

  • In the monitors tab, click Channel Actions and then Create Channel Action.
  • Under Content, select Email and add the recipient's email address. (You could add others too.)
  • Leave the other fields at their defaults.

(For more about the message body, see Customizing Alert Messages)

Click Continue to go to the Channel section:

  • You haven't created a channel yet, so skip selecting one for now.
  • Change the default throttling if you like.

Click Continue to go to the Name section:

  • Give this action a name: "TestWebApp owner: a-developer@example.com".
  • Click Create Channel Action to save.
TestWebApp owner a-developer@example.com

Now create a second channel action for the Slack message:

  • Select Webhook for the action type.
  • Fill in your Slack webhook URL. Slack uses POST, so Method is ok.
  • Under Headers, add a "Content-type" header with the value "application/json".
  • Use the default Body template.

Click Continue to go to the Channel section:

  • No channel yet, click Continue to go to the Name section.
  • Name this action "TestWebApp Slack #test-web-app" and save.

Note: You can send alerts to any service that accepts incoming webhooks. For Slack’s docs, see Incoming webhooks for Slack.

Create a Channel

Now that you have some actions, create a channel so you can send both types of alerts at the same time:

  • From the Monitors tab, click Channels and then Create Channel.
  • Give this channel a meaningful name, like "TestWebApp Team".
  • Under Channel Actions, add the two actions you created.
  • You don't have a monitor yet, so click Create Channel to save.
TestWebApp channel configuration with two actions

Create a Monitor

With that setup done, later when you create a monitor you only need to select the appropriate channel. For this example, presume you have a worksheet that shows errors from TestWebApp. (You can create monitors from landing pages too.)

You want to trigger an alert when there are more than 50 errors in the previous 10 minutes, so create a monitor with that condition:

  • In the worksheet, click the More button More button and select Create a Monitor.

  • You want to compare the number of events to a static value, so choose a Count monitor.

  • Click the green options to set your desired condition: greater than, 50, 10, and minutes.

For more about condition options, see Monitor Notification Options.

Click Continue to go to the Notification section.

  • This is a basic count notification, so choose Don't group.
  • But it is urgent, so select Important.
  • For Notification Delivery, select the TestWebApp Team channel. (You could add others as well.)

Click Continue to go to the Name section.

  • Give your monitor a meaningful name, like "TestWebApp errors".
  • Click Create Monitor to save.

Your new monitor displays in the Monitors tab:

TestWebApp errors, the number of Events is greater than 50 during the last 10 minutes

Create a New Channel While Creating a Monitor

You may also create a new channel at the same time you are creating a monitor. In the Notification section of the Create a Monitor page, select Create New Channel at the bottom of the list of channels. Then complete your monitor configuration as above.

Your new channel won’t have any actions, so return to the Monitors tab and create a new channel action. Then add it to your channel.