Alerting example: Channels, Channel Actions, and Monitors

The following procedures provide you with an example that illustrates how Channels and Channel Actions work with Monitors to create alerts.

Example

You want to send an alert when too many errors occur in the TestWebApp application. Alerts

should go to the owner of the application, [email protected], and a Slack channel, #test-web-app.

Use a Monitor to watch the number of errors and alert when it exceeds a threshold. You expect similar conditions for alerting in the future, so set up a Channel and Channel Actions in advance:

  • Create Channel Actions for each type of recipient, email or webhook.
  • Create a Channel containing the desired actions (recipients.)
  • Create a Monitor whose notifications send alerts to this Channel when the triggering condition occurs.

Monitor notifications may send alerts to multiple Channels, and a Channel may contain multiple actions. Channels allow you to configure alerts for a group of recipients without selecting them individually for each new monitor.

You can also create Channels when you create a Monitor. See below for more.

Creating Channel Actions

A Channel Action sends a single type of alert. For multiple alert types or different alert messages for different recipients, create a Channel Action for each.

To send an email to [email protected], use the following steps:

  1. From the left navigation panel, click Other, click Channel Actions and then New Channel Action.
  2. Under Content, select Email and add the recipient email address. You can add other email addresses as well.
  3. Keep the other fields as the default settings.
  4. For more about the message body, see Customizing Alert Messages

  5. Under Sample Data and Preview, click Send Test New Notification to send a sample email alert.
  6. Click Continue to go to the Delivery section.

    You haven’t created a Channel yet, so skip selecting one for now.

  7. Change the default Delivery Frequency to receive notifications at the desired frequency.
  8. Under Status Updates, enable Send reminders or Send update that notification has ended.
  9. Enabling Reminders and Send Updates adds additional alert messages to Monitors actively triggering, or ended.

  10. Click Continue to go to the Channel Action Name section.
  11. Give this action a name, such as, "TestWebApp owner: [email protected]ample.com".
  12. Click Create Channel Action to save.
TestWebApp owner a-developer@example.com

Figure 1 - Email notification example

Now create a second Channel Action for the Slack message:

  1. Select Webhook for the action type.
  2. Fill in your Slack Webhook URL.
  3. For the Method, select POST.
  4. Under Headers, add a "Content-type" header with the value "application/json".
  5. Use the default Body template.
  6. Under Sample Data and Preview, you can optionally send a test notification to Slack.
  7. Click Continue to go to the Delivery section.
  8. You haven't configured a Channel yet, so skip this parameter.
  9. Change the Delivery Frequency if desired.
  10. Click Continue to create the Channel Action Name section.
  11. Name this action "TestWebApp Slack #test-web-app" and click Save.

Note: You can send alerts to any service that accepts incoming Webhooks. To view Slack documentation, see Incoming Webhooks for Slack.

Creating a Channel

Create a Channel so you can send both alerts at the same time.

  1. From the left menu, click Other, then click Channels.
  2. Click New Channel.
  3. Give this Channel a meaningful name, such as "TestWebApp Team".
  4. Under Channel Actions, add the two actions you created.
  5. You don't have a monitor yet, so click Create Channel to save.
TestWebApp channel configuration with two actions

Figure 2 - Creating a Channel with two actions

Creating a Monitor

After you finish creating a Channel Action and a Channel, and you create a Monitor, you only need to select the appropriate Channel.

You want to trigger an alert when more than 50 errors occur in the previous 10 minutes, so you must create a Monitor with that condition.

  1. From the right navigation, click Other, and select Monitors.
  2. Select the Event Stream for your Test Web App.
  3. You want to compare the number of events to a static value, so choose a Count monitor.
  4. Under Trigger Conditions, set your desired condition:
    • greater than
    • 50
    • 10
    • minutes

    For more about condition options, see Monitor Notification Options.

  5. Click Continue to go to the Notification section.
  6. This is a basic count notification, so choose Don't group.
  7. You want this to be an urgent notification, so select Important.
  8. For Notification Delivery, select the TestWebApp Team channel. You can add other Channels as well.
  9. Under Delivery, add a Custom description/comments.
  10. Click Continue to go to the Monitor Name section.
  11. Give your Monitor a meaningful name, such as "TestWebApp errors".
  12. Click Create Monitor to create the monitor.

Your new Monitor displays in the Monitors list:

TestWebApp errors, the number of Events is greater than 50 during the last 10 minutes

Figure 3 - Monitors list

Creating a New Channel While Creating a Monitor

You can also create a new Channel at the same time you are creating a Monitor. In the Notification section of the Create a Monitor page, select Create New Channel at the bottom of the list of Channels. Then complete your Monitor configuration as above.

Your new Channel won’t have any Channel Actions, so return to the Channel Actions list in the left menu and create a New Channel Action. Then add it to your new Channel.

Adding a Monitor to a dataset

You can create a monitor for a dataset using the following steps:

  1. Log into Observe and click Explore.

  2. Click Datasets.

  3. Select the dataset you want to monitor.

  4. From the top of the panel, click Monitor.

  5. Add a Monitor using the steps in Creating a Monitor.