Introduction to Monitors and Alerts

Observe Monitors provide a flexible way to alert for patterns in your incoming data. Define who should receive alerts with Shared Actions, then create Monitors to watch for your desired conditions. When one occurs, Observe sends alerts to everyone (or every service) in its Shared Actions configuration. You can send alerts to any combination of email addresses and Webhook-enabled services.

What does a Monitor do?

A Monitor watches a dataset for a particular condition, such as a count of events or a specific text value. When you create a Monitor, Observe makes a new dataset based on the contents of the page and your conditions. This allows multiple Monitors from the same page to be independent of each other.

Monitors provide four options to use for the notifications:

  • Threshold Metrics - Compare the count of events to a static threshold.

  • Count - Compare a numeric value to a static threshold.

  • Text Value - Use a specific text field to monitor the dataset.

  • Promote - Promote each event in the dataset into a Notification.

The Notifications tab lists active Important notifications for alerts currently matching or exceeding the triggering conditions. Click on a notification to view the history. If an alert triggering condition continues, it remains open and active, including reminders, if configured, for up to two weeks. For more about Important and Informational notifications, see Monitor Notification Options.

Using Monitor Templates

Observe provides easy-to-use templates to create monitors for events. By default, you can select from two templates:

  • (TEMPLATE)Failed Job - monitor preset dataset for any failed jobs within Observe. The monitor contains a preset dataset, Builds, with the following OPAL statements:

	filter result="FAILURE"
            
make_col kind:strcat("Jenkins job ", label, " failed on node ", datacenter, "/", host)  
make_col description:strcat("Job URL: ", link, "\n Logfile: ", filename)

  • (TEMPLATE)Long Job > 100s - monitor Jenkins jobs for jobs that last more than 100 seconds. This monitor contains a preset dataset, Builds, with the following OPAL statements:

	filter buildSecs>100
            
	make_col kind:strcat("Jenkins job ", label, " ran long on ", 	datacenter, "/", host)  
	make_col description:strcat("Job URL: ", link, "\n Logfile: ", filename)

To use the templates, configure the desired template and then save it as a new monitor.

When you install the AWS app on Observe, the app also installs templates for the selected AWS integrations. To view the list of AWS Templates, perform the following steps:

  1. From the left navigation panel, click the Other icon More icon, and select Monitors.

  2. Click Templates to view the list of installed templates.

Note

The template list varies based on the installed AWS services on your Observe instance.

To use the AWS templates, configure the desired template and then save it as a new monitor.

When you install the Host Monitoring app, the app also installs templates for the following services:

  • Fluentbit

  • Telegraf

To use the Host Monitoring templates, configure the desired template and then save it as a new monitor.

You can sort or group the templates using the Sort icon Sort or Group icon located next to New Monitor.

Monitor Overview

Figure 2 - Sort by

Monitor Overview

Figure 3 - Group by

Creating a new Monitor

To create a new Monitor in Observe, use the following steps:

  1. Log into Observe and click the Other icon More icon to display the menu, and then choose Monitors from the list.

  2. On the Monitors page, click New Monitor.

  3. From the Select your monitor type panel, select the type of monitor you want to create in Observe:

    • Threshold metric - create a monitor to alert you on the values of a metric when the values cross a user-defined threshold over a period of time.

    • Count - compare the count of Resources to a static threshold.

    • Text Value - monitor a dataset for the appearance of a specific text value.

    • Promotion - promote each resource in a dataset into a Notification.

What are Shared Actions?

A Shared Action consists of a set of alert recipients. A Shared Action specifies the type of alert, such as email or Webhook, the receiving location, and the template for the message or payload.

A Shared Action can be used for multiple Monitors. When a Monitor triggers an alert, emails or Webhook requests go to all recipients in the configured Shared Action. You don’t have to select individual recipients each time you create a new Monitor.

You may also enable optional status updates for a Shared Action and allow a Monitor to send reminder notifications that an alert is still active or that it has ended.

Configuring Alerts

Set up Shared Actions, and then create Monitors from any Dataset. Each Shared Action can have a custom alert message to send your HTML to an email recipient or JSON in a Webhook request.

For an example, see this page: Alerting Example: Channels, Channel Actions, and Monitors.

For more about trigger conditions, see Monitor Notification Options