Introduction to Monitors and Alerts¶

Observe Monitors provide a flexible way to alert for patterns in your incoming data. Define who should receive alerts with Shared Actions, then create Monitors to watch for your desired conditions. When one occurs, Observe sends alerts to everyone (or every service) in its Shared Actions configuration. You can send alerts to any combination of email addresses and Webhook-enabled services.

What does a Monitor do?¶

A Monitor watches a dataset for a particular condition, such as a count of events or a specific text value. When you create a Monitor, Observe makes a new dataset based on the contents of the page and your conditions. This allows multiple Monitors from the same page to be independent of each other.

Monitors provide four options to use for the notifications:

Threshold Metrics - Compare the count of events to a static threshold.
Count - Compare a numeric value to a static threshold.
Promote - Promote each event in the dataset into a Notification.
Threshold log - Alert on values of a log count when exceeding a predefined threshold over a period of time.
Text Value - Use a specific text field to monitor the dataset.

Monitors Overview¶

When you select Monitors from the More menu on the main login page, Observe displays a list of existing Monitors configured on your instance.

Figure 1 - List of Configured Monitors

The page displays the following columns of information:

Status - the current status of a Monitor.
Name - the name configured for the Monitor.
Kind - the type of configured Monitor such as Threshold, Count, Text, or Promote.
Created By - the name of the person who configured the Monitor.
App - the app associated with the Monitor.
Options - select from
- Enabled - enable or disable the Monitor.
- Duplicate - add a duplicate Monitor.
- Export Terraform - export the Monitor configuration in Terraform.
- Delete - remove the Monitor from the Observe instance.

From the All Statuses menu, you can select from the following options:

Figure 2 - Select from a List of Monitor Statuses

All Statuses - a list of all Monitors in any status.
Triggering - Monitors in the process of triggering an alert.
Error - Monitors in an error state.
Disabled - disabled Monitors.
Monitoring - Monitors actively watching for trigger events.
Creating - Monitors in the process of creation after configuring a Monitor.

You can sort or group the Monitors by clicking the Sort icon, and displaying the options for sorting or grouping them.

Figure 3 - Select Sort By or Group By

To add a Monitor, click +Create Monitor.

Figure 4 - Select the Type of Monitor

Select one of the following types of Monitors to configure for Alerts:

Threshold metric - Create a Monitor to alert on metrics values when they cross a defined threshold over a period of time.
Count - Compare the Resource count to a static threshold.
Promotion - Promote each Resource, Event, or Interval in a dataset into a Notification.
Threshold Log - Create an Alert on values of a log count when they cross a user-defined threshold over a period of time.
Text Value - Create an Alert when a specific text value appears in a field.

Using Monitor Templates¶

Observe provides easy-to-use templates to create monitors for events. By default, you can select from two templates:

(TEMPLATE)Failed Job - monitor preset dataset for any failed jobs within Observe. The monitor contains a preset dataset, Builds, with the following OPAL statements:

filter result="FAILURE"
            
make_col kind:strcat("Jenkins job ", label, " failed on node ", datacenter, "/", host)  
make_col description:strcat("Job URL: ", link, "\n Logfile: ", filename)

(TEMPLATE)Long Job > 100s - monitor Jenkins jobs for jobs that last more than 100 seconds. This monitor contains a preset dataset, Builds, with the following OPAL statements:

filter buildSecs>100
            
make_col kind:strcat("Jenkins job ", label, " ran long on ", datacenter, "/", host)  
make_col description:strcat("Job URL: ", link, "\n Logfile: ", filename)

To use the templates, configure the desired template and then save it as a new Monitor.

Figure 5 - List of Available Monitor Templates

The Templates page displays the following columns:

Status - the current status of the Template
Name - the name of the Template
Kind - the type of Monitor Template
Created by - if related to an app, the name of the person who installed the app.
Package - the associated package for the Template
App - the app name associated with the Template
Options - select from
- Duplicate - add a duplicate Monitor.
- Export Terraform - export the Monitor configuration in Terraform.

You can sort or group the templates using the Sort icon located next to Create Monitor.

When you install an app on Observe, the app also installs templates for the selected integrations. To view the list of app Templates, perform the following steps:

From the left navigation panel, click the Other icon , and select Monitors.
Click Templates to view the list of installed templates.

Note

The template list varies based on the installed app services on your Observe instance.

To use the installed app Templates, configure the desired template and then save it as a new monitor.

Creating a new Monitor¶

To create a new Monitor in Observe, use the following steps:

Log into Observe and click the Other icon to display the menu, and then choose Monitors from the list.
On the Monitors page, click New Monitor.
From the Select your monitor type panel, select the type of monitor you want to create in Observe:
- Threshold metric - create a monitor to alert you on the values of a metric when the values cross a user-defined threshold over a period of time.
- Count - compare the count of Resources to a static threshold.
- Text Value - monitor a dataset for the appearance of a specific text value.
- Promotion - promote each resource in a dataset into a Notification.

What are Shared Actions?¶

A Shared Action consists of a set of alert recipients. A Shared Action specifies the type of alert, such as email or Webhook, the receiving location, and the template for the message or payload.

A Shared Action can be used for multiple Monitors. When a Monitor triggers an alert, emails or Webhook requests go to all recipients in the configured Shared Action. You don’t have to select individual recipients each time you create a new Monitor.

You may also enable optional status updates for a Shared Action and allow a Monitor to send reminder notifications that an alert is still active or that it has ended.

Configuring Alerts¶

Set up Shared Actions, and then create Monitors from any Dataset. Each Shared Action can have a custom alert message to send your HTML to an email recipient or JSON in a Webhook request.

For an example, see this page: Alerting Example: Channels, Channel Actions, and Monitors.

For more about trigger conditions, see Monitor Notification Options