Creating a Promote Monitor¶
Note
The Monitors v2 engine is currently in private preview. Contact your Observe Data Engineer to enable this feature flag. See documentation for Monitors v1.
Use a Promote Monitor to forward the matching events to a destination. For instance, a dataset that enriches error reports with initial conditions and correlated resources is a good subject for a Promote Monitor.
From a Worksheet¶
Start with a scenario that you want to alert on. For instance, you might have a complex query for Application Crashes that also performs lookups to your service map, relevant spans, and customer names. To create a Promote Monitor, click the ellipses button the top right of the active stage on the left side of the screen, click Create a monitor and Promote monitor to open the Monitor creation form with a preconfigured input.
Note
While Monitors v2 is in preview, only single-stage worksheets can be used for Monitors. Collapse multiple stages to a single stage before proceeding.
See Monitors Introduction for more details on alerting rules and actions.
From a Dataset¶
The number of rows in dataset can be important to monitor. For instance, an unexpected change in the Accounts reported by the Observe for Snowflake app could indicate a number of interesting problems. To create a Promote Monitor from this dataset, click the Create Monitor button at the top right of the dataset screen and select Promote monitor. This will open the Monitor creation form with a preconfigured input. Note that if you only care that the number of accounts has changed, a Count monitor might be a better choice.
From the Monitors List¶
Click New Monitor on the top right, then Promote. Select a dataset to proceed.
Monitor name¶
Name the monitor before proceeding. Monitors must have a unique name within the instance. You can prepend a name with an App name and a slash for organizational purposes.
Monitor query¶
No matter how you’ve started a Promote monitor, the flow is the same to proceed. First, review the Monitor query to ensure that it is gathering the data that you intend to monitor. Use the time selector at the top right of the preview panel. You have access to the entire set of Observe data manipulation tools: click Chart to organize the data, use filters to trim it, and add queries or formulas to enrich the monitor.
Queries and time¶
A monitor applies to a sliding window of time. As new data arrives and triggers an evaluation, the monitor query will:
Start at the time set by the stabilization delay, if configured. See Monitor query, Advanced options, Delay monitor evaluation to review or change.
Look back the amount of time set by the evaluation period. See Monitor query, Evaluate the number of rows over the last time period to review or change. For instance, a Promote monitor that has a stabilization delay of 5 minutes and a lookback of 10 minutes will continuously monitor a sliding window from 15 minutes ago to 5 minutes ago. A Promote monitor with no stabilization delay and a lookback of 120 seconds will monitor from now to two minutes ago.
Rules¶
You can construct multiple rules in a monitor, using conditional tests from the data to set a severity level. The preview panel will update in real time so you can review where your rules are matching.
Promote monitors accept rules based on the count of rows within the sliding evaluation window. Given a ten minute window, if the count matches the condition in a rule at any point the rule will trigger and an alert will be created.
To further constrain matches or set severity by grouped values in your data, click “For any group” and select a grouped value. For instance if you are grouping by an Alarm ID, select “Alarm ID”, choose “equal to” or “not equal to”, and enter an Alarm ID.
Description¶
Use the Monitor description field as a free form text entry to inform users, link runbooks, or tag monitors. You can search Monitors or alerts by the contents of this field.
Notification actions¶
Once an alert is created, notifications can happen. If no notification is configured, the alert will still be visible in monitoring logs and Alert Explorer.
Observe supports Email, Slack, PagerDuty, and generic Webhook actions. For each action, use the Conditions area to select the matching severities that will trigger this action. For instance, you might use Slack for a Warning, but PagerDuty for a Critical.
All actions can use Observe’s extended Mustache formatting to refer to data. See Customizing Alert Messages for details.
Actions can send reminders on a periodic basis; this can be useful for Slack or Email to larger teams. Click Send Reminders beneath the action to select a time frame, such as “1 day”. Mustache variables can be used to control these alternate behaviors.
Actions can send end notifications, which is frequently used to close a ticket in a receiving system such as PagerDuty or OpsGenie. Click Send an update when the monitor has stopped triggering beneath the action to enable this. Mustache variables can be used to control these alternate behaviors.
Once configured, an action can be shared with your team members as a Saved Action, by clicking Share action with team in the title row. See Shared Actions for more information.
Note
Known Issue A promote monitor built from a Resource dataset can miss the most recently updated data in some cases. Contact your Observe Data Engineer for assistance with working around this issue.