PagerDuty

Note

The Monitors v2 engine is currently in private preview. Contact your Observe Data Engineer to enable this feature flag. See documentation for Monitors v1.

Overview

Use monitors to send alerts to PagerDuty by configuring a Webhook channel action. Incoming events from Observe trigger a new incident for a PagerDuty service, or grouped with alerts for an existing incident.

Requirements

To configure this integration, you need a PagerDuty admin account. If your login does not have the Admin base role, contact an Admin or Account Owner in your organization.

Integration Tutorial

In PagerDuty

Integrating with a PagerDuty service

  1. From the Configuration menu, select Services.

  2. Follow the appropriate steps, depending on if you have an existing service or are creating a new service:

    • If you are creating a new service for your integration:

      • Follow the instructions at Create a New Service in the PagerDuty documentation, selecting Observe as the Integration Type in step 4.

      • When you have finished creating your service, skip to the In Observe section, below.

    • If you are adding an integration to an existing service:

      • Click the name of the service you want to add the integration to.

      • Select the Integrations tab for this service and click New Integration.

      • Continue with the instructions in this section.

  3. Enter an Integration Name in the format monitoring-tool-service-name (e.g. “Observe-K8s-Cluster”.)

  4. In the Integration Type menu, select Observe.

  5. Click Add Integration to save the integration. PagerDuty redirects you to the Integrations tab for your service.

  6. Copy your Integration Key and save it in a safe location. You need this for the next section.

    The PagerDuty Service directory (service name) Integrations page. In the Integrations tab, the Observe integration key is highlighted.

    Figure 1 - PagerDuty integration key

In Observe

Creating a Shared Action

After you have your PagerDuty integration key, return to Observe and complete the following steps:

  1. From the left rail, open the Monitors pane.

  2. Under Monitors, click Shared Actions.

  3. Click Create Shared Action.

  4. For Action Type, select PagerDuty.

  5. In the URL field, enter the following URL:

     https://events.pagerduty.com/v2/enqueue

  6. Accept the default Method of POST.

  7. In Headers, add the following header:

       Content-Type = application/json

  8. Paste the following template into the Body, replacing the default template:

{
  "dedup_key":"{{notification.uuid}}",
  "payload": {
    "summary": "{{monitor.name}}: {{notification.description}}",
    "timestamp": "{{notification.startTime}}",
    "source": "Observe Kubernetes Environment",
    "severity": "info",
    "custom_details": {
      "Observe Monitor": "{{monitor.name}}",
      "Trigger at": "{{notification.startTime}}",
      "Importance": "{{monitor.notifyWhenDetails.importance}}",
      "Trigger Condition": "{{monitor.triggerWhen}}",
      "Notification Condition": "{{monitor.notifyWhen}}",
        {{#notification.hasValues}}
        "Triggered_By": "  {{#notification.valuesWithFieldName}}  {{fieldName}}: {{#values}}{{.}}{{/values}},\n{{/notification.valuesWithFieldName}}",
        {{/notification.hasValues}}
      "Description": "{{notification.description}}",
      {{#notification.resourcesWithLinkType}}
        "{{linkType}}": "{{#instances}}{{name}}{{/instances}}",
      {{/notification.resourcesWithLinkType}}
      {{#notification.valuesByFieldName.scope}}
        "Scope": "{{.}}",
      {{/notification.valuesByFieldName.scope}}
      {{#notification.valuesByFieldName.target}}
        "Target": "{{.}}",
      {{/notification.valuesByFieldName.target}}
      "Profile": "high"
    }
  },
  "routing_key": "ROUTING_KEY",
  {{#notification.hasResources}}
    "links": [
      {{#notification.resources}}
      {
        "href": "{{url}}",
        "text": "{{name}}"
      },
      {{/notification.resources}}
      {
        "href": "#",
        "text": "End of List"
      }
    ],
  {{/notification.hasResources}}
  "event_action":{{#notification.isNew}} "trigger"{{/notification.isNew}}{{#notification.isEnded}}"resolve"{{/notification.isEnded}},
  "client": "Observe Notification Link",
  "client_url": "{{notification.url}}"
}

  1. Replace ROUTING_KEY in the above template with your routing key, the integration key that you created earlier.

Note

Note: You may customize your payload, for details see Customizing Alert Messages.

  1. Click Send Test Notification and verify no errors.

  2. You may also select non-synthetic sample data by expanding the Sample Data and Preview section and changing the dropdown from Static Sample Data. (Optional)

  3. Click Create Shared Action to save.

Auto-resolve PagerDuty Incidents

You may optionally configure an ending notification to automatically resolve a PagerDuty incident.

In Observe

To enable auto-resolving incidents, configure a new Webhook channel action:

  1. Edit an existing Shared Action, or create a new one following the steps in Creating a Shared Action, above.

  2. Update the Body template to add a new key, is_ended, in custom_details:

    "custom_details": {
  "is_ended": "{{notification.isEnded}}",
  "Observe Monitor": "{{monitor.name}}",
  "Trigger at": "{{notification.startTime}}",

    When an incident is resolved and Observe sends an ended notification, the is_ended key triggers the PagerDuty auto-resolve event rule.

  3. In the Delivery section, under Status Updates, enable Send update that notification has ended to generate a notification when the monitor condition has resolved.

In PagerDuty

  1. Navigate to the Service Directory and select the service you wish to configure.

  2. In the Service panel, select the Settings tab.

  3. Under Event Management, click Basic Service Event Rules and then New Event Rule.

  4. Under When events match these conditions, choose Custom Details from the dropdown and enter the custom_details value you configured in Observe. Example: is_ended.

  5. In the second condition dropdown, select equals and enter the value true.

    Run this rule when All of the following conditions are met, Custom Details is_ended equals true

    Figure 2 - PagerDuty event rule conditions

  6. Click Do these things to open the actions panel.

  7. Click Advanced, check Set a custom trigger/resolve action and select Always resolve an alert.

    Do these things, create an incident. One your conditions are met, what would you like to happen? Set a custom trigger / resolve action, Always resolve an alert.

    Figure 3 - PagerDuty event rule action

  8. Click Save Rule.

Test the auto-resolve rule

To test creating and resolving incidents:

  • In Observe, open your channel action configuration and click Send Test New Notification.

  • In PagerDuty, confirm a new incident was opened.

  • In Observe, select Send Test Ended Notification from the dropdown next to the test notification button.

  • In PagerDuty, confirm the incident was closed.

How to Uninstall

Follow these steps to uninstall the PagerDuty Alert Channel from Observe:

  1. From the left navigation pane, open the Other pane by clicking “…”

  2. Under Alerting, click Shared Actions.

  3. Find the action you previously created.

  4. Click “…” to the right of the Shared Action and select Delete.

  5. Confirm by clicking Delete Shared Action.