PagerDuty

Overview

Use monitors to send alerts to PagerDuty by configuring a webhook channel action. Incoming events from Observe trigger a new incident for a PagerDuty service, or are grouped with alerts for an existing incident.

Requirements

To configure this integration, you will need the following:

• A PagerDuty admin login. If your login does not have the “Admin” base role, contact an Admin or Account Owner in your organization.

Integration Walkthrough

In PagerDuty

Integrate with a PagerDuty service

1. From the Configuration menu, select Services.

2. Follow the appropriate steps, depending on if you have an existing service or are creating a new service:

   • If you are creating a new service for your integration:

     • Follow the instructions at Create a New Service in the PagerDuty documentation, selecting Observe as the Integration Type in step 4.

     • When you have finished creating your service, skip to the In Observe section, below.

   • If you are adding an integration to an existing service:

     • Click the name of the service you want to add the integration to.

     • Select the Integrations tab for this service and click New Integration.

     • Continue with the instructions in this section.

3. Enter an Integration Name in the format monitoring-tool-service-name (e.g. “Observe-K8s-Cluster”.)

4. In the Integration Type menu, select Observe.

5. Click Add Integration to save the integration. You are redirected to the Integrations tab for your service.

6. Copy your Integration Key and save it in a safe location. You will need this for the next section.

In Observe

Create a channel action

After you have your PagerDuty integration key, return to Observe and complete the following steps:

1. From the left rail, open the Other pane by clicking “…”

2. Under Alerting, click Channel Actions.

3. Click New Channel Action.

4. Configure a new channel action:

   In the Content (section 1):

   1. For Channel Action Type, select Webhook.

   2. In URL, enter the following URL:

      https://events.pagerduty.com/v2/enqueue
      

   3. Accept the default Method of POST

   4. In Headers, add the following header:

      Content-Type = application/json
      

   5. Paste the following template into Body, replacing the default template:

      {
        "payload": {
          "summary": "{{monitor.name}}: {{notification.description}}",
          "timestamp": "{{notification.startTime}}",
          "source": "Observe Kubernetes Environment",
          "severity": "info",
          "custom_details": {
            "Observe Monitor": "{{monitor.name}}",
            "Trigger at": "{{notification.startTime}}",
            "Importance": "{{monitor.notifyWhenDetails.importance}}",
            "Trigger Condition": "{{monitor.triggerWhen}}",
            "Notification Condition": "{{monitor.notifyWhen}}",
              {{#notification.hasValues}}
              "Triggered_By": "  {{#notification.valuesWithFieldName}}  {{fieldName}}: {{#values}}{{.}}{{/values}},\n{{/notification.valuesWithFieldName}}",
              {{/notification.hasValues}}
            "Description": "{{notification.description}}",
            {{#notification.resourcesWithLinkType}}
              "{{linkType}}": "{{#instances}}{{name}}{{/instances}}",
            {{/notification.resourcesWithLinkType}}
            {{#notification.valuesByFieldName.scope}}
              "Scope": "{{.}}",
            {{/notification.valuesByFieldName.scope}}
            {{#notification.valuesByFieldName.target}}
              "Target": "{{.}}",
            {{/notification.valuesByFieldName.target}}
            "Profile": "high"
          }
        },
        "routing_key": "ROUTING_KEY",
        {{#notification.hasResources}}
          "links": [
            {{#notification.resources}}
            {
              "href": "{{url}}",
              "text": "{{name}}"
            },
            {{/notification.resources}}
            {
              "href": "#",
              "text": "End of List"
            }
          ],
        {{/notification.hasResources}}
        "event_action":{{#notification.isNew}} "trigger"{{/notification.isNew}}{{#notification.isEnded}}"resolve"{{/notification.isEnded}},
        "client": "Observe Notification Link",
        "client_url": "{{notification.url}}"
      }
      

   6. Replace ROUTING_KEY in the above template with your routing key.

      Note

      Note: You may customize your payload, for details see Customizing alert messages

5. Click Send Test Notification and verify no errors are returned. You may also select non-synthetic sample data by expanding the Sample Data and Preview section and changing the dropdown from Static Sample Data. (Optional.)

In Delivery (section 2):

1. In Send Notifications From The Following Channels, select a channel or create a new channel. (Optional.)

2. In Delivery Frequency, update the frequency if different rate limiting is desired. (Optional.)

In Channel Action Name (section 3):

1. In Channel Action Name, give this action a unique name.

2. Click Create Channel Action to save.

How to Uninstall

Follow these steps to uninstall the PagerDuty alert channel from Observe:

1. From the left rail, open the Other pane by clicking “…”

2. Under Alerting, click Channel Actions.

3. Find the action you previously created

4. Click “…” to the right of the Channel Action and select Delete.

5. Confirm by clicking Delete Channel Action.

Support

If you need help with this integration, please contact [email protected]