Overview

Use Monitors to send alerts to PagerDuty by configuring a Webhook channel action. Incoming events from Observe trigger a new incident for a PagerDuty service, or grouped with alerts for an existing incident.

Requirements

To configure this integration, you need a PagerDuty admin account. If your login does not have the Admin base role, contact an administrator or account owner in your organization.

Integration tutorial

Begin by creating an integration in an existing PagerDuty service, or by creating a new service and integration.

In PagerDuty, create a service and integration

Create a new service and integration, or add an integration to an existing service. After you add the integration, record the integration key.

Create a new PagerDuty service and integration

To create a new service and integration, follow the instructions at Create a Service in the PagerDuty documentation, selecting Observe in the Integrations step.

After you create a new service and integration, be sure to record your integration key.

Add an integration to an existing PagerDuty service

Perform the following steps in PagerDuty to add an Observe integration to an existing service:

1. Select Services > Service Directory.
2. Click the name of the service you want to add the integration to.
3. Select the Integrations tab for this service and click Add another integration.
4. Enter an Integration Name in the format monitoring-tool-service-name, such as Observe-K8s-Cluster.
5. In the Integration Type menu, select Observe.
6. Click Add Integration to save the integration. PagerDuty redirects you to the Integrations tab for your service.
7. Record your integration key.

Record your integration key

Go to your new or existing integration in PagerDuty and click the expand icon (). Record the integration key for the next step.

In Observe, create a shared action

After you have your PagerDuty integration key, return to Observe and complete the following steps:

1. From the left rail, open the Monitors pane.

2. Click View shared actions.

3. Click New shared action.

4. For Action Type, select PagerDuty.

5. In the URL field, enter the following URL:

   Text

    https://events.pagerduty.com/v2/enqueue

6. Accept the default Method of POST.

7. In Headers, add the following header:

   Text

      Content-Type = application/json

8. Paste the following template into the Body, replacing the default template:

{
"dedup_key":"00000000-0000-0000-0000-000000000000",
"payload": {
"summary": "example title",
"timestamp": "2024-06-17T20:52:00Z",
"source": "Observe Monitor MONITORID",
"severity": "error",
"custom_details": {
  "Observe Monitor": "Count Monitor",
  "Condition started at": "20:52 UTC, Jun 17",
  "Importance": "Error",
    "Message": "example message",
  "Description": ""
}
},
"routing_key": "REPLACE_WITH_PAGERDUTY_ROUTING_KEY",
"links": [
{
  "href": "https://102.observe-o2.com/workspace/0/alert-data",
  "text": "View Data"
}
],
"event_action": "trigger",
"client": "Observe Alert Link",
"client_url": "https://102.observe-o2.com/workspace/0/alert?alarmId=00000000-0000-0000-0000-000000000000&monitorId=0\""
}

9. Replace REPLACE_WITH_PAGERDUTY_ROUTING_KEY with your routing key, the integration key that you created earlier.

📘

Note

You may customize your payload, for details see Customize alert messages.

10. Click Send Test Notification and verify there are no errors.

11. (Optional) Select non-synthetic sample data by expanding the Sample Data and Preview section and changing the dropdown from Static Sample Data.

12. Click Create Shared Action to save.

Auto-resolve PagerDuty incidents

You may optionally configure an ending notification to automatically resolve a PagerDuty incident.

In Observe, configure a new webhook channel action

To enable auto-resolving incidents, configure a new Webhook channel action:

1. Edit an existing Shared Action, or create a new one following the steps in In Observe, create a shared action, above.
2. Update the Body template to add a new key, is_ended, in custom_details:
   Text

   "custom_details": {
     "is_ended": "{{notification.isEnded}}",
     "Observe Monitor": "{{monitor.name}}",
     "Trigger at": "{{notification.startTime}}",

   When an incident is resolved and Observe sends an ended notification, the is_ended key triggers the PagerDuty auto-resolve event rule.
3. In the Delivery section, under Status Updates, enable Send update that notification has ended to generate a notification when the monitor condition has resolved.

In PagerDuty, define a new event rule

Perform the following steps in PagerDuty to define a new rule using the action you created in Observe:

1. Navigate to the Service Directory and select the service you wish to configure.

2. In the Service panel, select the Settings tab.

3. Under Event Management, click View Event Orchestration Rules.

4. Click New Rule.

5. Select If events match certain conditions, choose Custom Details from the dropdown and enter the custom_details value you configured in Observe. Example: is_ended.

6. In the second condition dropdown, select matches and enter the value true.

7. Click Next.
8. On the Actions page, click Alert Data .
9. Select an alert severity, then click Always resolve an alert.

7. Click Save.

Test the auto-resolve rule

To test creating and resolving incidents:

• In Observe, open your channel action configuration and click Send Test New Notification.
• In PagerDuty, confirm a new incident was opened.
• In Observe, select Send Test Ended Notification from the dropdown next to the test notification button.
• In PagerDuty, confirm the incident was closed.

Uninstall the PagerDuty channel

Follow these steps to uninstall the PagerDuty Alert channel from Observe:

1. From the left navigation pane, open the Other pane by clicking "..."
2. Under Alerting, click Shared Actions.
3. Find the action you previously created.
4. Click "..." to the right of the Shared Action and select Delete.
5. Confirm by clicking Delete Shared Action.