How To: Datadog Monitor Integration with HeyOnCall

While HeyOnCall can provide a great first line basic alerting and uptime monitoring, many of our users wish to integrate HeyOnCall with other APM solutions that can provide deeper insight and more nuanced analytics for complex applications. HeyOnCall can integrate with most of them using our simple, flexible HTTP-based API. This guide covers how to integrate HeyOnCall with Datadog using their provided webhooks integration so you can alert the correct responsible individual directly from the Datadog monitors you already have set up.

Set up HeyOnCall

First, make sure you have a HeyOnCall API key you wish to use for Datadog. To find your API keys go to your organization's dashboard and click the API Keys link. We recommend using a new key for every service that will be connecting to our API, such as one key for Datadog and another key for your own application code.

You will need the API Key later to enter it in Datadog.

Now create a new Trigger in HeyOnCall of type Datadog. This is a special type of trigger that will parse part of the payload sent from the webhook.

Once the trigger is created, note the trigger URL HeyOnCall gives you, as you will need to enter it in Datadog.

Set up Webhook on Datadog

Now head over to Datadog and open the webhooks integration tile.

Create a new variable named HEYONCALL_API_KEY and enter the API Key you made earlier as the value, then hit save. We recommend clicking the “hide from view” checkbox.

Now hit the + New button to create a new webhook. Give it a name, in this case we are using heyoncall_general_trigger. Paste the HeyOnCall trigger URL into the URL field.

For the payload, HeyOnCall only requires that the "alert_transition": "$ALERT_TRANSITION" key/value pair be present. This is the value we use to know if the alert is triggering or has been resolved. All the other fields are simply stored in and displayed on the trigger when you get an alert. We recommend adding a minimal set of additional fields that includes the $LINK variable so the responsible individual can quickly jump to the right place in Datadog. Here is an example payload you can use:

{
    "alert_transition": "$ALERT_TRANSITION",
    "body": "$EVENT_MSG",
    "title": "$EVENT_TITLE",
    "link": "$LINK"
}

Click the “custom headers” checkbox and add the following snippet so that Datadog adds the Authorization header to the request.

{
     "Authorization": "Bearer $HEYONCALL_API_KEY"
}

The resulting webhook should look like this

Use the Webhook

In order to trigger the HeyOnCall trigger from Datadog, you simply have to include @webhook-WEBHOOK_NAME anywhere inside the message of any monitor in Datadog. Create a new monitor, or edit an existing monitor. Scroll down to the “Notify your team” section, and simply add @webhook-heyoncall_general_trigger (or whatever your trigger is called).

Since we only have one rotation in our organization, it suffices to have only one webhook set up that we use for every Datadog monitor. When a monitor becomes triggered, the currently on call individual will receive the alert. (If you have more than one on-call rotation in HeyOnCall, simply create multiple Datadog webhooks with appropriate names and use a different HeyOnCall trigger for each.)

If you wish you can scroll to the bottom of the page and click Test Notifications. This allows you to send the Alert and Alert Recovery webhooks to HeyOnCall. Note: this will alert the currently on-call individual, so make sure you either silence the trigger beforehand on HeyOnCall or set yourself as the currently on-call individual while testing.

Done

That’s it! Datadog will send a webhook to our API when your monitor fires and HeyOnCall will do the rest based on the rotations and escalation policies you have set up.

Happy Monitoring!