Datadog Integration

Integrate Shoreline's auto-remediation with your existing Datadog monitors and dashboards.

Integrating Shoreline with Datadog takes just a few minutes and requires four major steps:

For existing customers: Please enter your Shoreline Cluster URL in the field below to tailor the content to your environment.

Create a Webhook

Datadog integrates with Shoreline via a webhook widget, which makes a secure POST request to Shoreline when a Datadog monitor enters an alert state. The monitor acts as an auto-generated Shoreline Alarm.

  1. In the Datadog app, click the Integrations button on the left.

  2. Enter webhook in the Search for an integration text box.

  3. Click the Webhooks Integration card to open the configuration window.

  4. Click the button.

  5. Enter undefined in the Name field.

    Alternatively, if you're using a different webhook name, enter it both in the Name field and in the field below for future reference.

  6. In the URL field, enter the following webhook endpoint.

    datadog Webhook URL
    https://acme.us.api.shoreline-prod.io/v1/integrations/datadog/webhook
  7. In the Payload field paste the following JSON.

    {
      "last_updated": "$LAST_UPDATED",
      "event_type": "$EVENT_TYPE",
      "title": "$EVENT_TITLE",
      "date": "$DATE",
      "id": "$ID",
      "alert_metric": "$ALERT_METRIC",
      "alert_query": "$ALERT_QUERY",
      "alert_scope": "$ALERT_SCOPE",
      "alert_status": "$ALERT_STATUS",
      "alert_type": "$ALERT_TYPE",
      "alert_id": "$ALERT_ID",
      "alert_URL": "$LINK",
      "alert_title": "$ALERT_TITLE",
      "event_tags": "$TAGS",
      "event_title": "$EVENT_TITLE",
      "host": "$HOSTNAME",
      "customer_id": "$ORG_ID",
      "customer_name": "$ORG_NAME"
    }
    
  8. Tick the Custom Headers checkbox.

  9. (Optional) Enter the Datadog auth token provided by Shoreline in the field below.

  10. Enter the following in the Custom Headers textbox.

    {
      "Authorization": "Bearer undefined",
      "Idempotency-Key": "0"
    }
    
  11. Finally, click the button.

    Your Shoreline integration is now configured and ready for use by creating a new Datadog Monitor.

Add Shoreline to a Monitor

Add Shoreline to a Dashboard

Add the Shoreline widget to a new or existing dashboard to start using Shoreline with your configured monitor.

Modify a Dashboard

  1. Open your desired Datadog dashboard.

  2. Click the large Add widgets button.

  3. Within the widgets banner, scroll to and click on the Shoreline widget. This opens the Custom widget Editor dialog.

  4. Scroll down to Widget options and enter the appropriate values in each field based on your full Shoreline cluster URL.

    For example, if your Shoreline cluster URL is https://acme.us.app.shoreline-prod.io then you're using the shoreline-prod cluster in the us region and your customer name is acme.

    FieldValueExample
    RegionSHORELINE_CLUSTER_REGIONus
    Customer nameSHORELINE_CUSTOMER_NAMEacme
    Cluster nameSHORELINE_CLUSTER_NAMEshoreline-prod.io
  5. (Optional) Enter a query in the Monitors query field to filter the monitors displayed in this Shoreline widget.

    You can filter by the monitor's status or a specific tag using the <type>:<value> format. Consider the following examples.

    ValueDescription
    status:alertDisplays alerting monitors
    status:okDisplays OK monitors
    tag:shorelineDisplays monitors with a shoreline tag
    tag:"env:prod"Displays monitors with a env:prod tag
  6. (Optional) Enter a title in the Widget title field.

  7. Click the button to save your changes and return to your dashboard.

You should now see a list of existing monitors in the widget panel.

dashboard/monitors

Configuration

Remediation Panel

The Shoreline remediation panel allows you to:

  1. Open your Datadog dashboard to access the Shoreline remediation panel.
  2. Within the Shoreline widget, find the Datadog monitor you want to configure.
  3. Click the Create button under the REMEDIATION column. This opens the Shoreline remediation panel.

Debugging

Shoreline's WebCLI is integrated in the remediation panel, allowing you to execute Op commands to debug your entire fleet.

  1. To get started, click the button at the top of the remediation panel.

    The Debug panel initially displays a single WebCLI cell.

    webcli
  2. Click inside the Op cell, which emulates a normal Op CLI terminal window.

  3. Type a valid Op command and press the Enter key to execute the command.

    For example, using the most basic hosts command returns all active host Resources.

    op>
    hosts
    NAME
    i-0fe11d2a2fe02481b
    i-0f9af94660f63d7a8
    i-0f727279214d2dea8
  4. Use the pods Op command to list all pods.

    op>
    pods
    NAME
    kube-system.aws-node-termination-handler-mbngq
    java-bookstore.java-bookstore-wz69z
    monitoring.cust2-prometheus-0
    kube-system.aws-node-nwxn5
    monitoring.cust2-alertmanager-1
    kube-system.coredns-5c97f79574-67flz
    test3-cust.shoreline-w22mc
    kube-system.kube-proxy-n49rb
    monitoring.cust2-node-exporter-pmxhc
    monitoring.cust2-kube-state-metrics-59bcf64f5f-mccz4
    monitoring.cust2-alertmanager-0

    The pods servicing the bookstore app have a tag of namespace="java-bookstore". Op uses the familiar bash syntax. The Op pipe operator passes the result of the previous command to the following command, just like the pipe operator in the Unix shell environment.

  5. Enter the following Resource query to return only bookstore pods.

    op>
    pods | namespace="java-bookstore"
    NAME
    java-bookstore.java-bookstore-wz69z
    java-bookstore.java-bookstore-2s784
    java-bookstore.java-bookstore-pbp2x
  6. Create a named Resource to simplify future commands, so that typing bookstore invokes the full hosts | pods | namespace="java-bookstore" Resource query.

    op>
    resource bookstore = hosts | pods | namespace="java-bookstore"
    Defined resource 'bookstore'
    
    op>
    bookstore
    NAME
    java-bookstore.java-bookstore-wz69z
    java-bookstore.java-bookstore-2s784
    java-bookstore.java-bookstore-pbp2x   
    
  7. Create a Metric shorthand called pod_cpu.

    op>
    metric pod_cpu = pod_cpu_usage
    Defined metric 'pod_cpu'
    

    Shoreline also supports a feature called Dynamic Filters. With simple resource queries you can retrieve Resources based on type or tag.

  8. Try using a Dynamic Filter to show all host Resources with over 15% CPU usage.

    op>
    hosts | filter(cpu_usage > 15)
    dynamic-filter
  9. Execute the top Linux command on the bookstore Resources.

    op>
    bookstore | `top -n 1 -b`

    The output dialogs show any stdout, stderr, and exit status:

    STDOUT                                                          STATUS: 0
    top - 19:01:39 up 16 days, 23:11, 0 users, load average: 1.86, 1.25, 0.89
    Tasks: 7 total, 1 running, 3 sleeping, 0 stopped, 3 zombie
    %Cpu(s): 17.9 us, 8.9 sy, 0.0 ni, 73.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
    MiB Mem : 15716.5 total, 860.6 free, 2035.1 used, 12820.8 buff/cache
    MiB Swap: 0.0 total, 0.0 free, 0.0 used. 13798.5 avail Mem
    
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    1 root 20 0 2426368 45748 27796 S 0.0 0.3 1:19.00 java
    274 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sleep
    286 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sleep
    300 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sleep
    301 root 20 0 5488 3296 3016 S 0.0 0.0 0:00.00 bash
    309 root 20 0 4240 804 720 S 0.0 0.0 0:00.00 timeout
    310 root 20 0 9672 3452 3060 R 0.0 0.0 0:00.00 top
    

Using Actions

View Shoreline Events

With Shoreline now fully integrated and an Action associated with a Datadog monitor, you'll immediately start seeing Shoreline auto-remediation events in the Datadog Events streams, including directly on the associated monitor's status page.

The Shoreline remediation event provides information on the event type, the name of the Datadog monitor, the executed Op statement, any stdout or stderr output, and the returned exit status or variable value.

For example, the following event was generated by the bookstore_monitor and executed the jvm_remediator Action on java-bookstore containers within a specific host instance. The output shows a heap dump, thread dump, and garbage collection/heap info was created.

Shoreline bot executed an Op statement to remediate the Datadog monitor, "bookstore_monitor" :
"host | instance_id="i-05ddda1989b647d3b" | pod | pod_name="java-bookstore" | container | container_name="java-bookstore" | jvm_remediator"
#monitor_id:41486260

Stdout: using jcmd to take heap dump.
using jstack to take thread dump.
using jstat to get gc info.
using jmap to get heap stats.

ExitStatus: 0
Stderr: