Events

New
An Event record is generated whenever an Alarm, Action, Bot, or other critical Shoreline object triggers.

Events give you a powerful way to evaluate the status of every Alarm, Action, Bot, and Resource within your infrastructure. Every Event is associated with one or more Resources, allowing you to execute a multitude of Op commands and functions to further filter the results. You'll also see every step within each individual Event, giving you the precise timestamp, action performed, and relevant descriptions.

Alarm Events

An Alarm Event is created every time an Alarm triggers, resolves, or ends.

op>
events | type="alarm"
RESOURCE_ID | RESOURCE_NAME                        | RESOURCE_TYPE | ALARM_NAME                           | STATUS    | STEP_TYPE   | TIMESTAMP                 | DESCRIPTION
1           | i-0192cd8c4f55219c2                  | HOST          | JSON_Metric_auto_name                | triggered |             |                           | JSON auto description
            |                                      |               |                                      |           | ALARM_FIRE  | 2021-08-31T14:47:20-07:00 | JSON auto Raise Description Template
            |                                      |               |                                      |           | ALARM_CLEAR | 1969-12-31T16:00:00-08:00 | JSON auto Resolve Description Template
99          | test2-cust.shoreline-l2jf7           | POD           | simple_pod_alarm                     | resolved  |             |                           | Fire when pod cpu usage > 10 in 15 out of 30 seconds
            |                                      |               |                                      |           | ALARM_FIRE  | 2021-08-30T11:05:44-07:00 | More than 10% pod cpu usage short template
            |                                      |               |                                      |           | ALARM_CLEAR | 2021-08-30T11:06:02-07:00 | Less than 10% pod cpu usage short template
99          | test2-cust.shoreline-l2jf7           | POD           | mixed_metric_linux_cmd_alarm         | resolved  |             |                           | Fire when pod cpu usage > 10 in 15 out of 30 seconds
            |                                      |               |                                      |           | ALARM_FIRE  | 2021-08-30T11:05:44-07:00 | More than 10% pod cpu usage short template
            |                                      |               |                                      |           | ALARM_CLEAR | 2021-08-30T11:06:02-07:00 | Less than 10% pod cpu usage short template
99          | test2-cust.shoreline-l2jf7           | POD           | linux_cmd_duration_res_env_var_alarm | resolved  |             |                           | Fire when pod cpu usage > 10 in 15 out of 30 seconds
            |                                      |               |                                      |           | ALARM_FIRE  | 2021-08-30T11:05:44-07:00 | More than 10% pod cpu usage short template
            |                                      |               |                                      |           | ALARM_CLEAR | 2021-08-30T11:06:02-07:00 | test file succeed over 5 times short template
10          | i-00f4a29cc5a8716d1                  | HOST          | always_fire_alarm                    | resolved  |             |                           |
            |                                      |               |                                      |           | ALARM_FIRE  | 2021-08-30T11:05:43-07:00 | CPU usage over 50%
            |                                      |               |                                      |           | ALARM_CLEAR | 2021-08-30T11:05:52-07:00 | CPU usage below 50%

Action Events

An Action Event is created every time an Action executes, completes, or fails.

op>
events | type="action"
RESOURCE_ID | RESOURCE_NAME                        | RESOURCE_TYPE | ACTION_NAME                           | BOT_NAME                           | STATUS    | STEP_TYPE    | TIMESTAMP
17          | test2-cust.shoreline-b2jtz.shoreline | CONTAINER     | simple_pod_action                     | simple_pod_bot                     | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T05:32:58-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T05:32:59-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T05:33:00-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
17          | test2-cust.shoreline-b2jtz.shoreline | CONTAINER     | mixed_metric_linux_cmd_action         | mixed_metric_linux_cmd_bot         | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T05:32:58-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T05:32:59-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T05:33:00-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
17          | test2-cust.shoreline-b2jtz.shoreline | CONTAINER     | linux_cmd_duration_res_env_var_action | linux_cmd_duration_res_env_var_bot | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T05:32:58-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T05:32:59-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T05:33:00-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00

Bot Events

A Bot Event is created every time a Bot executes, completes, or fails.

op>
events | type="bot"
RESOURCE_ID | RESOURCE_NAME                        | RESOURCE_TYPE | ALARM_NAME                           | ACTION_NAME                           | BOT_NAME                           | STATUS    | STEP_TYPE    | TIMESTAMP
9           | i-0f3ebe788cf2268d9                  | HOST          | always_fire_alarm                    | host_top_action                       | host_top_bot                       | completed |              |
            |                                      |               |                                      |                                       |                                    |           | ALARM_FIRE   | 2021-08-31T14:45:23-07:00
            |                                      |               |                                      |                                       |                                    |           | BOT_DISPATCH | 2021-08-31T14:45:24-07:00
            |                                      |               |                                      |                                       |                                    |           | BOT_END      | 2021-08-31T14:45:25-07:00
            |                                      |               |                                      |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
107         | test2-cust.shoreline-kfmwx           | POD           | mixed_metric_linux_cmd_alarm         | mixed_metric_linux_cmd_action         | mixed_metric_linux_cmd_bot         | completed |              |
            |                                      |               |                                      |                                       |                                    |           | ALARM_FIRE   | 2021-08-31T14:45:21-07:00
            |                                      |               |                                      |                                       |                                    |           | BOT_DISPATCH | 2021-08-31T14:45:22-07:00
            |                                      |               |                                      |                                       |                                    |           | BOT_END      | 2021-08-31T14:45:22-07:00
            |                                      |               |                                      |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
107         | test2-cust.shoreline-kfmwx           | POD           | linux_cmd_duration_res_env_var_alarm | linux_cmd_duration_res_env_var_action | linux_cmd_duration_res_env_var_bot | completed |              |
            |                                      |               |                                      |                                       |                                    |           | ALARM_FIRE   | 2021-08-31T14:45:21-07:00
            |                                      |               |                                      |                                       |                                    |           | BOT_DISPATCH | 2021-08-31T14:45:22-07:00
            |                                      |               |                                      |                                       |                                    |           | BOT_END      | 2021-08-31T14:45:22-07:00
            |                                      |               |                                      |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00

Resource Events

A Resource Event is created every time a Resource is registered or de-registered.

op>
events | type="resource"
RESOURCE_ID | RESOURCE_NAME                                                     | RESOURCE_TYPE | STATUS        | STEP_TYPE           | TIMESTAMP
1           | i-03a2b4958db5953c9                                               | HOST          | de-registered |                     |
            |                                                                   |               |               | RESOURCE_REGISTER   | 2021-09-01T05:29:39-07:00
            |                                                                   |               |               | RESOURCE_DEREGISTER | 2021-09-01T09:33:40-07:00
10          | monitoring.test5c-prometheus-1.prometheus-server                  | CONTAINER     | normal        |                     |
            |                                                                   |               |               | RESOURCE_REGISTER   | 2021-09-01T09:34:14-07:00
            |                                                                   |               |               | RESOURCE_DEREGISTER | 1969-12-31T16:00:00-08:00
100         | monitoring.test5c-prometheus-0.prometheus-server                  | CONTAINER     | normal        |                     |
            |                                                                   |               |               | RESOURCE_REGISTER   | 2021-09-01T09:34:33-07:00
            |                                                                   |               |               | RESOURCE_DEREGISTER | 1969-12-31T16:00:00-08:00
101         | kube-system.kube-proxy-p8nms.kube-proxy                           | CONTAINER     | normal        |                     |
            |                                                                   |               |               | RESOURCE_REGISTER   | 2021-09-01T09:34:33-07:00
            |                                                                   |               |               | RESOURCE_DEREGISTER | 1969-12-31T16:00:00-08:00
102         | test5-cust.shoreline-k4bn2.shoreline                              | CONTAINER     | normal        |                     |
            |                                                                   |               |               | RESOURCE_REGISTER   | 2021-09-01T09:34:33-07:00
            |                                                                   |               |               | RESOURCE_DEREGISTER | 1969-12-31T16:00:00-08:00
103         | cert-manager.cert-manager-cainjector-685b87b86-q8t72.cert-manager | CONTAINER     | normal        |                     |
            |                                                                   |               |               | RESOURCE_REGISTER   | 2021-09-01T09:34:33-07:00
            |                                                                   |               |               | RESOURCE_DEREGISTER | 1969-12-31T16:00:00-08:00

Filter Events

Count

Group all Events by current type, then sum by Event stage for each affected Resource.

For example, the following command gets all Alarm Events:

op>
events | type="alarm" | count
GROUP     | RESOURCE_ID | RESOURCE_TYPE | RESOURCE_NAME                        | AZ         | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all |             |               |                                      |            | ALARMS     | 54    | 54      | 0      | 54
6         |             |               |                                      |            | ALARMS     | 16    | 16      | 0      | 16
          | 6           | POD           | test5-cust.shoreline-nvbdm           | us-west-2a |            |       |         |        |
35        |             |               |                                      |            | ALARMS     | 1     | 1       | 0      | 1
          | 35          | CONTAINER     | test5-cust.shoreline-729bc.shoreline | us-west-2b |            |       |         |        |
30        |             |               |                                      |            | ALARMS     | 16    | 16      | 0      | 16
          | 30          | POD           | test5-cust.shoreline-729bc           | us-west-2b |            |       |         |        |
3         |             |               |                                      |            | ALARMS     | 1     | 1       | 0      | 1
          | 3           | HOST          | i-0d991c45ccdb96db4                  | us-west-2b |            |       |         |        |
26        |             |               |                                      |            | ALARMS     | 1     | 1       | 0      | 1
          | 26          | CONTAINER     | test5-cust.shoreline-wtjfd.shoreline | us-west-2c |            |       |         |        |
20        |             |               |                                      |            | ALARMS     | 16    | 16      | 0      | 16
          | 20          | POD           | test5-cust.shoreline-wtjfd           | us-west-2c |            |       |         |        |
12        |             |               |                                      |            | ALARMS     | 3     | 3       | 0      | 3
          | 12          | CONTAINER     | test5-cust.shoreline-nvbdm.shoreline | us-west-2a |            |       |         |        |

The result shows how many Events were fired, cleared, active, including totals. The first group_all column shows Event totals across all returned Resources.

Limit

The limit command restricts the number of returned Events.

op>
events | type="action" | limit = 2
RESOURCE_ID | RESOURCE_NAME                        | RESOURCE_TYPE | ACTION_NAME                           | BOT_NAME                           | STATUS    | STEP_TYPE    | TIMESTAMP
35          | test5-cust.shoreline-729bc.shoreline | CONTAINER     | mixed_metric_linux_cmd_action         | mixed_metric_linux_cmd_bot         | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T05:47:23-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T05:47:25-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T05:47:26-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
35          | test5-cust.shoreline-729bc.shoreline | CONTAINER     | linux_cmd_duration_res_env_var_action | linux_cmd_duration_res_env_var_bot | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T05:47:23-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T05:47:25-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T05:47:26-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00

Status

Use status to filter Events based on their status property. The valid values depend on the Event Type.

Event TypeStatus
Alarmtriggered
Alarmresolved
Alarmcanceled
Actionexecuting
Actioncompleted
Actionfailed
Botexecuting
Botcompleted
Botfailed
Resourcenormal
Resourcealarming
Resourcede-registered

For example, this command gets the first five Alarm Events with a resolved status:

op>
events | type="alarm" | status="resolved" | limit=5
RESOURCE_ID | RESOURCE_NAME              | RESOURCE_TYPE | ALARM_NAME                           | STATUS   | STEP_TYPE   | TIMESTAMP                 | DESCRIPTION
4           | test4-cust.shoreline-9qm7j | POD           | simple_pod_alarm                     | resolved |             |                           | Fire when pod cpu usage > 10 in 15 out of 30 seconds
            |                            |               |                                      |          | ALARM_FIRE  | 2021-09-01T15:44:07-07:00 | More than 10% pod cpu usage short template
            |                            |               |                                      |          | ALARM_CLEAR | 2021-09-01T15:44:36-07:00 | Less than 10% pod cpu usage short template
4           | test4-cust.shoreline-9qm7j | POD           | mixed_metric_linux_cmd_alarm         | resolved |             |                           | Fire when pod cpu usage > 10 in 15 out of 30 seconds
            |                            |               |                                      |          | ALARM_FIRE  | 2021-09-01T15:43:07-07:00 | More than 10% pod cpu usage short template
            |                            |               |                                      |          | ALARM_CLEAR | 2021-09-01T15:44:36-07:00 | Less than 10% pod cpu usage short template
4           | test4-cust.shoreline-9qm7j | POD           | linux_cmd_duration_res_env_var_alarm | resolved |             |                           | Fire when pod cpu usage > 10 in 15 out of 30 seconds
            |                            |               |                                      |          | ALARM_FIRE  | 2021-09-01T15:43:07-07:00 | More than 10% pod cpu usage short template
            |                            |               |                                      |          | ALARM_CLEAR | 2021-09-01T15:44:36-07:00 | test file succeed over 5 times short template
35          | test4-cust.shoreline-zrwhf | POD           | mixed_metric_linux_cmd_alarm         | resolved |             |                           | Fire when pod cpu usage > 10 in 15 out of 30 seconds
            |                            |               |                                      |          | ALARM_FIRE  | 2021-09-01T15:43:00-07:00 | More than 10% pod cpu usage short template
            |                            |               |                                      |          | ALARM_CLEAR | 2021-09-01T15:43:18-07:00 | Less than 10% pod cpu usage short template
35          | test4-cust.shoreline-zrwhf | POD           | linux_cmd_duration_res_env_var_alarm | resolved |             |                           | Fire when pod cpu usage > 10 in 15 out of 30 seconds
            |                            |               |                                      |          | ALARM_FIRE  | 2021-09-01T15:43:00-07:00 | More than 10% pod cpu usage short template
            |                            |               |                                      |          | ALARM_CLEAR | 2021-09-01T15:43:18-07:00 | test file succeed over 5 times short template

By Resource

You can pipe an events command onto any valid Resource query to filter only Events associated with the targeted Resource(s).

For example, here we're retrieving Bot Events from the shoreline-mxxhw pod Resource:

op>
pods | name=~"shoreline-mxxhw" | events | type="bot" | count
GROUP     | RESOURCE_ID | RESOURCE_TYPE | RESOURCE_NAME              | EVENT_TYPE | TRIGGERED | FINISHED | ACTIVE | TOTAL_COUNT
group_all |             |               |                            | BOTS       | 18        | 18       | 0      | 18
27        |             |               |                            | BOTS       | 18        | 18       | 0      | 18
          | 27          | POD           | test4-cust.shoreline-mxxhw |            |           |          |        |

By Column or Tag

You can filter Events by the returned column name or an underlying Resource tag using one of the following comparators:

  • =
  • !=
  • =~

The basic syntax is below.

events | <column_or_tag> = <value>
events | <column_or_tag> = [<value>, <value>, ...]
events | <column_or_tag> != <value>
events | <column_or_tag> =~ <value>
For example, count all Alarm Events within the us-west-2a Availability Zone:
op>
events | type="alarm" | az="us-west-2a" | count
GROUP     | RESOURCE_ID | RESOURCE_TYPE | RESOURCE_NAME                        | AZ         | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all |             |               |                                      |            | ALARMS     | 46    | 43      | 3      | 46
7         |             |               |                                      |            | ALARMS     | 5     | 4       | 1      | 5
          | 7           | HOST          | i-0627f066048c1b3d9                  | us-west-2a |            |       |         |        |
5         |             |               |                                      |            | ALARMS     | 4     | 3       | 1      | 4
          | 5           | HOST          | i-0192cd8c4f55219c2                  | us-west-2a |            |       |         |        |
2         |             |               |                                      |            | ALARMS     | 5     | 4       | 1      | 5
          | 2           | HOST          | i-03f82cb1e755c65a3                  | us-west-2a |            |       |         |        |
...

Notice the displayed column name is AZ, but it must be lower-cased in the comparison statement.

You can get results from both us-west-2a and us-west-2b using an array comparison value.

op>
events | type="alarm" | az=["us-west-2a", "us-west-2b"] | count
GROUP     | RESOURCE_ID | RESOURCE_TYPE | RESOURCE_NAME                        | AZ         | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all |             |               |                                      |            | ALARMS     | 99    | 93      | 6      | 99
8         |             |               |                                      |            | ALARMS     | 4     | 3       | 1      | 4
          | 8           | HOST          | i-0777445c24c2b8344                  | us-west-2b |            |       |         |        |
7         |             |               |                                      |            | ALARMS     | 5     | 4       | 1      | 5
          | 7           | HOST          | i-0627f066048c1b3d9                  | us-west-2a |            |       |         |        |
6         |             |               |                                      |            | ALARMS     | 6     | 5       | 1      | 6
          | 6           | HOST          | i-00f4a29cc5a8716d1                  | us-west-2b |            |       |         |        |
5         |             |               |                                      |            | ALARMS     | 4     | 3       | 1      | 4
          | 5           | HOST          | i-0192cd8c4f55219c2                  | us-west-2a |            |       |         |        |
...

You can use regex to match partial values, such as the Alarm name.

op>
events | type="alarm" | alarm_name=~"simple"
RESOURCE_ID | RESOURCE_NAME              | RESOURCE_TYPE | ALARM_NAME       | STATUS   | STEP_TYPE   | TIMESTAMP
99          | test2-cust.shoreline-bkk4c | POD           | simple_pod_alarm | resolved |             |
            |                            |               |                  |          | ALARM_FIRE  | 2021-09-01T22:44:47-07:00
            |                            |               |                  |          | ALARM_CLEAR | 2021-09-01T22:44:48-07:00
70          | test2-cust.shoreline-cbzgj | POD           | simple_pod_alarm | resolved |             |
            |                            |               |                  |          | ALARM_FIRE  | 2021-09-01T22:44:43-07:00
            |                            |               |                  |          | ALARM_CLEAR | 2021-09-01T22:44:47-07:00
23          | test2-cust.shoreline-q82hl | POD           | simple_pod_alarm | resolved |             |
            |                            |               |                  |          | ALARM_FIRE  | 2021-09-01T22:44:33-07:00
            |                            |               |                  |          | ALARM_CLEAR | 2021-09-01T22:44:46-07:00
35          | test2-cust.shoreline-mpccj | POD           | simple_pod_alarm | resolved |             |
            |                            |               |                  |          | ALARM_FIRE  | 2021-09-01T22:44:01-07:00
            |                            |               |                  |          | ALARM_CLEAR | 2021-09-01T22:44:46-07:00
...

If you specify an unrecognized column name for the column_or_tag key, the query attempts to match it against a Resource tag of the same name.

For example, here we're comparing the k8s_node_name tag to get Events associated with that specific Kubernetes node:

op>
hosts | events | k8s_node_name="ip-10-35-153-73.us-west-2.compute.internal" | count
GROUP     | RESOURCE_ID | RESOURCE_TYPE | RESOURCE_NAME       | AZ         | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all |             |               |                     |            | ALARMS     | 4     | 3       | 1      | 4
1         |             |               |                     |            | ALARMS     | 4     | 3       | 1      | 4
          | 1           | HOST          | i-0df4e06e625e46962 | us-west-2b |            |       |         |        |

Any tag name with invalid characters can be surrounded by quotes to allow it in a comparison statement.

op>
hosts | events | "kubernetes.io/hostname"="ip-10-35-153-73.us-west-2.compute.internal" | count
GROUP     | RESOURCE_ID | RESOURCE_TYPE | RESOURCE_NAME       | AZ         | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all |             |               |                     |            | ALARMS     | 4     | 3       | 1      | 4
1         |             |               |                     |            | ALARMS     | 4     | 3       | 1      | 4
          | 1           | HOST          | i-0df4e06e625e46962 | us-west-2b |            |       |         |        |

Time Ranges

base, offset

Get Events that existed around the base Unix time, in milliseconds.

For example, return the number of Events that occurred between July 13th, 2021 12:00:00 PM and July 13th, 2021 1:00:00 PM (UTC):

op>
events | base=1626177600000 | offset=3600000 | count
GROUP     | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all | ALARMS     | 2     | 2       | 7      | 9

In the above we've set the base timestamp to July 13th, 2021 12:00:00 PM (UTC) and offset it by adding one hour.

You can also set a negative offset value to create a timestamp range starting before the base value:

op>
events | base=1626177600000 | offset=-3600000 | count
GROUP     | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all | ALARMS     | 0     | 0       | 0      | 0

from, to

Get Events that existed between the specified timespan, in milliseconds, since the Unix epoch.

For example, return the number of Events during the 24-hour period of July 13th, 2021 (UTC):

op>
events | from=1626134400000 | to=1626220800000 | count
GROUP     | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all | ALARMS     | 2     | 2       | 7      | 9

Get Events between 5:00 and 6:00 PM on July 13th, 2021 (UTC):

op>
events | from=1626195600000 | to=1626199200000 | count
GROUP     | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all | ALARMS     | 0     | 0       | 0      | 0

window

The window(<number>[<unit>]) command retrieves Events that occurred within the previous defined period.

  • window(10s) - Get Events within the last ten minutes
  • window(5m) - Get Events within the last five minutes
  • window(24h) - Get Events within the last 24 hours
op>
events | window(10m) | count
GROUP     | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all | ALARMS     | 2     | 2       | 9      | 11

Type

Use type to view only specific types of Events. Valid values are:

See the named object Event sections above for examples.

Grouping

Group up Event results with the group("<column_name>") function.

For example, here we're grouping by resource_type to see all Alarm Events across each Resource type:
op>
events | group("resource_type") | count
GROUP     | RESOURCE_ID | RESOURCE_TYPE | RESOURCE_NAME                        | AZ         | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all |             |               |                                      |            | ALARMS     | 57    | 57      | 0      | 57
POD       |             |               |                                      |            | ALARMS     | 49    | 49      | 0      | 49
          | 27          | POD           | test4-cust.shoreline-mxxhw           | us-west-2c |            |       |         |        |
          | 35          | POD           | test4-cust.shoreline-zrwhf           | us-west-2b |            |       |         |        |
          | 4           | POD           | test4-cust.shoreline-9qm7j           | us-west-2a |            |       |         |        |
HOST      |             |               |                                      |            | ALARMS     | 3     | 3       | 0      | 3
          | 3           | HOST          | i-0e1d2e73331dd57b1                  | us-west-2b |            |       |         |        |
          | 2           | HOST          | i-0746dfd68699f2227                  | us-west-2c |            |       |         |        |
CONTAINER |             |               |                                      |            | ALARMS     | 5     | 5       | 0      | 5
          | 36          | CONTAINER     | test4-cust.shoreline-mxxhw.shoreline | us-west-2c |            |       |         |        |
          | 42          | CONTAINER     | test4-cust.shoreline-zrwhf.shoreline | us-west-2b |            |       |         |        |
          | 12          | CONTAINER     | test4-cust.shoreline-9qm7j.shoreline | us-west-2a |            |       |         |        |

Grouping by alarm_name lets you see how each specific Alarm is behaving.

op>
events | group("alarm_name") | count
GROUP                                | RESOURCE_ID | RESOURCE_TYPE | RESOURCE_NAME                        | AZ         | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all                            |             |               |                                      |            | ALARMS     | 57    | 57      | 0      | 57
timeout_action_alarm                 |             |               |                                      |            | ALARMS     | 1     | 1       | 0      | 1
                                     | 12          | CONTAINER     | test4-cust.shoreline-9qm7j.shoreline | us-west-2a |            |       |         |        |
simple_pod_alarm                     |             |               |                                      |            | ALARMS     | 9     | 9       | 0      | 9
                                     | 27          | POD           | test4-cust.shoreline-mxxhw           | us-west-2c |            |       |         |        |
                                     | 35          | POD           | test4-cust.shoreline-zrwhf           | us-west-2b |            |       |         |        |
                                     | 4           | POD           | test4-cust.shoreline-9qm7j           | us-west-2a |            |       |         |        |
no_timeout_action_alarm              |             |               |                                      |            | ALARMS     | 1     | 1       | 0      | 1
                                     | 12          | CONTAINER     | test4-cust.shoreline-9qm7j.shoreline | us-west-2a |            |       |         |        |
mixed_metric_linux_cmd_alarm         |             |               |                                      |            | ALARMS     | 20    | 20      | 0      | 20
                                     | 27          | POD           | test4-cust.shoreline-mxxhw           | us-west-2c |            |       |         |        |
                                     | 35          | POD           | test4-cust.shoreline-zrwhf           | us-west-2b |            |       |         |        |
                                     | 4           | POD           | test4-cust.shoreline-9qm7j           | us-west-2a |            |       |         |        |
linux_cmd_duration_res_env_var_alarm |             |               |                                      |            | ALARMS     | 20    | 20      | 0      | 20
                                     | 27          | POD           | test4-cust.shoreline-mxxhw           | us-west-2c |            |       |         |        |
                                     | 35          | POD           | test4-cust.shoreline-zrwhf           | us-west-2b |            |       |         |        |
                                     | 4           | POD           | test4-cust.shoreline-9qm7j           | us-west-2a |            |       |         |        |
duplicate_delay_clear_alarm          |             |               |                                      |            | ALARMS     | 3     | 3       | 0      | 3
                                     | 36          | CONTAINER     | test4-cust.shoreline-mxxhw.shoreline | us-west-2c |            |       |         |        |
                                     | 42          | CONTAINER     | test4-cust.shoreline-zrwhf.shoreline | us-west-2b |            |       |         |        |
                                     | 12          | CONTAINER     | test4-cust.shoreline-9qm7j.shoreline | us-west-2a |            |       |         |        |
always_fire_alarm                    |             |               |                                      |            | ALARMS     | 3     | 3       | 0      | 3
                                     | 3           | HOST          | i-0e1d2e73331dd57b1                  | us-west-2b |            |       |         |        |
                                     | 2           | HOST          | i-0746dfd68699f2227                  | us-west-2c |            |       |         |        |

Order

Order the Event results based on a valid type-specific column_name.

For example, here we're ordering Action Events by the ACTION_START timestamp:
op>
events | type="action" | order("timestamp_start")
RESOURCE_ID | RESOURCE_NAME                        | RESOURCE_TYPE | ACTION_NAME                           | BOT_NAME                           | STATUS    | STEP_TYPE    | TIMESTAMP
40          | test2-cust.shoreline-mpccj.shoreline | CONTAINER     | timeout_action                        | timeout_action_bot                 | failed    |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T22:39:35-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T22:39:36-07:00
            |                                      |               |                                       |                                    |           | ACTION_FAIL  | 2021-09-01T22:39:40-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
17          | test2-cust.shoreline-gj2zh.shoreline | CONTAINER     | mixed_metric_linux_cmd_action         | mixed_metric_linux_cmd_bot         | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T22:39:42-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T22:39:43-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T22:39:43-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
40          | test2-cust.shoreline-mpccj.shoreline | CONTAINER     | linux_cmd_duration_res_env_var_action | linux_cmd_duration_res_env_var_bot | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T22:39:45-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T22:39:47-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T22:39:48-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00

Here we're sorting by the Action name property:

op>
events | type="action" | order("action_name") | limit=5
RESOURCE_ID | RESOURCE_NAME                        | RESOURCE_TYPE | ACTION_NAME                           | BOT_NAME                           | STATUS    | STEP_TYPE    | TIMESTAMP
9           | i-0f3ebe788cf2268d9                  | HOST          | host_top_action                       | host_top_bot                       | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T22:41:43-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T22:41:44-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T22:41:44-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
115         | test2-cust.shoreline-qjmxc.shoreline | CONTAINER     | linux_cmd_duration_res_env_var_action | linux_cmd_duration_res_env_var_bot | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T22:44:32-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T22:44:33-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T22:44:33-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
115         | test2-cust.shoreline-qjmxc.shoreline | CONTAINER     | mixed_metric_linux_cmd_action         | mixed_metric_linux_cmd_bot         | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T22:44:32-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T22:44:33-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T22:44:33-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
40          | test2-cust.shoreline-mpccj.shoreline | CONTAINER     | no_timeout_action                     | no_timeout_action_bot              | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T22:39:35-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T22:39:36-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T22:40:46-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
40          | test2-cust.shoreline-mpccj.shoreline | CONTAINER     | timeout_action                        | timeout_action_bot                 | failed    |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T22:39:35-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T22:39:36-07:00
            |                                      |               |                                       |                                    |           | ACTION_FAIL  | 2021-09-01T22:39:40-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00

Pagination

Paginate the returned Events using start_index and limit.

op>
events | type="action" | start_index=10 | limit=3
RESOURCE_ID | RESOURCE_NAME                        | RESOURCE_TYPE | ACTION_NAME                           | BOT_NAME                           | STATUS    | STEP_TYPE    | TIMESTAMP
50          | test2-cust.shoreline-28lbk.shoreline | CONTAINER     | linux_cmd_duration_res_env_var_action | linux_cmd_duration_res_env_var_bot | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T22:43:46-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T22:43:47-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T22:43:47-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
93          | test2-cust.shoreline-5bzm4.shoreline | CONTAINER     | mixed_metric_linux_cmd_action         | mixed_metric_linux_cmd_bot         | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T22:43:42-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T22:43:43-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T22:43:43-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00
93          | test2-cust.shoreline-5bzm4.shoreline | CONTAINER     | linux_cmd_duration_res_env_var_action | linux_cmd_duration_res_env_var_bot | completed |              |
            |                                      |               |                                       |                                    |           | ALARM_FIRE   | 2021-09-01T22:43:42-07:00
            |                                      |               |                                       |                                    |           | ACTION_START | 2021-09-01T22:43:43-07:00
            |                                      |               |                                       |                                    |           | ACTION_END   | 2021-09-01T22:43:43-07:00
            |                                      |               |                                       |                                    |           | ALARM_CLEAR  | 1969-12-31T16:00:00-08:00

Filter Resources by Events

You can filter Resources by Events, based on things like the Event type, name, count, and so forth.

For example, below we're filtering all host Resources to those that have triggered the my_cpu_alarm:

op>
hosts | filter(events | type="alarm" | name="my_cpu_alarm")
RESOURCE_ID | RESOURCE_NAME       | RESOURCE_TYPE | ALARM_NAME        | STATUS   | STEP_TYPE   | TIMESTAMP
2           | i-0746dfd68699f2227 | HOST          | my_cpu_alarm      | resolved |             |
            |                     |               |                   |          | ALARM_FIRE  | 2021-09-01T15:38:45-07:00
            |                     |               |                   |          | ALARM_CLEAR | 2021-09-01T15:38:46-07:00
2           | i-0746dfd68699f2227 | HOST          | my_cpu_alarm      | resolved |             |
            |                     |               |                   |          | ALARM_FIRE  | 2021-09-01T15:38:43-07:00
            |                     |               |                   |          | ALARM_CLEAR | 2021-09-01T15:38:44-07:00
3           | i-0e1d2e73331dd57b1 | HOST          | my_cpu_alarm      | resolved |             |
            |                     |               |                   |          | ALARM_FIRE  | 2021-09-01T15:38:09-07:00
            |                     |               |                   |          | ALARM_CLEAR | 2021-09-01T15:43:18-07:00
Filters excluded 1/3 hosts

When passing the count command to the Event query it returns the number of matching Events, which is a great way to further reduce the returned Resources.

Here we're seeing that only one host triggered my_cpu_alarm multiple times.

op>
hosts | filter((events | type="alarm" | name="my_cpu_alarm" | count) >= 2)
GROUP     | RESOURCE_ID | RESOURCE_TYPE | RESOURCE_NAME       | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
group_all |             |               |                     | ALARMS     | 3     | 3       | 0      | 3
3         |             |               |                     | ALARMS     | 1     | 1       | 0      | 1
          | 3           | HOST          | i-0e1d2e73331dd57b1 |            |       |         |        |
2         |             |               |                     | ALARMS     | 2     | 2       | 0      | 2
          | 2           | HOST          | i-0746dfd68699f2227 |            |       |         |        |
Filters excluded 2/3 hosts

You can also use additional Event query options such as status to filter only Resources that triggered an Alarm.

op>
hosts | filter((events | type="alarm" | status="triggered"))
Filters excluded 3/3 hosts