Actions

Actions are customizable auto-remediation and diagnostic tools that act upon Resources to help resolve Alarms.

Actions execute shell commands on associated Resources. Whenever an Alarm fires the associated Bot triggers the corresponding Action, closing the basic auto-remediation loop of Shoreline.

Action Properties

Each Action has many properties that determine its behavior. The required properties are:

  • name - The name of the Action.
  • command - The shell command executed when the Action triggers.

Check out Action Properties for details on all available properties and how to use them.

Create an Action

Actions are created using either the CLI or the Configuration UI.

As a demonstration, below we'll create an Action that outputs the three processes with the highest CPU usage.

Create an Action with Op using the following syntax: action <name> = <command>.

  • name - The name of the Action.
  • command - The shell command executed when the Action triggers.
  1. Begin by defining an Action named high_cpu_action and set the command property to the following:

    op>
    action high_cpu_action = `ps -Ao comm,pid,pcpu --sort=-pcpu | head -n 4`
    Created action 'high_cpu_action'.
    
    Understanding Action Resource Query

    By default, an Action executes its command against the Resources defined by the triggering Alarm's resource_query.

    However, if you wish to manually execute an Action or use it in context outside of the normal automated remediation loop, you'll need to specify the Action's resource_query property.

    For example, after creating the high_cpu_action above and before defining its own resource_query, manually executing the Action in the Op CLI throws an error:

    op>
    high_cpu_action()
    Error: Linux commands must be preceded by a resource query
    

    As the error states, without a defined resource_query property the Action does not know which Resources to execute against.

  2. Set the resource_query property to execute against all containers with names containing "shoreline"

    op>
    high_cpu_action.resource_query = container | name=~"shoreline"
    Updated action 'high_cpu_action'.
    
  3. Try manually executing the Action once again

    op>
    high_cpu_action()
    ID | TYPE      | NAME                                  | STDOUT
    8  | CONTAINER | test10-cust.shoreline-67ln6.shoreline | COMMAND           PID %CPU
       |           |                                       | metric_scraper    224  1.9
       |           |                                       | python3           201  0.8
       |           |                                       | python3           236  0.8
       |           |                                       |
    17 | CONTAINER | test10-cust.shoreline-2ghfk.shoreline | COMMAND           PID %CPU
       |           |                                       | metric_scraper    223  2.3
       |           |                                       | python3           199  0.9
       |           |                                       | python3           235  0.9
       |           |                                       |
    38 | CONTAINER | test10-cust.shoreline-2kkh2.shoreline | COMMAND           PID %CPU
       |           |                                       | metric_scraper    220  5.6
       |           |                                       | python3           234  2.0
       |           |                                       | python3           233  1.8
       |           |                                       |
    

    Now that the Action knows which Resources to execute against we get the expected results.

    See the full Action.resource_query property description for more details.

Edit an Action

You can edit an existing Action from either the CLI or the UI.

To edit an existing Action in the CLI simply set a new value for the appropriate Action property.

See Action Properties for details.

Delete an Action

You can delete an existing Action from either the CLI or the UI.

To delete an Action via the CLI use the delete Op command:

op>
delete high_cpu_action
Deleted action 'high_cpu_action'.

Built-in Environment Variables

The Agent auto-populates a handful of system environment variables during execution. These variables are ideal for use in your own Linux command Alarms or Actions, and are also used extensively throughout our pre-built Op Packs.

The following table lists the built-in environment variables, the executing Resource types they're available on, and a brief description.

Variable NameResource AvailabilityDescription
ACTION_INSTANCE_IDallInstance ID
ACTION_CONTAINER_NAMEcontainer, podKubernetes container name
ACTION_NAMESPACEcontainer, podKubernetes namespace
ACTION_NODE_NAMEallKubernetes node name
ACTION_POD_NAMEcontainer, podKubernetes pod name
ALARM_INSTANCE_IDallInstance ID
ALARM_CONTAINER_NAMEcontainer, podKubernetes container name
ALARM_NAMESPACEcontainer, podKubernetes namespace
ALARM_NODE_NAMEallKubernetes node name
ALARM_POD_NAMEcontainer, podKubernetes pod name

As the names suggest, ACTION_ variables are available within the context of an executing Action and ALARM_ variables within the context of an Alarm.