Op Packs

Op Packs are collections of Terraform configuration files that tightly integrate with Shoreline's auto-remediation utilities.

Op Packs automate common remediation tasks such as increasing Kubernetes disk sizes, retiring Kubernetes nodes, and logging excessive Java Virtual Machine memory usage.

Utilize the Shoreline Terraform Provider to completely customize, automate, codify, and commit all of your incident resolution scripts.

Prerequisites

Install Terraform

Terraform is available as a binary package for most operating systems. Follow these steps to install Terraform on your local machine.

  1. Visit the official downloads page

  2. Download the appropriate package for your operating system and architecture

  3. Unzip the downloaded file

    unzip terraform_1.0.2_linux_amd64.zip
    
  4. Move the terraform binary to /usr/local/bin/ for Mac/Linux systems:

    mv ~/Downloads/terraform /usr/local/bin/
    

    On Windows, move the terraform binary to a directory defined in your PATH system variable.

  5. Verify the installation

    $ terraform --version
    Terraform v1.0.2
    on linux_amd64
    

Install the Op CLI

You'll need a local Op CLI installation to authenticate with your Shoreline cluster.

Authenticating

You'll need to authenticate with the desired cluster via the Op CLI before the Shoreline Provider functions.

The auth Op command provides auth functionality within the CLI.

Start by opening the Op CLI:

oplang_cli

To authenticate with a Shoreline cluster use the auth <url> command, where <url> is the location of your Shoreline cluster API:

op>
auth https://acme.us.api.shoreline-acme.io
This command opens a browser window and authenticates with the identity provider configured with your Shoreline environment, e.g., Okta.

After you authenticate, the CLI prompt indicates which Shoreline cluster you've connected to:

op:acme-test-shoreline>

The .ops_auth.yaml file locally stores all Op auth tokens and their respective Shoreline API endpoints.

By default, it is located in /home/<username>/.ops_auth.yaml.

auth:
  - Token: <token>
    Url: https://acme.us.api.shoreline-acme.io
  - Token: <token>
    Url: https://acme2.us.api.shoreline-acme.io
token: <token>
url: https://acme.us.api.shoreline-acme.io

Alternatively, you can manually define the Shoreline cluster URL and auth token by setting SHORELINE_URL and SHORELINE_TOKEN environment variables.

$ export SHORELINE_URL=https://acme.us.api.shoreline-acme.io
$ export SHORELINE_TOKEN=<token>

Shoreline Terraform Provider

Terraform is an infrastructure as code (IaC) tool, allowing you to use configuration files to manage your infrastructure.

The Shoreline Terraform Provider and associated Op Packs are remediation-as-code (RaC) tools. Generate auto-remediation and diagnostic objects throughout your fleet using standard Terraform configuration files and patterns.

Creating a Terraform Configuration

All Terraform configurations consist of one or more plain text .tf files. Each .tf file defines one or more blocks using the JSON-like HCL syntax:

<BLOCK TYPE> "<BLOCK LABEL>" "<BLOCK LABEL>" {
  # Block body
  <IDENTIFIER> = <EXPRESSION> # Argument
}

The Terraform CLI automatically detects and references all .tf files in the current execution directory.

Therefore, we recommend naming your .tf files based on their purpose when creating a Shoreline Terraform configuration. For example, you might use a provider.tf file to configure the Shoreline Terraform Provider or an alarms.tf file to configure your Alarm blocks.

Using the Shoreline Provider

To use the Shoreline Provider, add the following to your Terraform configuration:

terraform {
  required_providers {
    shoreline = {
      source = "shorelinesoftware/shoreline"
      version = ">= 1.0.2"
    }
  }
}

provider "shoreline" {
  url = "<cluster_api_url>"
  # Optional (default: 2)
  retries = 2
}

This is the minimum configuration, beyond which you'll add extra Op Pack configurations or add custom configurations.

Using Op Packs

Op Packs are special purpose Terraform configurations that use Op to generate Shoreline Actions, Alarms, Bots, Metrics, and Resources.

Custom Configuration

Customizing your Shoreline Provider configuration is just like configuring any other Terraform Provider.

The Shoreline Terraform Provider defines several Terraform resources, each of which corresponds to core Shoreline objects.

Action

To create an Action use the shoreline_action Terraform resource type:

resource "shoreline_action" "ls_action" {
  name = "my_ls_action"
  command = "`ls /tmp`"
}

Required Properties

  • name: string
  • command: string

Optional Properties

  • description: string
  • enabled: boolean
  • resource_query: Op string
  • params: [string]
  • shell: string
  • user: string
  • timeout: number
  • res_env_var: string
  • start_title_template: string
  • start_long_template: string
  • start_short_template: string
  • complete_title_template: string
  • complete_long_template: string
  • complete_short_template: string
  • error_title_template: string
  • error_long_template: string
  • error_short_template: string

Alarm

To create an Alarm use the shoreline_alarm Terraform resource type:

resource "shoreline_alarm" "cpu_alarm" {
  name = "my_cpu_alarm"
  fire_query = "(cpu_usage >= 40 | sum(60)) >= 48.0"
  clear_query = "(cpu_usage < 40 | sum(60)) >= 48.0"
  resource_query = "host"
}

The above example creates an Alarm named my_cpu_alarm that fires when at least 80% of a host Resource's CPU usage metric measurements are equal to or exceed 40% over the previous minute.

Required Properties
  • name: string
  • resource_query: string
  • fire_query: Op string
  • clear_query: Op string

Optional Properties

  • description: string
  • enabled: boolean
  • mute_query: Op string
  • fire_title_template: string
  • fire_long_template: string
  • fire_short_template: string
  • resolve_title_template: string
  • resolve_long_template: string
  • resolve_short_template: string
  • family: string
  • raise_for: string
  • metric_name: string
  • condition_type: string
  • condition_value: string
  • check_interval_sec: number

Bot

To create a Bot use the shoreline_bot Terraform resource type:

resource "shoreline_bot" "cpu_bot" {
  name = "my_cpu_bot"
  command = "if ${shoreline_alarm.cpu_alarm.name} then ${shoreline_action.ls_action.name} fi"
}

Required Properties

Optional Properties

  • description: string
  • enabled: boolean

Metric

To create a Metric use the shoreline_metric Terraform resource type:

resource "shoreline_metric" "avg_cpu_usage_prev_min" {
  name = "avg_cpu_usage_prev_min"
  value = "cpu_usage | window(60s) | mean(60)"
}

Required Properties

Optional Properties

  • description: string
  • unit: string
  • resource_type: Op string

File

To transfer a File use the shoreline_file Terraform resource type:

resource "shoreline_file" "transfer_books_file" {
  name = "${var.prefix}_transfer_books_file"
  input_file = "${path.module}/books.json"
  destination_path = "/tmp/books.json"
}

Required Properties

  • name: string
  • input_file: string
  • destination_path: string

Optional Properties

  • description: string
  • resource_query: Op string
  • resource_type: string
  • enabled: boolean
  • file_length: number

Basic Auto-Remediation Group

The following configuration creates a basic auto-remediation loop within Shoreline via Terraform:

  • The Alarm defines when there is an issue.
  • The Action defines what to do about the issue.
  • The Bot connects the two by triggering the Action when the Alarm fires.
# Defines the required Shoreline provider and version
# @ref https://docs.shoreline.io/op/packs/tutorial#create-a-configuration-file
terraform {
  required_providers {
    shoreline = {
      source  = "shorelinesoftware/shoreline"
      version = ">= 1.0.7"
    }
  }
}

# Set provider-specific arguments
# @ref https://docs.shoreline.io/op/packs/tutorial#create-a-configuration-file
provider "shoreline" {
  # Set to the Shoreline cluster API URL, e.g. https://acme.us.api.shoreline-acme.io
  url = "<CLUSTER_API_ENDPOINT>"
}

# Create an Alarm that fires when host CPU usage exceeds 35% for 48 of the previous 60 seconds.
# This Alarm clears when CPU usage is below 35% for the previous 180 seconds.
# @ref https://docs.shoreline.io/op/packs/tutorial#create-an-alarm
# @ref https://docs.shoreline.io/alarms
resource "shoreline_alarm" "high_cpu_alarm" {
  name                   = "high_cpu_alarm"
  fire_query             = "(cpu_usage > 35 | sum(60)) >= 48"
  clear_query            = "(cpu_usage < 35 | sum(180)) >= 180"
  description            = "Watch CPU usage."
  resource_query         = "hosts"
  enabled                = true
  resolve_short_template = "high_cpu_alarm resolved"
}

# Create an Action that executes a Linux command to count the active background jobs.
# @ref https://docs.shoreline.io/op/packs/tutorial#create-an-action
# @ref https://docs.shoreline.io/actions
resource "shoreline_action" "background_jobs_action" {
  name                    = "background_jobs_action"
  command                 = "`top -b -n 1 | head -n 15`"
  description             = "Count background jobs"
  start_title_template    = "background_jobs_action started"
  complete_title_template = "background_jobs_action completed"
  error_title_template    = "background_jobs_action failed"
  enabled                 = true
}

# Create a Bot that triggers the 'background_jobs_action' when the 'high_cpu_alarm' fires.
# @ref https://docs.shoreline.io/op/packs/tutorial#create-a-bot
# @ref https://docs.shoreline.io/bots
resource "shoreline_bot" "cpu_bot" {
  name        = "cpu_bot"
  command     = "if ${shoreline_alarm.high_cpu_alarm.name} then ${shoreline_action.background_jobs_action.name} fi"
  description = "Count background jobs on high CPU usage."
  enabled     = true
}

Usage

Once your configuration is complete, you're ready to execute standard Terraform commands to update your infrastructure accordingly.

Initialization

The first time you use the Shoreline Terraform Provider you must initialize the current working directory.

terraform init

This command will download the Shoreline Terraform Provider (along with any other necessary Providers) to the local .terraform directory so they are usable.

See the Terraform Command: init documentation for details.

Plan

The next step is to create an execution plan.

terraform plan

This command analyzes remote objects within your cluster and compares them against what you've defined in your Terraform configuration. Differences are output, and Terraform indicates which changes will occur if you apply this configuration.

See the Terraform Command: plan documentation for details.

Apply

Use terraform apply to execute the changes defined by your Terraform configuration.

terraform apply

See the Terraform Command: apply documentation for details.

Destroy

Executing terraform destroy destroys all remote objects.

terraform destroy

See the Terraform Command: destroy documentation for details.