Jobs

Automatically generate background jobs for long-running Linux commands.

Jobs allow you to issue and track CLI commands seamlessly across a large number of resources without worrying about getting stuck due to network failures, node outages, or system responsiveness. Jobs are generated using a feature called statement backgrounding.

When invoking a Linux command to one or more Resources, if the execution time of that command takes longer than the background_timeout time, then Shoreline automatically creates a job for that issued statement.

To illustrate, consider a short-running command that takes about a second to complete. Here, we'll simulate the delay using the Unix sleep command.

op>
hosts | `sleep 1 && echo "Complete"`

The result is a quick round trip, and you get results after just over a second, as expected.

ID | TYPE | NAME                | REGION    | AZ         | STATUS | STDOUT
65 | HOST | i-00f9034231ac2bec0 | us-west-2 | us-west-2a |   0    | Complete
66 | HOST | i-0ba70a62ee3b38683 | us-west-2 | us-west-2c |   0    | Complete
67 | HOST | i-00e36a7da4ff269b1 | us-west-2 | us-west-2b |   0    | Complete

Next, consider a long-running command that takes up to 30 seconds to complete.

op>
hosts | `sleep 30 && echo "Complete"`

Without statement backgrounding, invoking the above command effectively locks the CLI while waiting for a response. However, with statement backgrounding, you get a response after just a few seconds.

ID | TYPE | NAME                | REGION    | AZ
65 | HOST | i-00f9034231ac2bec0 | us-west-2 | us-west-2a
66 | HOST | i-0ba70a62ee3b38683 | us-west-2 | us-west-2c
67 | HOST | i-00e36a7da4ff269b1 | us-west-2 | us-west-2b
Some commands were backgrounded.
Use 'jobs' to view all jobs.
Use 'jobs | id = 300009015' to view detailed info for this job.

The CLI indicates when a statement is backgrounded because it exceeds your background_timeout duration.

Enable Jobs

Jobs and statement backgrounding are disabled by default. To enable statement backgrounding you must set a background_timeout value.

The background_timeout setting determines the number of seconds that an Op statement has to complete execution and return a result before a background job is created.

  • Default value: 5
  • Minimum value: 0
  • Maximum value: 30
op>
config set background_timeout 3
Set config[background_timeout]: 3

List Jobs

Get the most recent jobs with the jobs command.

op>
jobs
ID        | STATUS    | DURATION_MS | USER              | OP_STMT
300009197 | RUNNING   | 2109        | gabe@shoreline.io | host | `sleep 5 && echo foo` | `echo bar`
300009192 | RUNNING   | 6262        | gabe@shoreline.io | pods | app="shoreline" | filter(pod_cpu_usage > 4) | `sleep 5 && echo "slept 5 on filtered pods"`
300009180 | SUCCEEDED | 6308        | gabe@shoreline.io | host | filter(cpu_usage > 6) | `sleep 5 && echo "over 6"`
300009168 | SUCCEEDED | 6113        | gabe@shoreline.io | host | timeout(10s) | `if [[ "$ACTION_INSTANCE_ID" =~ ^(i-0fc14f864dbee109a)$ ]]; then sleep 15 && echo slept15; else sleep 5 && echo done_sleeping; fi`
300009162 | SUCCEEDED | 22213       | gabe@shoreline.io | host | filter(`sleep 10 && echo "slept 10 sec"`) | `sleep 10 && echo "slept 10 sec"`
300009156 | SUCCEEDED | 25126       | gabe@shoreline.io | host | `sleep $((5 + $RANDOM % 25)) && echo hello`
300009154 | SUCCEEDED | 6109        | gabe@shoreline.io | host | `if [[ "$ACTION_INSTANCE_ID" =~ ^(i-0dad66d1996e75ad4|i-0d22fb6dea9e664d6|i-0e2d852eb4a7c0012)$ ]]; then echo "Before IT"; else sleep 5 && echo "After IT"; fi`
300009148 | SUCCEEDED | 16129       | gabe@shoreline.io | host | `sleep 15 && echo "slept 15 secs"`
300009015 | SUCCEEDED | 31142       | gabe@shoreline.io | host | `sleep 30 && echo "Complete"`
300008958 | SUCCEEDED | 31141       | gabe@shoreline.io | host | `sleep 30 && echo "Complete"`

By default, jobs returns the ten most recent jobs.

Limit

Set the number of jobs returned using the limit parameter.

op>
jobs | limit = 20
ID   | STATUS          | DURATION_MS | USER                                | OP_STMT
2896 | SUCCEEDED       | 11253       | gabe@shoreline.io                   | containers | limit = 5 | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then sleep 15; else sleep 5 && echo done; fi`
2756 | SUCCEEDED       | 11237       | gabe@shoreline.io                   | containers | limit = 5 | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then sleep 15; else sleep 5 && echo done; fi`
2507 | SUCCEEDED       | 4242        | gabe@shoreline.io                   | containers | limit = 5 | timeout(10s) | `sleep 3 && echo done`
2493 | SUCCEEDED       | 3223        | gabe@shoreline.io                   | containers | limit = 5 | timeout(10s) | `sleep 2 && echo done`
2488 | SUCCEEDED       | 6223        | gabe@shoreline.io                   | containers | limit = 5 | timeout(10s) | `sleep 5 && echo done`
2418 | SUCCEEDED       | 11240       | gabe@shoreline.io                   | containers | limit = 5 | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then sleep 15 && echo slept 15; else sleep 5 && echo done_sleeping; fi`
2347 | SUCCEEDED       | 6248        | gabe@shoreline.io                   | containers | limit = 5 | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ ^(bookstore)$ ]]; then sleep 15 && echo slept 15; else sleep 5 && echo done_sleeping; fi`
2127 | SUCCEEDED       | 6357        | integration-test-admin@shoreline.io | host | filter(cpu_usage > 1) | `sleep 5 && echo hello`
2096 | SUCCEEDED       | 56313       | integration-test-admin@shoreline.io | host | .pod | app="shoreline" | `sleep $((5 + $RANDOM % 25)) && echo "Linux cmd 1"` | `sleep $((5 + $RANDOM % 25)) && echo "Linux cmd 2"`
2086 | SUCCEEDED       | 11151       | integration-test-admin@shoreline.io | host | timeout(10s) | `if [[ "$ACTION_INSTANCE_ID" =~ ^(i-0e95acb5362fedaf5)$ ]]; then sleep 15 && echo done; else sleep 5 && echo done_sleeping; fi`
2056 | PARTIAL_FAILURE | 35085       | integration-test-admin@shoreline.io | host | timeout(20s) | `sleep 15; echo done`
2049 | SUCCEEDED       | 11115       | integration-test-admin@shoreline.io | host | `sleep 10 && echo world`
2037 | SUCCEEDED       | 11125       | integration-test-admin@shoreline.io | host | `sleep 10; echo "done statement 2"`
2036 | SUCCEEDED       | 6115        | integration-test-admin@shoreline.io | host | `sleep 5; echo " done statement 1"`
2030 | SUCCEEDED       | 6571        | gabe@shoreline.io                   | container | app="bookstore" | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ ^(bookstore)$ ]]; then sleep 15 && echo slept 15; else sleep 5 && echo done_sleeping; fi`
2026 | SUCCEEDED       | 12268       | integration-test-admin@shoreline.io | host | filter(`sleep 5 && echo hello`) | `sleep 5 && echo world`
2016 | SUCCEEDED       | 6122        | integration-test-admin@shoreline.io | host | `if [[ "$ACTION_INSTANCE_ID" =~ ^(i-0d66949e63d67dd3e)$ ]]; then echo done; else sleep 5 && echo done_sleeping; fi`
2011 | SUCCEEDED       | 6139        | integration-test-admin@shoreline.io | host | `sleep 5; ls`

Status

Filter jobs by status with the status parameter.

Valid values:

  • succeeded: The job was successful.
  • failed: The job failed.
  • running: The job is active.
  • partial_failure: The job succeeded on some, but not all, targeted Resources.

For example, consider the following list of recent jobs.

op>
jobs
ID   | STATUS          | DURATION_MS  | USER                                 | OP_STMT
3524 | RUNNING         | 1.010232e+06 | gabe@shoreline.io                    | host | limit = 3 | timeout(3600s) | `long-running-script.sh`
2418 | SUCCEEDED       | 11240        | gabe@shoreline.io                    | containers | limit = 5 | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then sleep 15 && echo slept 15; else sleep 5 && echo done_sleeping; fi`
2347 | SUCCEEDED       | 6248         | gabe@shoreline.io                    | containers | limit = 5 | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ ^(bookstore)$ ]]; then sleep 15 && echo slept 15; else sleep 5 && echo done_sleeping; fi`
2086 | SUCCEEDED       | 11151        | integration-test-admin@shoreline.io  | host | timeout(10s) | `if [[ "$ACTION_INSTANCE_ID" =~ ^(i-0e95acb5362fedaf5)$ ]]; then sleep 15 && echo done; else sleep 5 && echo done_sleeping; fi`
2056 | PARTIAL_FAILURE | 35085        | integration-test-admin@shoreline.io  | host | timeout(20s) | `sleep 15; echo done`
2049 | SUCCEEDED       | 11115        | integration-test-admin@shoreline.io  | host | `sleep 10 && echo world`
2037 | SUCCEEDED       | 11125        | integration-test-admin@shoreline.io  | host | `sleep 10; echo "done statement 2"`

Get only the successful jobs with status = "succeeded":

op>
jobs | status = "succeeded"
ID   | STATUS          | DURATION_MS  | USER                                 | OP_STMT
2418 | SUCCEEDED       | 11240        | gabe@shoreline.io                    | containers | limit = 5 | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then sleep 15 && echo slept 15; else sleep 5 && echo done_sleeping; fi`
2347 | SUCCEEDED       | 6248         | gabe@shoreline.io                    | containers | limit = 5 | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ ^(bookstore)$ ]]; then sleep 15 && echo slept 15; else sleep 5 && echo done_sleeping; fi`
2086 | SUCCEEDED       | 11151        | integration-test-admin@shoreline.io  | host | timeout(10s) | `if [[ "$ACTION_INSTANCE_ID" =~ ^(i-0e95acb5362fedaf5)$ ]]; then sleep 15 && echo done; else sleep 5 && echo done_sleeping; fi`
2049 | SUCCEEDED       | 11115        | integration-test-admin@shoreline.io  | host | `sleep 10 && echo world`
2037 | SUCCEEDED       | 11125        | integration-test-admin@shoreline.io  | host | `sleep 10; echo "done statement 2"`

Or, get just the running jobs:

op>
jobs | status="running"
ID   | STATUS          | DURATION_MS  | USER                                 | OP_STMT
3524 | RUNNING         | 1.010232e+06 | gabe@shoreline.io                    | host | limit = 3 | timeout(3600s) | `long-running-script.sh`

User

Filter jobs by the executing user with the user parameter.

op>
jobs | user = "gabe@shoreline.io"
ID   | STATUS          | DURATION_MS  | USER                                 | OP_STMT
3524 | RUNNING         | 1.010232e+06 | gabe@shoreline.io                    | host | limit = 3 | timeout(3600s) | `long-running-script.sh`
2418 | SUCCEEDED       | 11240        | gabe@shoreline.io                    | containers | limit = 5 | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then sleep 15 && echo slept 15; else sleep 5 && echo done_sleeping; fi`
2347 | SUCCEEDED       | 6248         | gabe@shoreline.io                    | containers | limit = 5 | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ ^(bookstore)$ ]]; then sleep 15 && echo slept 15; else sleep 5 && echo done_sleeping; fi`
2086 | SUCCEEDED       | 11151        | integration-test-admin@shoreline.io  | host | timeout(10s) | `if [[ "$ACTION_INSTANCE_ID" =~ ^(i-0e95acb5362fedaf5)$ ]]; then sleep 15 && echo done; else sleep 5 && echo done_sleeping; fi`
2056 | PARTIAL_FAILURE | 35085        | integration-test-admin@shoreline.io  | host | timeout(20s) | `sleep 15; echo done`
2049 | SUCCEEDED       | 11115        | integration-test-admin@shoreline.io  | host | `sleep 10 && echo world`
2037 | SUCCEEDED       | 11125        | integration-test-admin@shoreline.io  | host | `sleep 10; echo "done statement 2"`
op>
jobs | user = "integration-test-admin@shoreline.io"
ID   | STATUS          | DURATION_MS  | USER                                 | OP_STMT
2086 | SUCCEEDED       | 11151        | integration-test-admin@shoreline.io  | host | timeout(10s) | `if [[ "$ACTION_INSTANCE_ID" =~ ^(i-0e95acb5362fedaf5)$ ]]; then sleep 15 && echo done; else sleep 5 && echo done_sleeping; fi`
2056 | PARTIAL_FAILURE | 35085        | integration-test-admin@shoreline.io  | host | timeout(20s) | `sleep 15; echo done`
2049 | SUCCEEDED       | 11115        | integration-test-admin@shoreline.io  | host | `sleep 10 && echo world`
2037 | SUCCEEDED       | 11125        | integration-test-admin@shoreline.io  | host | `sleep 10; echo "done statement 2"`

Filter Chaining

Combine multiple job filters by chaining them together using pipes.

For example, get all jobs executed by a specific user and status:

op>
jobs | user = "gabe@shoreline.io" | status = "running"
ID   | STATUS   | DURATION_MS  | USER              | OP_STMT
3524 | RUNNING  | 1.010232e+06 | gabe@shoreline.io | host | limit = 3 | timeout(3600s) | `long-running-script.sh`

Job Details

Get the details of a specific job with the jobs | id = <job_id> command.

op>
jobs | id = 300009192
ID | NAME                | STATUS    | TOTAL_DURATION_MS | EXECUTION_TIME_MS | REQUEST
65 | i-00f9034231ac2bec0 | COMPLETED | 135               | 6                 | metric_query
65 | i-00f9034231ac2bec0 | COMPLETED | 6225              | 6028              | sleep 5 && echo "slept 5 on filtered pods"
66 | i-0ba70a62ee3b38683 | COMPLETED | 135               | 5                 | metric_query
66 | i-0ba70a62ee3b38683 | COMPLETED | 6229              | 6026              | sleep 5 && echo "slept 5 on filtered pods"
67 | i-00e36a7da4ff269b1 | COMPLETED | 135               | 5                 | metric_query
67 | i-00e36a7da4ff269b1 | COMPLETED | 6223              | 6026              | sleep 5 && echo "slept 5 on filtered pods"

ID  | TYPE      | NAME                                 | REGION    | AZ         | STATUS | STDOUT
79  | CONTAINER | alpha-cust.shoreline-r4ncg.shoreline | us-west-2 | us-west-2a |   0    | slept 5 on filtered pods
93  | CONTAINER | alpha-cust.shoreline-7d25h.shoreline | us-west-2 | us-west-2c |   0    | slept 5 on filtered pods
101 | CONTAINER | alpha-cust.shoreline-drwch.shoreline | us-west-2 | us-west-2b |   0    | slept 5 on filtered pods

The first table shows each Agent-installed Resource on which the command was executed, along with each execution step and its corresponding request. For example, we can see that the Metric query filter only took a few milliseconds, while the intentionally-lengthy sleep command took much longer.

The second table shows the standard Op statement table with the actual Resources targeted by the Resource query, on which the command was executed. We see the container Resources and the stdout returned by the above example.

Examples

Metric Query Dynamic Filters

With Dynamic Filters you can filter by Metric queries to determine which Resources to target with your Linux command.

For example, we're filtering pods to get only those with at least 80% CPU usage, then draining those nodes.

op>
pods | app="shoreline" | filter(pod_cpu_usage > 80) | `grace_period=30 timeout=60 /agent/scripts/graceful-k8s-instance-termination.sh`
ID | TYPE      | NAME                                 | REGION    | AZ
53 | CONTAINER | test5-cust.shoreline-l8srx.shoreline | us-west-2 | us-west-2b
76 | CONTAINER | test5-cust.shoreline-6fmhz.shoreline | us-west-2 | us-west-2a
Some commands were backgrounded.
Use 'jobs' to view all jobs.
Use 'jobs | id = 4397' to view detailed info for this job.
Filters excluded 0/3 hosts, 1/3 pods, 1/3 containers

Viewing the job details shows all three matching pod Resources. The Linux command is not issued to the filtered-out pod. However, the two passing pods are currently processing via a background job.

op>
jobs | id = 4397
ID | NAME                | STATUS       | TOTAL_DURATION_MS | EXECUTION_TIME_MS | REQUEST
1  | i-0e95acb5362fedaf5 | COMPLETED    | 151               | 6                 | metric_query
1  | i-0e95acb5362fedaf5 | BACKGROUNDED | 4320              | 0                 | grace_period=30 timeout=60 /agent/scripts/graceful-k8s-instance-termination.sh
2  | i-0d66949e63d67dd3e | COMPLETED    | 151               | 5                 | metric_query
2  | i-0d66949e63d67dd3e | COMPLETED    | 232               | 0                 | grace_period=30 timeout=60 /agent/scripts/graceful-k8s-instance-termination.sh
3  | i-0a626b11673eb6368 | COMPLETED    | 151               | 6                 | metric_query
3  | i-0a626b11673eb6368 | BACKGROUNDED | 4320              | 0                 | grace_period=30 timeout=60 /agent/scripts/graceful-k8s-instance-termination.sh

ID | TYPE      | NAME                                 | REGION    | AZ
53 | CONTAINER | test5-cust.shoreline-l8srx.shoreline | us-west-2 | us-west-2b
76 | CONTAINER | test5-cust.shoreline-6fmhz.shoreline | us-west-2 | us-west-2a
Filters excluded 0/3 hosts, 1/3 pods, 1/3 containers

Finally, after the Linux command completes the two filtered Resources return the expected 0 status code.

op>
jobs | id = 4397
ID | NAME                | STATUS    | TOTAL_DURATION_MS | EXECUTION_TIME_MS | REQUEST
1  | i-0e95acb5362fedaf5 | COMPLETED | 151               | 6                 | metric_query
1  | i-0e95acb5362fedaf5 | COMPLETED | 20266             | 20033             | grace_period=30 timeout=60 /agent/scripts/graceful-k8s-instance-termination.sh
2  | i-0d66949e63d67dd3e | COMPLETED | 151               | 5                 | metric_query
2  | i-0d66949e63d67dd3e | COMPLETED | 232               | 0                 | grace_period=30 timeout=60 /agent/scripts/graceful-k8s-instance-termination.sh
3  | i-0a626b11673eb6368 | COMPLETED | 151               | 6                 | metric_query
3  | i-0a626b11673eb6368 | COMPLETED | 24259             | 24026             | grace_period=30 timeout=60 /agent/scripts/graceful-k8s-instance-termination.sh

ID | TYPE      | NAME                                 | REGION    | AZ         | STATUS | STDOUT
53 | CONTAINER | test5-cust.shoreline-l8srx.shoreline | us-west-2 | us-west-2b |   0    |
76 | CONTAINER | test5-cust.shoreline-6fmhz.shoreline | us-west-2 | us-west-2a |   0    |
Filters excluded 0/3 hosts, 1/3 pods, 1/3 containers

Incremental Results for Chained Statements

Consider the following Op command that combines two long-running statements using a dynamic Linux command filter.

op>
hosts | limit = 3 | filter(`sleep 10 && echo "slept 10 sec"`) | `sleep 10 && echo "slept 10 more sec"`
ID | TYPE | NAME                | REGION    | AZ
1  | HOST | i-0023d0079f68f135c | us-west-2 | us-west-2b
2  | HOST | i-056c982a4438c424b | us-west-2 | us-west-2a
3  | HOST | i-0a9da380d1e52cd8f | us-west-2 | us-west-2a
Some commands were backgrounded.
Use 'jobs' to view all jobs.
Use 'jobs | id = 5456' to view detailed info for this job.

The Agent recognizes two long-running commands, the latter of which depends on completing the former. Therefore, getting job details five seconds after invoking the initial Op command shows that all statements are currently BACKGROUNDED.

op>
jobs | id = 5456
ID | NAME                | STATUS       | TOTAL_DURATION_MS | EXECUTION_TIME_MS | REQUEST
1  | i-0023d0079f68f135c | BACKGROUNDED | 5271              | 0                 | sleep 10 && echo "slept 10 sec"
1  | i-0023d0079f68f135c | BACKGROUNDED | 5271              | 0                 | sleep 10 && echo "slept 10 more sec"
2  | i-056c982a4438c424b | BACKGROUNDED | 5271              | 0                 | sleep 10 && echo "slept 10 sec"
2  | i-056c982a4438c424b | BACKGROUNDED | 5271              | 0                 | sleep 10 && echo "slept 10 more sec"
3  | i-0a9da380d1e52cd8f | BACKGROUNDED | 5271              | 0                 | sleep 10 && echo "slept 10 sec"
3  | i-0a9da380d1e52cd8f | BACKGROUNDED | 5271              | 0                 | sleep 10 && echo "slept 10 more sec"

After 10 seconds, the initial statement completes, and the EXECUTION_TIME_MS shows how long that statement ran.

op>
jobs | id = 5456
ID | NAME                | STATUS       | TOTAL_DURATION_MS | EXECUTION_TIME_MS | REQUEST
1  | i-0023d0079f68f135c | COMPLETED    | 11132             | 11041             | sleep 10 && echo "slept 10 sec"
1  | i-0023d0079f68f135c | BACKGROUNDED | 11597             | 0                 | sleep 10 && echo "slept 10 more sec"
2  | i-056c982a4438c424b | COMPLETED    | 11164             | 11078             | sleep 10 && echo "slept 10 sec"
2  | i-056c982a4438c424b | BACKGROUNDED | 11597             | 0                 | sleep 10 && echo "slept 10 more sec"
3  | i-0a9da380d1e52cd8f | COMPLETED    | 11116             | 11039             | sleep 10 && echo "slept 10 sec"
3  | i-0a9da380d1e52cd8f | BACKGROUNDED | 11597             | 0                 | sleep 10 && echo "slept 10 more sec

After 20 seconds, both statements are complete, and the final output from the queried Resources returns.

op>
jobs | id = 5456
ID | NAME                | STATUS    | TOTAL_DURATION_MS | EXECUTION_TIME_MS | REQUEST
1  | i-0023d0079f68f135c | COMPLETED | 11132             | 11041             | sleep 10 && echo "slept 10 sec"
1  | i-0023d0079f68f135c | COMPLETED | 22208             | 11032             | sleep 10 && echo "slept 10 more sec"
2  | i-056c982a4438c424b | COMPLETED | 11164             | 11078             | sleep 10 && echo "slept 10 sec"
2  | i-056c982a4438c424b | COMPLETED | 22259             | 11076             | sleep 10 && echo "slept 10 more sec"
3  | i-0a9da380d1e52cd8f | COMPLETED | 11116             | 11039             | sleep 10 && echo "slept 10 sec"
3  | i-0a9da380d1e52cd8f | COMPLETED | 22231             | 11037             | sleep 10 && echo "slept 10 more sec"

ID | TYPE | NAME                | REGION    | AZ         | STATUS | STDOUT
1  | HOST | i-0023d0079f68f135c | us-west-2 | us-west-2b |   0    | slept 10 more sec
2  | HOST | i-056c982a4438c424b | us-west-2 | us-west-2a |   0    | slept 10 more sec
3  | HOST | i-0a9da380d1e52cd8f | us-west-2 | us-west-2a |   0    | slept 10 more sec

Timeout

Background jobs and the timeout function work well together.

For example, imagine your container Resources have a long.sh and short.sh shell script. long.sh may take a long time to execute, while short.sh requires about five seconds.

In scenarios where you don't want the long.sh to execute for too long, you can use the timeout function. For example, below, we're executing long.sh on bookstore-named containers and short.sh on the others.

op>
containers | limit = 5 | timeout(10s) | `if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then long.sh; else short.sh && echo done; fi`
ID | TYPE      | NAME                                                    | REGION    | AZ
16 | CONTAINER | bookstore.bookstore-dc9468886-7ggnj.istio-proxy         | us-west-2 | us-west-2c
18 | CONTAINER | bookstore.bookstore-dc9468886-7ggnj.bookstore-container | us-west-2 | us-west-2c
20 | CONTAINER | bookstore.bookstore-dc9468886-kd79m.istio-proxy         | us-west-2 | us-west-2c
22 | CONTAINER | bookstore.bookstore-dc9468886-kd79m.bookstore-container | us-west-2 | us-west-2c
24 | CONTAINER | kube-system.fluentd-m69pw.fluentd                       | us-west-2 | us-west-2c
Some commands were backgrounded.
Use 'jobs' to view all jobs.
Use 'jobs | id = 2896' to view detailed info for this job.

Some of the statements are taking longer than the background_timeout setting, so a background job is generated.

After about five seconds, we check the job details and see that the short.sh script is complete on three containers, but the other two are still BACKGROUNDED.

op>
jobs | id = 2896
ID | NAME                | STATUS       | TOTAL_DURATION_MS | EXECUTION_TIME_MS | REQUEST
2  | i-0d66949e63d67dd3e | COMPLETED    | 6147              | 6025              | if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then long.sh; else short.sh && echo done; fi
2  | i-0d66949e63d67dd3e | BACKGROUNDED | 7246              | 0                 | if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then long.sh; else short.sh && echo done; fi
2  | i-0d66949e63d67dd3e | COMPLETED    | 6153              | 6028              | if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then long.sh; else short.sh && echo done; fi
2  | i-0d66949e63d67dd3e | BACKGROUNDED | 7246              | 0                 | if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then long.sh; else short.sh && echo done; fi
2  | i-0d66949e63d67dd3e | COMPLETED    | 6154              | 6027              | if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then long.sh; else short.sh && echo done; fi

ID | TYPE      | NAME                                                    | REGION    | AZ         | STATUS | STDOUT
16 | CONTAINER | bookstore.bookstore-dc9468886-7ggnj.istio-proxy         | us-west-2 | us-west-2c |   0    | done
18 | CONTAINER | bookstore.bookstore-dc9468886-7ggnj.bookstore-container | us-west-2 | us-west-2c | ---    | ---
20 | CONTAINER | bookstore.bookstore-dc9468886-kd79m.istio-proxy         | us-west-2 | us-west-2c |   0    | done
22 | CONTAINER | bookstore.bookstore-dc9468886-kd79m.bookstore-container | us-west-2 | us-west-2c | ---    | ---
24 | CONTAINER | kube-system.fluentd-m69pw.fluentd                       | us-west-2 | us-west-2c |   0    | done

After ten seconds has passed all five background jobs are COMPLETED, and we see that the long.sh did not complete before the background_timeout period, terminating the process.

op>
jobs | id = 2896
ID | NAME                | STATUS    | TOTAL_DURATION_MS | EXECUTION_TIME_MS | REQUEST
2  | i-0d66949e63d67dd3e | COMPLETED | 6147              | 6025              | if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then long.sh; else short.sh && echo done; fi
2  | i-0d66949e63d67dd3e | COMPLETED | 11154             | 11031             | if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then long.sh; else short.sh && echo done; fi
2  | i-0d66949e63d67dd3e | COMPLETED | 6153              | 6028              | if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then long.sh; else short.sh && echo done; fi
2  | i-0d66949e63d67dd3e | COMPLETED | 11158             | 11032             | if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then long.sh; else short.sh && echo done; fi
2  | i-0d66949e63d67dd3e | COMPLETED | 6154              | 6027              | if [[ "$ACTION_CONTAINER_NAME" =~ (bookstore) ]]; then long.sh; else short.sh && echo done; fi

ID | TYPE      | NAME                                                    | REGION    | AZ         | STATUS | STDOUT | MSG
16 | CONTAINER | bookstore.bookstore-dc9468886-7ggnj.istio-proxy         | us-west-2 | us-west-2c |   0    | done   |
18 | CONTAINER | bookstore.bookstore-dc9468886-7ggnj.bookstore-container | us-west-2 | us-west-2c |  124   |        | Command timed out after 10 sec and was killed
20 | CONTAINER | bookstore.bookstore-dc9468886-kd79m.istio-proxy         | us-west-2 | us-west-2c |   0    | done   |
22 | CONTAINER | bookstore.bookstore-dc9468886-kd79m.bookstore-container | us-west-2 | us-west-2c |  124   |        | Command timed out after 10 sec and was killed
24 | CONTAINER | kube-system.fluentd-m69pw.fluentd                       | us-west-2 | us-west-2c |   0    | done   |