AWS CloudTrail Lake Integration

v11.0

Automatically transmit Shoreline activity to your AWS CloudTrail Lake event stream.

AWS CloudTrail records AWS-related activities, such as user, role, or AWS service events. Shoreline's AWS CloudTrail Lake integration allows you to record Shoreline Alarm and Bot Events within AWS CloudTrail Lake.

Getting started

To get started with the Shoreline CloudTrail Lake integration you'll need to follow a few simple steps.

1. Create a CloudTrail Lake event data store
A CloudTrail Lake event data store contains all Shoreline activity for easy SQL-based querying.
2. Create an integration
Shoreline uses the CloudTrail channel to transmit audit log events to CloudTrail.
3. Enable the Shoreline CloudTrail Lake integration
Learn how to adjust your Shoreline configuration and enable the integration.

Create an event data store (optional)

Create an event data store in CloudTrail Lake to log Shoreline events to.

You'll find instructions for both the AWS Console and the AWS CLI below.

Navigate to the AWS CloudTrail > Lake > Event data stores page.
Click on the button.
Enter a data store name in the Event data store name field.
Enter your desired retention period in the Retention period field.
(Optionally) Tick any other desired fields and add tags.
Click the button.
Select Events from integrations in the Event type section.
In the Events from integration section, select Shoreline.io from the Source dropdown field.
(Optionally) Enter a new channel name in the Channel name field.
(Optionally) Click the button to add an AWS account to the channel.
Click the button.
Verify your configuration and click the button to create your new data store.

Create an integration

You can create an integration with Shoreline by using either the AWS Console or the AWS CLI.

Navigate to the AWS CloudTrail > Lake > Integrations page.
Click on the button.
Enter a channel name in the Channel name field.
Select Shoreline.io in the Source dropdown field.
Under Event delivery location, select the Event data store you'll send Shoreline events to.

You can select an existing event data store or create a new one.
Select Use existing event data store in the Choose delivery location radio field.

Find and tick the event data store you want to use in the Event data stores dropdown field.
(Optionally) Click the button to add a particular AWS account as a principal in the resource policy for this integration's channel.
Click the button to create your Shoreline integration.

Enable the CloudTrail Lake integration

To enable the Shoreline CloudTrail Lake integration you must update your Shoreline Agent configuration with a few CLOUDTRAIL_ environment variables.

If you configured Shoreline using a Kubernetes DaemonSet then you must add the Shoreline CloudTrail environment variables to the Shoreline Agent YAML, as seen below.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: shoreline
  name: shoreline
  namespace: shoreline
spec:
  selector:
    matchLabels:
      app: shoreline
  template:
    metadata:
      labels:
        app: shoreline
    spec:
      containers:
        - env:
            - name: CLOUDTRAIL_CHANNEL_ARN
              value: 'arn:aws:cloudtrail:us-east-1:12345678910:channel/EXAMPLE7f40-71e8-4df9-9b2a-c4882aEXAMPLE'
            - name: CLOUDTRAIL_LAKE_ENABLED
              value: true
            - name: CLOUDTRAIL_LAKE_REGION
              value: 'us-east-1'

CLOUDTRAIL_CHANNEL_ARN

Set the CLOUDTRAIL_CHANNEL_ARN variable to the CloudTrail channel ARN you want Shoreline to send events to.

CLOUDTRAIL_LAKE_REGION

This optional value determines the AWS region with your target CloudTrail data lake.

By default, Shoreline uses the same AWS region as the Shoreline Agent.

CLOUDTRAIL_LAKE_ENABLED

Set this value to true to enable the Shoreline CloudTrail Lake integration.

Usage

Once enabled, the Shoreline CloudTrail Lake integration automatically ingests the following Shoreline audit event types:

Alarm trigger events
Bot trigger events

For example, Shoreline sends a new ActivityLog event to AWS CloudTrail whenever an Alarm fires, which is automatically ingested and available in your configured data store.

The JSON output below shows what a typical Shoreline audit event looks like in your AWS CloudTrail Lake.

[
  {
    "eventVersion": "1.08",
    "eventCategory": "ActivityAuditLog",
    "eventType": "ActivityLog",
    "eventID": "905419df-abe9-4cf9-bdb0-b01e02b99cb1",
    "eventTime": "2022-09-02 13:59:43.000",
    "awsRegion": "us-east-1",
    "addendum": "",
    "metadata": "{
      ingestiontime=2022-09-02 13:59:56.000,
      channelarn=null
    }",
    "eventData": "{
      version=1, 
      useridentity={
        type=, 
        principalid=Shoreline, 
        details=null
      },
      useragent=null,
      eventsource=shoreline,
      eventname=delete_file,
      eventtime=2022-09-02 13:59:43.000,
      uid=1662127183_4294967318_72_1_300003392_0,
      requestparameters=null,
      responseelements=null,
      errorcode=null,
      errormessage=null,
      sourceipaddress=null,
      recipientaccountid=123456789012,
      additionaleventdata={
        operationStatus=SUCCESS,
        resourceQuery=,
        entityName=delete_file,
        resourceData={
          \"resourceId\":\"72\",
          \"resourceName\":\"alpha-cust.shoreline-mz5l7\"
        },
        customerId=test,
        operationType=TRIGGER,
        operationSource=SHORELINE,
        info=cmd: rm /tmp/test_2.txt
          param_values: \"{
            \\\"ALARM_POD_NAME\\\":\\\"shoreline-mz5l7\\\",
            \\\"ALARM_NODE_NAME\\\":\\\"ip-10-36-173-34.us-west-2.compute.internal\\\",
            \\\"ALARM_NAMESPACE\\\":\\\"alpha-cust\\\",
            \\\"ALARM_INSTANCE_ID\\\":\\\"i-031ea6c70db5aa4ac\\\",
            \\\"ALARM_CONTAINER_NAME\\\":\\\"shoreline\\\",
            \\\"ACTION_POD_NAME\\\":\\\"shoreline-mz5l7\\\",
            \\\"ACTION_NODE_NAME\\\":\\\"ip-10-36-173-34.us-west-2.compute.internal\\\",
            \\\"ACTION_NAMESPACE\\\":\\\"alpha-cust\\\",
            \\\"ACTION_INSTANCE_ID\\\":\\\"i-031ea6c70db5aa4ac\\\",
            \\\"ACTION_CONTAINER_NAME\\\":\\\"shoreline\\\"
          }\"
          exit_status: 0
          stdout: 
          stderr: , 
        resourceType=
      }
    }",
    "recipientAccountId": "123456789012"
  }
]

Read on to learn more about querying Shoreline events.

Query Shoreline events

To retrieve data from AWS CloudTrail Lake you'll use an SQL-based query language.

Navigate to the AWS CloudTrail > Lake > Query page.
Select the Editor tab.
Under Event data store, select your connected event data store.
Enter the following query in the Query 1 field, replacing <EVENT_DATA_STORE_ID> with the copyable Event data store ID you selected previously.
```
SELECT
    *
FROM
    <EVENT_DATA_STORE_ID>
WHERE
    eventData.eventsource = 'shoreline'
```
Click the Run button to execute the query.
Select the Query results tab to view the event results.

Check out the Run a query page of the AWS documentation for more details.

Example queries

Below are some example AWS CloudTrail Lake queries you can try to work with Shoreline data.

Get the Shoreline Resource ID for each event.

SELECT
    json_extract_scalar(element_at(eventData.additionaleventdata, 'resourceData'), '$.resourceId') as shorelineResourceId,
    *
FROM
    <EVENT_DATA_STORE_ID>
WHERE
    eventData.eventsource = 'shoreline'

Important event fields

The following contains a list of some of the most important fields you'll find within the ingested Shoreline data found in AWS CloudTrail Lake.

eventDataparent

Shoreline audit event data.

eventData.useridentityparent

eventData.useridentity.principalidstring

The Shoreline User that triggered this event. Defaults to Shoreline for auto-triggered events.

Example

eventData:
  useridentity:
    principalid: "Shoreline"

eventData.eventsourcestring

Where the event sourced from. Should always be shoreline for Shoreline-related events.

Example

eventData:
  eventsource: "shoreline"

eventData.eventnamestring

The name of the Shoreline object that triggered the event, such as the Alarm or Bot.

Example

eventData:
  eventname: "delete_file"

eventData.eventtimestring

Timestamp of when the Shoreline event occurred.

Example

eventData:
  eventtime: "2022-09-02 13:59:43.000"

eventData.uidstring

Unique Shoreline Event ID.

Example

eventData:
  uid: "1662127183_4294967318_72_1_300003392_0"

eventData.additionaleventdataparent

eventData.additionaleventdata.operationStatusstring

The event status. Check out Audit: Status for more info.

Example

eventData:
  additionaleventdata:
    operationStatus: "CANCELLED"

eventData.additionaleventdata.infostring

Assorted information related to the triggering event. For example, this field contains the executed Action: command for logged Action events.

Example

eventData:
  additionaleventdata:
    info: "cmd: rm /tmp/test_2.txt"

eventData.additionaleventdata.resourceDataparent

Resource-specific information.

eventData.additionaleventdata.resourceData.resourceIdstring

The internal Shoreline Resource ID of the triggering object.

Example

eventData:
  additionaleventdata:
    resourceData:
      resourceId: 72

eventData.additionaleventdata.resourceData.resourceNamestring

The Resource name of the triggering object.

Example

eventData:
  additionaleventdata:
    resourceData:
      resourceName: "alpha-cust.shoreline-mz5l7"

Disable the CloudTrail Lake integration

To disable the Shoreline CloudTrail Lake integration simply set the CLOUDTRAIL_LAKE_ENABLED environment variable to false in your Shoreline Agent configuration.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: shoreline
  name: shoreline
  namespace: shoreline
spec:
  selector:
    matchLabels:
      app: shoreline
  template:
    metadata:
      labels:
        app: shoreline
    spec:
      containers:
        - env:
            - name: CLOUDTRAIL_LAKE_ENABLED
              value: false

Troubleshooting

Incomplete integration status

If your integration isn't working it may be in an incomplete state. To check the status of your integration:

Navigate to the AWS CloudTrail > Lake > Integrations page.
Check the Status column for the Shoreline integration you want to troubleshoot.

If the Status shows as Incomplete it may indicate that the channel ARN is not properly configured in your Shoreline Agent environment variables.

Please review your Agent configuration for CloudTrail Lake in the Enable the CloudTrail Lake integration section.

If your Agents are configured correctly, please contact Shoreline Support with your channel ARN.

Slow event ingestion

It may take a few minutes for recent Shoreline events to appear in your AWS CloudTrail Lake event data store. Please wait a few minutes and check again.

AWS CloudTrail Lake Integration

Getting started

1. Create a CloudTrail Lake event data store

2. Create an integration

3. Enable the Shoreline CloudTrail Lake integration

Create an event data store (optional)

Create an integration

Enable the CloudTrail Lake integration

CLOUDTRAIL_CHANNEL_ARN

How to find your CloudTrail Lake channel ARN

CLOUDTRAIL_LAKE_REGION

CLOUDTRAIL_LAKE_ENABLED

Usage

Query Shoreline events

Example queries

Important event fields

Disable the CloudTrail Lake integration

Troubleshooting

Incomplete integration status

How to find your CloudTrail Lake channel ARN

Slow event ingestion

AWS CloudTrail Lake Integration

Getting started

1. Create a CloudTrail Lake event data store →

2. Create an integration →

3. Enable the Shoreline CloudTrail Lake integration →

Create an event data store (optional)

Create an integration

Enable the CloudTrail Lake integration

CLOUDTRAIL_CHANNEL_ARN

How to find your CloudTrail Lake channel ARN

CLOUDTRAIL_LAKE_REGION

CLOUDTRAIL_LAKE_ENABLED

Usage

Query Shoreline events

Example queries

Important event fields

Disable the CloudTrail Lake integration

Troubleshooting

Incomplete integration status

How to find your CloudTrail Lake channel ARN

Slow event ingestion

1. Create a CloudTrail Lake event data store

2. Create an integration

3. Enable the Shoreline CloudTrail Lake integration