Amazon S3

There are two types of data that may be ingested from or about S3 buckets: access logs, with details of bucket activity (creates, deletes, etc.) and data contained in files that have been uploaded to a bucket. They both use the Observe Lambda Forwarder, part of the AWS Integration.

This page covers both types of sources:

Ingest data files from an S3 bucket

Ingest objects uploaded to an S3 bucket using the Observe Lambda forwarder and Amazon S3 Event Notifications.

Warning

If your notification writes to the same bucket that triggers the notification, it could cause an execution loop. For example, if the bucket triggers a Lambda function each time an object is uploaded, and the function uploads an object to the bucket, then the function indirectly triggers itself. To avoid this, use two buckets, or configure the trigger to only apply to a prefix used for incoming objects.

Grant S3 permissions to publish event notifications to Lambda

To publish event notification messages, the Amazon S3 principal must be able to call the API and publish messages to the Observe Lambda Forwarder. These permissions are configured for you when you enable event notifications on a bucket, described below. (For more information, see Granting permissions to invoke an AWS Lambda function in the AWS documentation.)

Enabling notifications in the S3 console

  1. Navigate to S3 in the AWS Console

  2. Select the bucket you’d like to forward data from

  3. Click on Properties

  4. Under Event notifications, click Create event notification

  5. In the General configuration section

    • Enter a description in Event name. If not provided, AWS generates a globally unique identifier (GUID) to use for the name.

    • If desired, provide a Prefix to filter event notifications by prefix. For example, you may use a prefix filter to receive notifications only when files are added to a specific folder (like images/.)

    • Similarly, filter event notifications by suffix by providing a value for Suffix. (Optional.)

    For more information, see Configuring event notifications using object key name filtering.

  6. Under Event types, select the event types you wish to receive notifications for.

    • We recommend All object create events

  7. In the Destination section

    • Choose Lambda function as the event notification destination.

    • In the Lambda function dropdown, choose the name of your Observe Lambda Forwarder function.

  8. Click Save

See the AWS S3 Documentation for full details.

Granting the Observe Lambda Forwarder permissions to access your S3 Bucket

A Lambda function has a policy, called an execution role, that grants it permission to access AWS services and resources. In order to GET Objects out of an S3 bucket in response to an Event Notification, your Observe Lambda Forwarder must have permissions to access that S3 bucket.

  1. Navigate to Lambda in the AWS Console

  2. Select the Observe Lambda function (created by the forwarder or integration installation process)

  3. Select the Configuration tab

    • Select Permissions on the left rail

    • Under Execution Role, click on the Role name. This displays the role details in a new IAM console tab.

    Lambda permissions configuration
  4. In the Permissions tab, click on AllowS3Read policy. If this policy is not visible, click Show more to show hidden policies.

    • Click Edit policy and then the JSON tab

    Editing Lambda policy in the UI
    • Add the following snippet under the Resource section for each S3 bucket you wish to forward events from

      Example:

      {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Action": [
                  "s3:Get*",
                  "s3:List*"
              ],
              "Resource": [
                  "arn:aws:s3:::observe-collection-bucket-123abc",
                  "arn:aws:s3:::observe-collection-bucket-123abc*",
                  "arn:aws:s3:::additional-bucket-1",
                  "arn:aws:s3:::additional-bucket-1*",
                  "arn:aws:s3:::log-bucket-2",
                  "arn:aws:s3:::log-bucket-2*"
              ],
              "Effect": "Allow"
          }
      ]
      }
      
  5. Click Review Policy

  6. Click Save changes

For each log bucket (“Target bucket”), add a trigger so the forwarder can send new files as they are generated.

S3 bucket access logs

Enable S3 access logging

S3 bucket access logging is disabled by default. If needed, first enable logging for the desired bucket:

  1. Navigate to S3 in the AWS Console

  2. Select the bucket you’d like to get access logs for

  3. Click on “Properties”

  4. Under “Server access logging”, click “Edit”

  5. Select “Enable” and provide the log destination bucket in “Target bucket”

  6. Click “Save changes”

    Editing server access logging in the AWS Console

See the AWS access logging documentation for full details.

Forward logs using Lambda

If needed, install the Observe AWS Integration or the standalone Observe Lambda forwarder following the instructions in the documentation.

If you are already using the Lambda forwarder, you do not need to install it again. If you are installig it for the first time, consider the AWS Integration to easily ingest additional AWS data.

For each log bucket (“Target bucket”), add a trigger so the forwarder can send access logs as they are generated.

  1. Navigate to Lambda in the AWS Console

  2. Select the Observe Lambda function (created by the forwarder or integration installation process)

  3. Select “Add Trigger”, then search for “S3”

    Type S3 in the form and select it to add an S3 trigger
  4. Configure the trigger with the following settings:

    • Bucket: the log bucket

    • Event type: the desired events to send, such as “All object create events”

    • Prefix or Suffix if desired (optional)

  5. Click “Add” to save.

Note

S3 access logs may take some time to be created in the target bucket. For details, see the AWS documentation about best-effort delivery.