Amazon S3

Two types of data may be ingested from or about S3 buckets:

• Data files in an S3 bucket

• S3 bucket access logs

Ingesting data files from an S3 bucket

Ingest objects uploaded to an S3 bucket using the Observe Lambda forwarder and Amazon S3 Event Notifications.

Warning

If your notification writes to the same bucket that triggers the notification, it could cause an execution loop. For example, if the bucket triggers a Lambda function each time an object is uploaded, and the function uploads an object to the bucket, then the function indirectly triggers itself. To avoid this, use two buckets, or configure the trigger to only apply to a prefix used for incoming objects.

Granting S3 permissions to publish event notifications to Lambda

To publish event notification messages, the Amazon S3 principal must be able to call the API and publish messages to the Observe Lambda Forwarder. These permissions are configured for you when you enable event notifications on a bucket, described below. (For more information, see Granting permissions to invoke an AWS Lambda function in the AWS documentation.)

Enabling notifications in the S3 console

1. Navigate to S3 in the AWS Console.

2. Select the bucket that you want to forward data.

3. Click on Properties.

4. Under Event notifications, click Create event notification.

5. In the General configuration section.

   • Enter a description in Event name. If not provided, AWS generates a globally unique identifier (GUID) to use for the name.

   • If desired, provide a Prefix to filter event notifications by prefix. For example, you may use a prefix filter to receive notifications only when files are added to a specific folder (like images/.)

   • Similarly, filter event notifications by suffix by providing a value for Suffix. (Optional)

   For more information, see Configuring event notifications using object key name filtering.

6. Under Event types, select the event types to receive notifications.

   • Observe recommends All object create events.

7. In the Destination section.

   • Choose Lambda function as the event notification destination.

   • In the Lambda function dropdown, choose the name of your Observe Lambda Forwarder function.

8. Click Save.

See the AWS S3 Documentation for full details.

Granting the Observe Lambda Forwarder permissions to access your S3 Bucket

A Lambda function has a policy, called an execution role, that grants permission to access AWS services and resources. In order to GET Objects out of an S3 bucket in response to an Event Notification, your Observe Lambda Forwarder must have permission to access the S3 bucket.

1. Navigate to Lambda in the AWS Console.

2. Select the Observe Lambda function (created by the forwarder or integration installation process).

3. Select the Configuration tab.

   • Select Permissions from the left menu.

   • Under Execution Role, choose the Role name. This displays the role details in a new IAM console tab.

   

   Figure 1 - Lambda Permissions Configuration

4. In the Permissions tab, click on AllowS3Read policy. If you don’t see this policy, click Show more to show hidden policies.

   • Click Edit policy and then the JSON tab.

   

   Figure 2 - Editing the Lambda policy

   • Add the following snippet under the Resource section for each S3 bucket you wish to forward events:

     Example:

     {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Action": [
                 "s3:Get*",
                 "s3:List*"
             ],
             "Resource": [
                 "arn:aws:s3:::observe-collection-bucket-123abc",
                 "arn:aws:s3:::observe-collection-bucket-123abc*",
                 "arn:aws:s3:::additional-bucket-1",
                 "arn:aws:s3:::additional-bucket-1*",
                 "arn:aws:s3:::log-bucket-2",
                 "arn:aws:s3:::log-bucket-2*"
             ],
             "Effect": "Allow"
         }
     ]
     }
     

5. Click Review Policy.

6. Click Save changes.

For each log bucket (“Target bucket”), add a trigger so the forwarder can send new files as they generate.

S3 bucket access logs

Enabling S3 access logging

S3 bucket access logging is disabled by default. If needed, first enable logging for the desired bucket:

1. Navigate to S3 in the AWS Console.

2. Select the bucket you want to access logs.

3. Choose Properties.

4. Under Server access logging, choose Edit.

5. Select Enable and provide the log destination bucket in Target bucket.

6. Choose Save changes.

Figure 3 - Editing server access logging in the AWS Console

See the AWS access logging documentation for full details.

Forwarding logs using Lambda

If necessary, install the Observe AWS Integration or the standalone Observe Lambda forwarder following the instructions in the documentation.

If you currently use the Lambda forwarder, you do not need to install it again. If you install it for the first time, consider using the AWS App to easily ingest additional AWS data.

Add a trigger for each log bucket, Target bucket, so the forwarder can send access logs as they generate.

1. Navigate to Lambda in the AWS Console.

2. Select the Observe Lambda function (created by the forwarder or integration installation process).

3. Select Add Trigger, then search for S3.

   

   Figure 4 - Adding the S3 trigger

4. Configure the trigger with the following settings:

   • Bucket - the log bucket

   • Event type - the desired events to send, such as All object create events

   • Prefix or Suffix if desired. (Optional)

5. Choose Add to save.

Note

S3 access logs may take some time to be created in the target bucket. For details, see the AWS documentation about best-effort delivery.