Get AWS data into Observe

Before you start

AWS offers many way to collect and store data. Make sure your data can be sent to Observe:

• If you have a global config bucket, make sure you deploy the stack to centralized management account such as Control Tower that has access to the global config bucket.

• If you are already sending events to another service, you must configure an EventBridge to also send those events to Observe.

• Manually upload a test file to the config bucket and verify that it gets piped into Observe.

Verify the following permissions are set:

• The Observe IAM role has the necessary kms:decrypt policy attached, and that the policy applies to the config bucket resource and files inside.

• Ensure that the Control Tower KMS key policy allows for kms:decrypt.

For example:

{
  "Sid": "Enable Decrypt for Observe",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::<AWS-ACCOUNT>:role/<“STACK_NAME>
  },
  "Action": "kms:Decrypt",
  "Resource": "*",
}

Use the Add Data portal to get your AWS data into Observe

Perform the following steps to get data from your AWS environment into Observe:

1. In Observe, select Add Data > AWS.

2. Provide your AWS Account ID, then click Continue.

3. On the Configure metrics screen, decide how you want to collect metrics: use Metric Stream to push data into Observe from CloudWatch, or use API Polling to pull metrics into Observe. See AWS data collection to compare the options and decide which one is better for you.

4. Select the AWS service you want to collect data from, then click Edit metrics for that service to identify the metrics you want to collect.

5. Configure logs and AWS resources by providing inclusion and exclusion patterns for the logs you want to ingest.

4. (Optional) Forward files from S3 buckets.

5. Click Continue.

6. Click Go to AWS and create stack. This will use the template to automatically configure the required AWS IAM role, associated policies, and AWS services necessary for data collection. Wait until you see the CREAT_COMPLETE status in AWS. This can take several minutes.

7. Return to Observe and confirm the CloudFormation stack creation, then click Continue.

8. Wait a few minutes, then click View in each tab to verify you are receiving logs, metrics, and resources data. When you are done verifying, click Finish.

Use Terraform to get your AWS data into Observe

You can use Terraform to ingest AWS data into Observe.

• To push AWS logs, configs, and metrics data to Observe, use the Observe stack module.

• To pull metrics data into Observe, configure a CloudWatch metrics poller.