Get AWS data into Observe by Snowflake
You can use the Add Data portal to get your AWS data into Observe by Snowflake, or you can use Terraform to push AWS logs, configs, and metrics to Observe by Snowflake, or pull metrics into Observe by Snowflake by configuring a CloudWatch poller.
Before you start
AWS offers many way to collect and store data. Make sure your data can be sent to Observe by Snowflake:
-
If you have a global config bucket, make sure you deploy the stack to centralized management account such as Control Tower that has access to the global config bucket. See AWS-at-scale data ingestion for more information about setting up a centralized account and using Control Tower.
-
If you are already sending events to another service, you must configure an EventBridge to also send those events to Observe by Snowflake.
-
Manually upload a test file to the config bucket and verify that it gets piped into Observe by Snowflake.
-
The role or user deploying the Observe by Snowflake stack must have permission to retrieve objects from public S3 buckets in the same AWS region as the deployment, as well as in the
us-west-2region.To validate the user or role is able to read and copy from a public bucket in
us-west-2, run the following command as that user or role:aws s3 cp s3://observeinc/cloudwatchmetrics/filters/default.yaml default.yamlIf the command fails, update the user or role permissions to allow the required access. You may also need to validate your SCP policies to ensure they do not include an explicit
Deny.
Verify the following permissions are set:
- The Observe by Snowflake IAM role has the necessary
kms:decryptpolicy attached, and that the policy applies to the config bucket resource and files inside. - Ensure that the Control Tower KMS key policy allows for
kms:decrypt.
For example:
{
"Sid": "Enable Decrypt for Observe",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<AWS-ACCOUNT>:role/<STACK_NAME>
},
"Action": "kms:Decrypt",
"Resource": "*",
}
Use the Add Data portal to get your AWS data into Observe by Snowflake
Perform the following steps to get data from your AWS environment into Observe by Snowflake:
- In the left navigation rail, select Data & integrations > Integrations.
- Click the AWS card.
- Click New integration.
- Follow the template to configure this integration.
Step 1. Initial Setup
Provide the following information:
| Field | Description |
|---|---|
| Name | This name will be used to determine the IAM role and other resources created by the module or stack. Once you click Continue, this name can't be changed, so choose it carefully as it will impact the identification of all associated resources. You can use the default suggested name, or type in your own. |
| Region | Pick the AWS region you will collect data for. |
| AWS Account ID | Log into your AWS account to retrieve the 12-digit AWS Account ID from which you will be collecting data. In your AWS console, your account ID appears with dashes, such as 1234-5678-9012. Enter your account ID without the dashes: 123456789012. |
Click Continue to go to the next step.
Step 2. Configure metrics
Configure how you want to collect CloudWatch metrics:
- In the Configure metrics section, decide how you want to collect metrics: use Metric Stream to push data into Observe by Snowflake from CloudWatch, or use API Polling to pull metrics into Observe by Snowflake. See AWS data collection to compare the options and decide which one is better for you.
- Slide the Attach resource tags toggle to the on position to add tags to all CloudWatch metrics prior to ingestion. This allows you to filter on AWS tags directly in Observe by Snowflake after the metrics are ingested. After enabling the this ability, configure how long to cache resource-tag lookups before refreshing them. The default is 600 seconds.
- Select the AWS service you want to collect data from, then click Edit metrics for that service to select the metrics you want to collect. See AWS metrics for a full listing of AWS services and metrics available.
- List the namespace containers you want to collect metrics from. See Namespaces in the AWS documentation for more information about namespaces.

Step 3. Configure logs and AWS resources
Slide the toggle in the Logs panel to the on position to automatically subscribe all your CloudWatch Log Groups to Observe by Snowflake. You can use the Inclusion patterns and Exclusion patterns fields to further restrict collection to a set of log group name prefixes or patterns in the CloudFormation template.
AWS limits the number of subscription filters that can be applied to a log group to 2 maximum. Please check if any of your log groups already have 2 filters applied. See Real-time processing of log data with subscriptions in the AWS documentation for more information.
Slide the toggle in the AWS Resources panel to the on position to install AWS Config and collect all resource types. If you are already collecting AWS Config, disable this option and subscribe your S3 bucket to our forwarder instead. You can fine tune collection to a subset of resource types in the CloudFormation template.

Step 4. (Optional) Forward files from S3 buckets
In the Forward files from S3 buckets panel, list the S3 buckets the Observe by Snowflake forwarder is allowed to read from. If you don't specify any buckets, the forwarder reads from all buckets. Make sure Amazon EventBridge is enabled for any S3 buckets you specify. See Enabling Amazon EventBridge in the AWS documentation for more information.
Click Continue.
Step 5. Create and verify CloudFormation stack
When you see this following modal, click Go to AWS and create stack.

This takes you to the Quick create stack page, which automatically uses the template to configure the required AWS IAM role, associated policies, and AWS services necessary for data collection. Scroll to the bottom of the page, acknowledge any transforms that may require access capabilities, then click Create stack.
Wait until you see the CREATE_COMPLETE status in AWS for all the stacks and events. This can take several minutes.

Step 6. Confirm CloudFormation stack creation in Observe by Snowflake
Return to Observe by Snowflake and confirm the CloudFormation stack creation. Click the checkbox in the modal, then click Continue.

Wait a few minutes, then click View in each tab to verify you are receiving logs, metrics, and resources data. When you are done verifying, click Finish.

Use Terraform to get your AWS data into Observe by Snowflake
You can use Terraform to ingest AWS data into Observe by Snowflake.
- To push AWS logs, configs, and metrics data to Observe by Snowflake, use the Observe by Snowflake stack module.
- To pull metrics data into Observe by Snowflake, configure a CloudWatch metrics poller.
Updated about 23 hours ago