Amazon Web Services (AWS)¶
The primary role that the AWS App plays is in seamlessly collecting CloudWatch Logs, CloudWatch Metrics, AWS Resources(Config), and other data from your AWS account and make these data useful for monitoring your AWS services.
Note
If you use the AWS App v0.59 or earlier, or use the Observe AWS Collection CloudFormation Templates, go to Amazon Web Services (AWS) Integration [Legacy]
Setup¶
Install the AWS App¶
On the left side navigation menu, click Applications
, go to the Install new
tab, and install the AWS App.
Collect AWS data¶
After the AWS App is installed, go to the Connections
tab.
Click
Get started
of thePush metrics, logs and resources from AWS region.
option.Put
AWS Account ID
andAWS Region
(e.g. us-west-2) to collect data from.Toggle
Enable Cloudwatch Logs
onToggle
Enable Cloudwatch Metricstream
onToogle
Enable Config
on. If you are already collecting AWS Config, toggle this option off and specify the S3 bucket name (e.g. config-bucket-091312542283) where you store AWS Config. You can find the S3 bucket name from the Settings page for AWS Config. Once you find the S3 bucket name, put it in theConfigDeliveryBucketName
field under theAWS Config
section in the CloudFormation template you will find in the next step.Optionally change the name which will be used as the name for CloudFormation stack or Terraform module in the next step.
Click
Create
Click
quick-create link
to install a pre-configured CloudFormation template in the AWS Console. This CloudFormation template is to create a required AWS IAM role, its policy, and necessary AWS services automatically for data collection. Once the CloudFormation stack is created successfully, data will be pushed to Observe.Complete the creating-stack steps in the AWS Console.
For CloudWatch Logs, you can further restrict collection to a set of log group name prefixes or patterns in the CloudFormation template. By default, it will collect logs from all log groups avaialble in your AWS account’s region you specified in Step 2.
For CloudWatch Metrics, You can provide further adjustments to collection in the CloudFormation template. You can check what metrics are being collected by default.
curl -o recommended.yaml https://observeinc.s3.us-west-2.amazonaws.com/cloudwatchmetrics/filters/recommended.yaml
For AWS Resources (Config), You can fine tune collection to a subset of resource types in the CloudFormation template.
Optionally, if you want to send files including CloudTrail events, VPC Flow logs, or other files you store in S3 buckets, you can provide a list of S3 bucket names in the
SourceBucketNames
field under theForwader Options
section in the CloudFormation template. The Forwader will read new files from the list of S3 buckets. Make sureAmazon EventBridge
is enabled for these S3 buckets. You can find theAmazon EventBridge
setting under the properties tab for S3 buckets.
If you want to filter metrics by tags or don’t want to use a metric stream for CloudWatch Metrics, you can pull metrics instead.
Note
If you already created a CloudFormation stack to collect logs, metrics, and resources, disable the existing metrics collection first. Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click Update
, click Use existing template
, empty the MetricStreamFilterUri
field under the CloudWatch Metrics section. Complete the rest of CloudFormation steps.
Click
Get started
of thePull CloudWatch Metrics from AWS
option.Put
AWS Account ID
andAWS Region
(e.g. us-west-2) to collect data from.Change
Interval Duration
to adject a collection interval. Default is 5 minutes.List AWS namespaces (e.g. AWS/EC2, AWS/Lambda) you’d like to collect metrics from.
Optionally change the name which will be used as the name for CloudFormation stack or Terraform module in the next step.
List of resource tags to filter metrics for.The supported format is
or =value1,value2,value3. For exmaple, if you want to collect metrics that include Environment
as a tag key, you can putEnvironment
here. If you want to collect metrics that includeEnvironment=Prod
orEnvironment=Staging
, you can putEnvironment=Prod,Staging
here.Click
Create
Click
quick-create link
to install a pre-configured CloudFormation template in the AWS Conosle. This CloudFormation template is to create a required AWS IAM role and its policy automatically for metric collection. Once the role and policy are created successfully, Observe poller will pull CloudWatch metrics based on your selections above.Complete the creating-stack steps in the AWS Console.
If you want to send files including CloudTrail events, VPC Flow logs, or other files you store in S3 buckets, this option helps you forward new fiels to Observe easily. The Forwader will read new files from the list of S3 buckets. Make sure Amazon EventBridge
is enabled for these S3 buckets. You can find the Amazon EventBridge
setting under the properties tab for S3 buckets.
Click
Get started
of theForward data from S3
option.Put
AWS Account ID
andAWS Region
(e.g. us-west-2) to collect data from.Optionally change the name which will be used as the name for CloudFormation stack or Terraform module in the next step.
Put a list of bucket names the forwarder will be allowed to read from.
Click
Create
Click
quick-create link
to install a pre-configured CloudFormation template in the AWS Console. This CloudFormation template is to create a required AWS IAM role, its policy, and necessary AWS services automatically for data collection. Once the CloudFormation stack is created successfully, data will be pushed to Observe.Complete the creating-stack steps in the AWS Console.
FAQ¶
How do I troubleshoot AWS data collection?¶
First, go to the CloudFormation in the AWS console and check whether CloudFormation stacks are successfully created.
Second, you can get traces for the AWS data collection by following the steps below.
Go to the AWS datastream and create a token.
Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click
Update
, clickUse existing template
, setDebugEndpoint
tohttps://<token>@<customerId>.collect.observeinc.com/v2/otel
(e.g. https://ds1Xd63TEfS5dYodBSyf:OaAdWXaFGaLlx2wdCjzhL5ih06-redacted@123456789.collect.observeinc.com/v2/otel) under theDebugging Options
section.Go to the AWS datastream and click on the token you created in Step 1.
Click
Open dataset
.
If you are facing issues or pulling metrics, use the in-product Contact Support button to contact Observe for more information. On the left side navigation menu, click Docs & Support, Contact Support, and Send Us a Message to contact an Observe Data Engineer.
How do I filter metrics?¶
Observe offers two ways to collect metrics:
Push-based data collection uses a CloudWatch metric stream and supports filtering metrics by namespace (e.g.
AWS/EC2
,AWS/RDS
) or by metric name (e.g.CPUUtilization
)Pull-based data collection uses CloudWatch API (ListMetrics, GetMetricData) and supports filtering metrics by namespace (e.g.
AWS/EC2
,AWS/RDS
), by metric name (e.g.CPUUtilization
), or by tag (e.g.Environment
,Environment=Prod,Staging
)
We recommend you should pick one way, either push
or pull
, to collect metrics. Using both methods at the same time could make it difficult to troubleshoot issues.
Note
AWS::CloudWatch::MetricStream supports one of the following:
Stream metrics from all metric namespaces in the account.
Stream metrics from all metric namespaces in the account, except for the namespaces that you list in
ExcludeFilters
.Stream metrics from only the metric namespaces that you list in
IncludeFilters
. You cannot specify bothIncludeFilters
andExcludeFilters
in the same metric stream.
Collect EC2 metrics only
Let’s suppose that you are only interested in collect EC2 metrics.
Create a
filter-metrics-example-01.yaml
with the following content.
IncludeFilters:
- Namespace: AWS/EC2
Upload
filter-metrics-example-01.yaml
to an S3 bucket and find the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-01.yaml)Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click
Update
, clickUse existing template
, put the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-01.yaml) into theMetricStreamFilterUri
field under the CloudWatch Metrics section. Complete the rest of CloudFormation steps.
Collect CPUUtilization
only from EC2 instances
Let’s suppose that you want to collect CPUUtilization
only from your EC2 instances (AWS/EC2
).
Create a
filter-metrics-example-02.yaml
with the following content.
IncludeFilters:
- Namespace: AWS/EC2
MetricNames:
- CPUUtilization
Upload
filter-metrics-example-02.yaml
to an S3 bucket and find the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-02.yaml)Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click
Update
, clickUse existing template
, put the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-02.yaml) into theMetricStreamFilterUri
field under the CloudWatch Metrics section. Complete the rest of CloudFormation steps.
You can find CloudWatch metrics that are available for your EC2 instances [here].(https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html)
Do NOT collect TruncateFinishedPoint
from RDS instances
Let’s suppose that you do NOT want to collect TruncateFinishedPoint
from your RDS instances (‘AWS/RDS`).
Create a
filter-metrics-example-03.yaml
with the following content.
ExcludeFilters:
- Namespace: AWS/RDS
MetricNames:
- TruncateFinishedPoint
Upload
filter-metrics-example-03.yaml
to an S3 bucket and find the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-03.yaml)Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click
Update
, clickUse existing template
, put the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-03.yaml) into theMetricStreamFilterUri
field under the CloudWatch Metrics section. Complete the rest of CloudFormation steps.
You can find CloudWatch metrics that are available for your RDS instances [here].(https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html)
Collect all CloudWatch metrics
Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click
Update
, clickUse existing template
, put the S3 URL (e.g. s3://observeinc/cloudwatchmetrics/filters/full.yaml) into theMetricStreamFilterUri
field under the CloudWatch Metrics section. Complete the rest of CloudFormation steps.
s3://observeinc/cloudwatchmetrics/filters/full.yaml
includes the following filter.
ExcludeFilters: []
Collect metrics that include Environment
as a tag key
Find the existing poller (e.g. arn:aws:iam::1234567890:role/observe-metrics-poller-xxx) under the
Connections
tab for the AWS App or in theAWS
datastream.Edit the poller and put
Environment
in theResource Tags
field.Click
Update
andClose
.
Collect metrics that include Environment=Prod
or Environment=Staging
Find the existing poller (e.g. arn:aws:iam::1234567890:role/observe-metrics-poller-xxx) under the
Connections
tab for the AWS App or in theAWS
datastream.Edit the poller and put
Environment=Prod,Staging
in theResource Tags
field.Click
Update
andClose
.
What is Filedrop?¶
Warning
Filedrop is an advanced feature in Public Preview. Use the in-product Contact Support button to contact Observe for more information. On the left side navigation menu, click Docs & Support, Contact Support, and Send Us a Message to contact an Observe Data Engineer.
By default, Observe uses a Lambda function to read files from customer’s S3 bucket and send them to Observe’s collection endpoint directly. You can enable Filedrop
to make the Lambda function send them to Observe’s S3 bucket, and rely on Observe’s Filedrop
service. This service could help you reduce data transfer(egress) cost.
What configuration variables are available?¶
Documentation for configuration variables can be found here: CloudFormation