Amazon Web Services (AWS)

The primary role that the AWS App plays is in seamlessly collecting CloudWatch Logs, CloudWatch Metrics, AWS Resources(Config), and other data from your AWS account and make these data useful for monitoring your AWS services.

Note

If you use the AWS App v0.59 or earlier, or use the Observe AWS Collection CloudFormation Templates, go to Amazon Web Services (AWS) Integration [Legacy]

Setup

Install the AWS App

On the left side navigation menu, click Applications, go to the Install new tab, and install the AWS App.

Collect AWS data

After the AWS App is installed, go to the Connections tab.

  1. Click Get started of the Push metrics, logs and resources from AWS region. option.

  2. Put AWS Account ID and AWS Region (e.g. us-west-2) to collect data from.

  3. Toggle Enable Cloudwatch Logs on

  4. Toggle Enable Cloudwatch Metricstream on

  5. Toogle Enable Config on. If you are already collecting AWS Config, toggle this option off and specify the S3 bucket name (e.g. config-bucket-091312542283) where you store AWS Config. You can find the S3 bucket name from the Settings page for AWS Config. Once you find the S3 bucket name, put it in the ConfigDeliveryBucketName field under the AWS Config section in the CloudFormation template you will find in the next step.

  6. Optionally change the name which will be used as the name for CloudFormation stack or Terraform module in the next step.

  7. Click Create

  8. Click quick-create link to install a pre-configured CloudFormation template in the AWS Console. This CloudFormation template is to create a required AWS IAM role, its policy, and necessary AWS services automatically for data collection. Once the CloudFormation stack is created successfully, data will be pushed to Observe.

  9. Complete the creating-stack steps in the AWS Console.

  • For CloudWatch Logs, you can further restrict collection to a set of log group name prefixes or patterns in the CloudFormation template. By default, it will collect logs from all log groups avaialble in your AWS account’s region you specified in Step 2.

  • For CloudWatch Metrics, You can provide further adjustments to collection in the CloudFormation template. You can check what metrics are being collected by default.

curl -o recommended.yaml https://observeinc.s3.us-west-2.amazonaws.com/cloudwatchmetrics/filters/recommended.yaml

  • For AWS Resources (Config), You can fine tune collection to a subset of resource types in the CloudFormation template.

  • Optionally, if you want to send files including CloudTrail events, VPC Flow logs, or other files you store in S3 buckets, you can provide a list of S3 bucket names in the SourceBucketNames field under the Forwader Options section in the CloudFormation template. The Forwader will read new files from the list of S3 buckets. Make sure Amazon EventBridge is enabled for these S3 buckets. You can find the Amazon EventBridge setting under the properties tab for S3 buckets.

If you want to filter metrics by tags or don’t want to use a metric stream for CloudWatch Metrics, you can pull metrics instead.

Note

If you already created a CloudFormation stack to collect logs, metrics, and resources, disable the existing metrics collection first. Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click Update, click Use existing template, empty the MetricStreamFilterUri field under the CloudWatch Metrics section. Complete the rest of CloudFormation steps.

  1. Click Get started of the Pull CloudWatch Metrics from AWS option.

  2. Put AWS Account ID and AWS Region (e.g. us-west-2) to collect data from.

  3. Change Interval Duration to adject a collection interval. Default is 5 minutes.

  4. List AWS namespaces (e.g. AWS/EC2, AWS/Lambda) you’d like to collect metrics from.

  5. Optionally change the name which will be used as the name for CloudFormation stack or Terraform module in the next step.

  6. List of resource tags to filter metrics for.The supported format is or =value1,value2,value3. For exmaple, if you want to collect metrics that include Environment as a tag key, you can put Environment here. If you want to collect metrics that include Environment=Prod or Environment=Staging, you can put Environment=Prod,Staging here.

  7. Click Create

  8. Click quick-create link to install a pre-configured CloudFormation template in the AWS Conosle. This CloudFormation template is to create a required AWS IAM role and its policy automatically for metric collection. Once the role and policy are created successfully, Observe poller will pull CloudWatch metrics based on your selections above.

  9. Complete the creating-stack steps in the AWS Console.

If you want to send files including CloudTrail events, VPC Flow logs, or other files you store in S3 buckets, this option helps you forward new fiels to Observe easily. The Forwader will read new files from the list of S3 buckets. Make sure Amazon EventBridge is enabled for these S3 buckets. You can find the Amazon EventBridge setting under the properties tab for S3 buckets.

  1. Click Get started of the Forward data from S3 option.

  2. Put AWS Account ID and AWS Region (e.g. us-west-2) to collect data from.

  3. Optionally change the name which will be used as the name for CloudFormation stack or Terraform module in the next step.

  4. Put a list of bucket names the forwarder will be allowed to read from.

  5. Click Create

  6. Click quick-create link to install a pre-configured CloudFormation template in the AWS Console. This CloudFormation template is to create a required AWS IAM role, its policy, and necessary AWS services automatically for data collection. Once the CloudFormation stack is created successfully, data will be pushed to Observe.

  7. Complete the creating-stack steps in the AWS Console.

FAQ

How do I troubleshoot AWS data collection?

First, go to the CloudFormation in the AWS console and check whether CloudFormation stacks are successfully created.

Second, you can get traces for the AWS data collection by following the steps below.

  1. Go to the AWS datastream and create a token.

  2. Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click Update, click Use existing template, set DebugEndpoint to https://<token>@<customerId>.collect.observeinc.com/v2/otel (e.g. https://ds1Xd63TEfS5dYodBSyf:OaAdWXaFGaLlx2wdCjzhL5ih06-redacted@123456789.collect.observeinc.com/v2/otel) under the Debugging Options section.

  3. Go to the AWS datastream and click on the token you created in Step 1.

  4. Click Open dataset.

If you are facing issues or pulling metrics, use the in-product Contact Support button to contact Observe for more information. On the left side navigation menu, click Docs & Support, Contact Support, and Send Us a Message to contact an Observe Data Engineer.

How do I filter metrics?

Observe offers two ways to collect metrics:

  1. Push-based data collection uses a CloudWatch metric stream and supports filtering metrics by namespace (e.g. AWS/EC2, AWS/RDS) or by metric name (e.g. CPUUtilization)

  2. Pull-based data collection uses CloudWatch API (ListMetrics, GetMetricData) and supports filtering metrics by namespace (e.g. AWS/EC2, AWS/RDS), by metric name (e.g. CPUUtilization), or by tag (e.g. Environment, Environment=Prod,Staging)

We recommend you should pick one way, either push or pull, to collect metrics. Using both methods at the same time could make it difficult to troubleshoot issues.

Note

AWS::CloudWatch::MetricStream supports one of the following:

  • Stream metrics from all metric namespaces in the account.

  • Stream metrics from all metric namespaces in the account, except for the namespaces that you list in ExcludeFilters.

  • Stream metrics from only the metric namespaces that you list in IncludeFilters. You cannot specify both IncludeFilters and ExcludeFilters in the same metric stream.

Collect EC2 metrics only

Let’s suppose that you are only interested in collect EC2 metrics.

  1. Create a filter-metrics-example-01.yaml with the following content.

IncludeFilters:
  - Namespace: AWS/EC2

  1. Upload filter-metrics-example-01.yaml to an S3 bucket and find the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-01.yaml)

  2. Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click Update, click Use existing template, put the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-01.yaml) into the MetricStreamFilterUri field under the CloudWatch Metrics section. Complete the rest of CloudFormation steps.

Collect CPUUtilization only from EC2 instances

Let’s suppose that you want to collect CPUUtilization only from your EC2 instances (AWS/EC2).

  1. Create a filter-metrics-example-02.yaml with the following content.

IncludeFilters:
  - Namespace: AWS/EC2
    MetricNames:
      - CPUUtilization

  1. Upload filter-metrics-example-02.yaml to an S3 bucket and find the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-02.yaml)

  2. Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click Update, click Use existing template, put the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-02.yaml) into the MetricStreamFilterUri field under the CloudWatch Metrics section. Complete the rest of CloudFormation steps.

You can find CloudWatch metrics that are available for your EC2 instances [here].(https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html)

Do NOT collect TruncateFinishedPoint from RDS instances

Let’s suppose that you do NOT want to collect TruncateFinishedPoint from your RDS instances (‘AWS/RDS`).

  1. Create a filter-metrics-example-03.yaml with the following content.

ExcludeFilters:
  - Namespace: AWS/RDS
    MetricNames:
      - TruncateFinishedPoint

  1. Upload filter-metrics-example-03.yaml to an S3 bucket and find the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-03.yaml)

  2. Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click Update, click Use existing template, put the S3 URL (e.g. s3://observe-filter-metrics/filter-metrics-example-03.yaml) into the MetricStreamFilterUri field under the CloudWatch Metrics section. Complete the rest of CloudFormation steps.

You can find CloudWatch metrics that are available for your RDS instances [here].(https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html)

Collect all CloudWatch metrics

  1. Find the existing CloudFormation stack (e.g. observe-collection-xxx) in the AWS Console, click Update, click Use existing template, put the S3 URL (e.g. s3://observeinc/cloudwatchmetrics/filters/full.yaml) into the MetricStreamFilterUri field under the CloudWatch Metrics section. Complete the rest of CloudFormation steps.

s3://observeinc/cloudwatchmetrics/filters/full.yaml includes the following filter.

ExcludeFilters: []

Collect metrics that include Environment as a tag key

  1. Find the existing poller (e.g. arn:aws:iam::1234567890:role/observe-metrics-poller-xxx) under the Connections tab for the AWS App or in the AWS datastream.

  2. Edit the poller and put Environment in the Resource Tags field.

  3. Click Update and Close.

Collect metrics that include Environment=Prod or Environment=Staging

  1. Find the existing poller (e.g. arn:aws:iam::1234567890:role/observe-metrics-poller-xxx) under the Connections tab for the AWS App or in the AWS datastream.

  2. Edit the poller and put Environment=Prod,Staging in the Resource Tags field.

  3. Click Update and Close.

What is Filedrop?

Warning

Filedrop is an advanced feature in Public Preview. Use the in-product Contact Support button to contact Observe for more information. On the left side navigation menu, click Docs & Support, Contact Support, and Send Us a Message to contact an Observe Data Engineer.

By default, Observe uses a Lambda function to read files from customer’s S3 bucket and send them to Observe’s collection endpoint directly. You can enable Filedrop to make the Lambda function send them to Observe’s S3 bucket, and rely on Observe’s Filedrop service. This service could help you reduce data transfer(egress) cost.

What configuration variables are available?

Documentation for configuration variables can be found here: CloudFormation