AWS Integration

The Observe AWS Integration streamlines the process of collecting data from AWS. Ingest logs and metrics from several common AWS services at once, then ingest data from additional services by sending it to the forwarders the AWS Integration sets up for you.

The AWS Integration works with the datasets in your workspace. Contact us for assistance creating datasets and modeling the relationships between them. We can automate many common data modeling tasks for you, ensuring an accurate picture of your infrastructure. We can also update your workspace with improved and new datasets as we release new functionality for this integration.

If you are already ingesting AWS data, we are happy to discuss if the AWS Integration could enhance your existing data collection strategy.

What data does it ingest?

Standard ingest sources

The AWS Integration automatically ingests the following types of data from a single region:

In addition, if installed in us-east-1 it also ingests Amazon CloudFront data for all regions:

  • CloudFront requests, errors, and (optional) real-time access logs for CloudFront Distributions.

(For more about configuring CloudFront ingest, See Amazon CloudFront.)

To ingest this data, the AWS Integration creates several forwarding paths:

These forwarders work in a single region, as many AWS services are specific to a particular region. For information about multi-region collection, see How do I collect data from multiple regions? in the FAQ.

Additional ingest sources

With these already configured and working, add additional services by configuring them to write to the bucket or send logs to one of the forwarders. Details for common services may be found in our documentation:

  • API Gateway execution and access logs from your REST API

  • AppSync request logs from your GraphQL API

  • CloudWatch logs from EC2, Route53, and other services

  • GuardDuty security findings for threat detection

  • S3 access logs for requests to S3 buckets

Using AWS Integration data

After shaping, the incoming data populates datasets like these:

  • CloudWatch Log Group - Application errors, log event detail

  • IAM

    • IAM Group - Which groups are accessing resources

    • IAM Policy - Policies in use, their descriptions and contents

    • IAM Role - Compare role permissions over time

    • IAM User - Most active users

  • EC2

    • EC2 EBS Volume - Volumes in use, size, usage and performance metrics

    • EC2 Instance - What instances are in which VPCs, instance type, IP address

    • EC2 Network Interface - Associated instance, type, DNS name

    • EC2 Subnet - CIDR block, number of addresses available

    • EC2 VPC - Account and region, if default

  • Account - View resources by account

  • Lambda Function - Active functions, associated Log Group, invocation metrics

  • S3 Bucket - Buckets by account and region

  • CloudFront Distribution - Requests, errors, and other details about Distributions

Setup

Installation

Use our CloudFormation template to automate installing the AWS integration. To install via the AWS Console:

  1. Navigate to the CloudFormation console and view existing stacks.

  2. Click Create stack. If prompted, select With new resources.

  3. Provide the template details:

    1. Under Specify template, select Amazon S3 URL.

    2. In the Amazon S3 URL field, enter https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/collection.yaml.

    3. Click Next to continue. (You may be prompted to view the function in Designer. Click Next again to skip.)

  4. Specify the stack details:

    1. In Stack name, provide a name for this stack. It must be unique within a region, and is used to name created resources.

    2. Under Required Parameters, provide your Customer ID in ObserveCustomer and ingest token in ObserveToken. For details on creating an ingest token for a datastream, see Data streams

    3. Click Next

  5. Under Configure stack options, there are no required options to configure. Click Next to continue.

  6. Review your stack options:

    1. Under Capabilities, check the box to acknowledge that this stack may create IAM resources.

    2. Click Create stack

Video instructions

Alternatively, you can deploy the CloudFormation template using the awscli utility:

Caution

If you have multiple AWS profiles, make sure you configure the appropriate AWS_REGION and AWS_PROFILE environment variables in addition to OBSERVE_CUSTOMER and OBSERVE_TOKEN.

$ curl -Lo collection.yaml https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/collection.yaml
$ aws cloudformation deploy --template-file ./collection.yaml \
	  --stack-name ObserveLambda \
	  --capabilities CAPABILITY_NAMED_IAM \
	  --parameter-overrides ObserveCustomer="${OBSERVE_CUSTOMER?}" ObserveToken="${OBSERVE_TOKEN?}"

If you would like to automatically subscribe existing and future CloudWatch Log Groups, see How do I send CloudWatch Log Groups to Observe? in the FAQ.

You may also use our Terraform module to install the AWS integration and created the needed Kinesis Firehose delivery stream. The following is an example instantiation of this module:

module "observe_collection" {
  source           = "github.com/observeinc/terraform-aws-collection"
  observe_customer = "${OBSERVE_CUSTOMER}"
  observe_token    = "${OBSERVE_TOKEN}"
}

We recommend that you pin the module version to the latest tagged version.

FAQ

Where are the integration’s forwarders located?

All resources are created in the region where you installed the AWS Integration, such as us-east-1. They are named based on the CloudFormation stack name or Terraform module name you provided.

For example, a CloudFormation stack called Observe-AWS-Integration would result in names like:

  • Lambda function Observe-AWS-Integration

  • S3 bucket observe-aws-integration-bucket-1a2b3c4d5e

  • Kinesis Firehose delivery stream Observe-AWS-Integration-Delivery-Stream-1a2b3c4d5e

Note

To ensure the generated resources comply with AWS naming rules, your stack or module name should contain only:

  • Letters (A-Z and a-z)

  • Numbers (0-9)

  • Hyphens (-)

  • Maximum of 30 characters

How do I collect data from multiple regions?

The Observe AWS integration operates on a per-region basis because some sources, such as CloudWatch metrics, are specific to a single region. For multiple regions, we recommend installing the integration in each region. You may do this with a CloudFormation StackSet, or by tying the Terraform module into your existing manifests.

For CloudFront, a global (not region-specific) service, install the AWS Integration in us-east-1.

What permissions are required?

The integration periodically queries the AWS API for information about certain services. To do this, its Lambda function has permissions to execute the following actions:

  • dynamodb:List*

  • dynamodb:Describe*

  • ec2:Describe*

  • ecs:List*

  • ecs:Describe*

  • elasticache:Describe*

  • elasticloadbalancing:Describe*

  • firehose:List*

  • firehose:Describe*

  • iam:Get*

  • iam:List*

  • lambda:List*

  • logs:Describe*

  • rds:Describe*

  • redshift:Describe*

  • route53:List*

  • s3:List*

You may change these permissions if needed. If the Lambda function does not have permission for a particular service, it will not collect that information.

The integration S3 bucket is subscribed to the Observe Lambda, with permissions that allow other AWS services to write to it. For example, ELB access logs or VPC flow logs.

How do I send CloudWatch Log Groups to Observe?

Subscribe all existing and future CloudWatch Log Groups to a delivery stream using our optional CloudFormation stack. You may do this at any time after installing the AWS integration. This stack subscribes Log Groups to the Kinesis Firehose delivery stream created for you by the AWS Integration.

To subscribe CloudWatch Log Groups to your delivery stream:

Prerequisites:

  • Complete your installation of the AWS Integration, if needed.

  • Locate the name of your AWS Integration CloudFormation Stack. This is the name provided for Stack name when you installed the AWS integration.

    • In the AWS console, go to CloudFormation to view your existing stacks.

    • Look for an Observe AWS Integration stack, such as Observe-AWS-Integration.

Deploy the subscribelogs CloudFormation Stack:

  1. Navigate to CloudFormation in the AWS console.

  2. Click Create stack. If prompted, select With new resources.

  3. Provide the template details:

    1. Under Specify template, select Amazon S3 URL.

    2. In the Amazon S3 URL field, enter https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/subscribelogs.yaml.

    3. Click Next to continue. (You may be prompted to view the function in Designer. Click Next again to skip.)

  4. Specify the stack details:

    1. In Stack name, provide a name for this stack. It must be unique within a region, and is used to name created resources.

    2. Under Required Parameters, provide the name of your AWS Integration stack in CollectionStackName.

    3. Configure optional parameters:

      • To restrict which Log Groups are automatically subscribed, specify a prefix in AllowedLogGroupPrefix. For example, /aws/rds/ subscribes RDS Log Groups.

  5. Click Next, and then Next again. There are no required options under Configure stack options.

  6. Review your stack options:

    • Under Capabilities, check the box to acknowledge that this stack may create IAM resources.

  7. Click Create stack.

The stack automatically subscribes any matching CloudWatch Log Groups, even if created later. If the stack is deleted, it cleans up any resources it created and unsubscribes any Log Groups it subscribed.

Note

A CloudWatch Log Group may be subscribed to a maximum of two delivery streams. If your desired Log Group is already subscribed to other delivery streams, you may remove one and then add it to your AWS Integration delivery stream by following the directions at AWS CloudWatch Logs.