AWS Integration

The Observe AWS Integration streamlines the process of collecting data from AWS. Ingest logs and metrics from several common AWS services at once, then ingest data from additional services by sending it to the forwarders that the AWS Integration sets up for you.

The AWS Integration works with the datasets in your workspace. Contact Observe for assistance with creating datasets and modeling the relationships between them. Observe can automate many common data modeling tasks for you, ensuring an accurate picture of your infrastructure. Observe can also update your workspace with improved and new datasets when new functionality releases for this integration.

If you currently ingest AWS data, consult with Observe to see if the AWS Integration could enhance your existing data collection strategy.

What data does Observe ingest?

Standard ingest sources

The AWS Integration automatically ingests the following types of data from a single region:

In addition, if installed in us-east-1, Observe also ingests Amazon CloudFront data for all regions:

  • CloudFront requests, errors, and (optional) real-time access logs for CloudFront Distributions.

For more about configuring CloudFront ingest, see Amazon CloudFront.

To ingest this data, the AWS Integration creates several forwarding paths:

These forwarders work in a single region, as many AWS services are specific to a particular region. For information about multi-region collection, see Collecting data from multiple regions? in the FAQ.

Additional ingest sources

With these previous sources configured and working, add additional services by configuring them to write to the bucket or send logs to one of the forwarders. Details for common services may be found in Observe documentation:

  • API Gateway execution and access logs from your REST API

  • AppSync request logs from your GraphQL API

  • CloudWatch logs from EC2, Route53, and other services

  • GuardDuty security findings for threat detection

  • S3 access logs for requests to S3 buckets

Using AWS Integration data

After shaping, the incoming data populates datasets such as these:

  • CloudWatch Log Group - Application errors, log event detail

  • IAM

    • IAM Group - The IAM groups access resources

    • IAM Policy - Policies in use, the descriptions and contents

    • IAM Role - Compare role permissions over time

    • IAM User - Most active users

  • EC2

    • EC2 EBS Volume - Volumes in use, size, usage, and performance metrics

    • EC2 Instance - What instances are in which VPCs, instance type, IP address

    • EC2 Network Interface - Associated instance, type, DNS name

    • EC2 Subnet - CIDR block, number of addresses available

    • EC2 VPC - Account and region, by default

  • Account - View resources by account.

  • Lambda Function - Active functions, associated Log Group, invocation metrics

  • S3 Bucket - Buckets by account and region

  • CloudFront Distribution - Requests, errors, and other details about Distributions

Setup

Installation

You can use the AWS app located on the Apps page to install AWS integrations. Select either creating a connection using CloudFormation or Terraform. When you click Create connection, you then create a token to use with the data stream.

Before you create the connection, click Configuration and select the type of AWS integration from the extensive list.

List of available AWS integrations for Observe

Figure 1 - List of available AWS integrations

After you select the integration, follow the instructions to install the app. You must have an AWS account to install the app and the integrations.

You can use our Terraform module to install the AWS integration and create the needed Kinesis Firehose delivery stream. The following is an example instantiation of this module:

module "observe_collection" {
  source           = "github.com/observeinc/terraform-aws-collection"
  observe_customer = "${OBSERVE_CUSTOMER}"
  observe_token    = "${OBSERVE_TOKEN}"
}

Observe recommends that you pin the module version to the latest tagged version.

Use the Observe CloudFormation template to automate installing the AWS integration. To install using the AWS Console, follow these steps:

  1. Navigate to the CloudFormation console and view existing stacks.

  2. Click Create stack. If prompted, select With new resources.

  3. Under Specify template, select Amazon S3 URL.

    1. In the Amazon S3 URL field, enter the URL for the Observe Collection CloudFormation template: https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/collection-latest.yaml.

  4. Click Next to continue. You may be prompted to view the function in Designer.

  5. Click Next again to skip.

  6. In Stack name, provide a name for this stack. The name must be unique within a region and is used to name created resources.

  7. Under Required Parameters, provide your Customer ID in ObserveCustomer and ingest token in ObserveToken. For details on creating an ingest token for a datastream, see Data streams

  8. Click Next.

  9. Under Configure stack options, there are no required options to configure. Click Next to continue.

  10. Under Capabilities, check the box to acknowledge that this stack may create IAM resources.

  11. Click Create stack

Video instructions

Alternatively, you can deploy the CloudFormation template using the awscli utility:

Caution

If you have multiple AWS profiles, make sure you configure the appropriate AWS_REGION and AWS_PROFILE environment variables in addition to OBSERVE_CUSTOMER and OBSERVE_TOKEN.

$ curl -Lo collection.yaml https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/collection-latest.yaml
$ aws cloudformation deploy --template-file ./collection.yaml \
	  --stack-name ObserveLambda \
	  --capabilities CAPABILITY_NAMED_IAM \
	  --parameter-overrides ObserveCustomer="${OBSERVE_CUSTOMER?}" ObserveToken="${OBSERVE_TOKEN?}"

If you want to automatically subscribe to existing and future CloudWatch Log Groups, see How do I send CloudWatch Log Groups to Observe? in the FAQ.

FAQ

Where are the integration’s forwarders located?

The AWS Integration creates all resources in the region where you install the AWS Integration, such as us-east-1. They are named based on the CloudFormation stack name or Terraform module name you provided.

For example, a CloudFormation stack called Observe-AWS-Integration results in names such as these:

  • Lambda function Observe-AWS-Integration

  • S3 bucket observe-aws-integration-bucket-1a2b3c4d5e

  • Kinesis Firehose delivery stream Observe-AWS-Integration-Delivery-Stream-1a2b3c4d5e

Note

To ensure the generated resources comply with AWS naming rules, your stack or module name should contain only the following types of characters:

  • Letters (A-Z and a-z)

  • Numbers (0-9)

  • Hyphens (-)

  • Maximum of 30 characters

Collecting data from multiple regions

The Observe AWS integration operates on a per-region basis because some sources, such as CloudWatch metrics, are specific to a single region. For multiple regions, Observe recommends installing the integration in each region. You may do this with a CloudFormation StackSet, or by associating the Terraform module with your existing manifests.

For CloudFront, a global, not region-specific, service, installs the AWS Integration in us-east-1.

What permissions are required?

The integration periodically queries the AWS API for information about certain services. To do this, the corresponding Lambda function contains permissions to execute the following actions:

  • dynamodb:List*

  • dynamodb:Describe*

  • ec2:Describe*

  • ecs:List*

  • ecs:Describe*

  • elasticache:Describe*

  • elasticloadbalancing:Describe*

  • firehose:List*

  • firehose:Describe*

  • iam:Get*

  • iam:List*

  • lambda:List*

  • logs:Describe*

  • rds:Describe*

  • redshift:Describe*

  • route53:List*

  • s3:List*

You may change these permissions if needed. If the Lambda function does not have permission for a particular service, it does not collect that information.

The integration S3 bucket subscribes to the Observe Lambda, with permissions that allow other AWS services to write to it. For example, ELB access logs or VPC flow logs can write to the S3 bucket.

How do I send CloudWatch Log Groups to Observe?

Subscribe all existing and future CloudWatch Log Groups to a delivery stream using the optional CloudFormation stack. You may do this at any time after installing the AWS integration. This stack subscribes Log Groups to the Kinesis Firehose delivery stream created for you by the AWS Integration.

Note

To subscribe Log Groups using Terraform, specify the groups to subscribe in your observe_collection module configuration. For more information, see the Observe AWS Collection README in GitHub.

To subscribe CloudWatch Log Groups to your delivery stream:

Prerequisites

Before you begin, confirm you have the necessary prerequisites:

  • Complete your installation of the AWS Integration, if needed.

  • Locate the name of your AWS Integration CloudFormation stack. This is the name provided for Stack name when you installed the AWS Integration.

    1. In the AWS console, go to CloudFormation to view your existing stacks.

    2. Look for an Observe AWS Integration stack, such as Observe-AWS-Integration. Make a note of the name.

Deploy the subscribelogs CloudFormation Stack

  1. Navigate to CloudFormation in the AWS console.

  2. Click Create stack. If prompted, select With new resources.

  3. Provide the template details:

  4. Click Next to continue. You may be prompted to view the function in Designer.

  5. Click Next again to skip.

  6. Specify the stack details:

    • In Stack name, provide a name for this stack. It must be unique within a region, and is used to name created resources.

    • Under Required Parameters, provide the name of your AWS Integration stack in CollectionStackName.

  7. Configure optional parameters:

    • To restrict the automatically subscribed Log Groups, specify a regex pattern in LogGroupMatches. For example, /aws/rds/.* subscribes RDS Log Groups. LogGroupMatches accepts a comma-separated list of regex patterns, so multiple patterns can be specified.

  8. Click Next, and then Next again. There are no required options under Configure stack options.

  9. Review your stack options:

    • Under Capabilities, check the box to acknowledge that this stack may create IAM resources.

  10. Click Create stack.

The stack automatically subscribes to any matching CloudWatch Log Groups, even if you create them later. If you delete the stack, Amazon CloudWatch removes up any created resources and unsubscribes from any Log Groups.

Note

A CloudWatch Log Group may be subscribed to a maximum of two delivery streams. If your desired Log Group is already subscribed to other delivery streams, you may remove one and then add it to your AWS Integration delivery stream by following the directions at Amazon CloudWatch Logs.

How do I get Canary artifacts generated by Canary Runs?

Each Canary run generates an artifact, a JSON file written to an S3 bucket. Create a bucket Event notification to tell the Observe Lambda function to send this file to Observe.

Specify the destination bucket when creating a Canary. If you do not provide a destination bucket, AWS creates one for you.

Prerequisites

The following information is required to configure an Event notification for Canary artifacts:

  • The name of your Observe Lambda forwarder function:

  1. Complete your installation of the AWS Integration, if needed.

  2. Locate the name of your AWS Integration CloudFormation stack. This is the name provided for Stack name when you installed the AWS Integration.

  3. In the AWS console, go to CloudFormation to view your existing stacks.

  4. Look for an Observe AWS Integration stack, such as Observe-AWS-Integration, and click on it.

  5. Click on the Resources tab and locate the associated Lambda function.

  6. Note the Physical ID.

  • The bucket where you store the Canary artifacts

You configure the destination bucket when creating a Canary. If you did not provide a destination bucket, AWS created one for you. Follow these steps to identify the name:

  1. Go to CloudWatch > Synthetic Canaries.

  2. Click on the name of your canary in the Name column

    Synthetics Canary Panel

    Synthetics Canary Panel

  3. Scroll to the bottom of the Canaries page and expand the section Canary artifacts and S3 location - note the top level S3 bucket name

    Canary artifacts S3 location

    Canary artifacts S3 location

Note

Click on the S3 bucket name to open the S3 bucket configuration which you need for creating the Event notification next.

Create the S3 Event Notification

This notification allows the Observe Lambda forwarder to identify when new Canary artifact files become available to be ingested. To create a notification, use the following steps:

  1. Navigate to Buckets in the AWS Console.

  2. Search for the top level S3 bucket (identified in the previous step) used for Canary data storage.

  3. Click on the Properties tab.

  4. Scroll down to Event Notifications.

  5. Click on Create event Notification.

  6. Configure the following information:

    • Event Name: the desired name of this event, such as “Canary artifact creation”

    • Suffix: .json

    • Event Types -> Object Creation: Check All object create events

    • Destination -> Lambda function: Select your Observe Lambda function

  7. Click Save Changes