AWS CloudWatch Logs

Use Amazon CloudWatch Logs to monitor, store, and access your log files from Amazon Elastic Compute Cloud (EC2) instances, AWS CloudTrail, Route 53, and other sources. Send the logs to Observe using one of the options described below.

Collection methods

Observe supports three methods of collecting AWS CloudWatch Logs.

CloudWatch collection methods

Figure 1 - Collection methods for AWS CloudWatch logs

To choose the right method, consider the needs of your environment:

  • For quick setup, use the Lambda forwarder, available as part of the Observe AWS Integration.

  • For production traffic, evaluate AWS Kinesis Firehose. AWS Integration includes a pre-configured Firehose delivery stream.

  • For higher throughput or sending data to multiple upstreams, use AWS Kinesis Data Streams.

Observe Lambda

Configuring the Observe’s lambda function provides the simplest way of getting started, incurs the lowest end-to-end latency, and is competitively priced for lower data volumes. The Observe lambda can handle multiple sources, and submits data to Observe using the http endpoint.

Note

The Observe lambda does not retry on failure. This avoids prolonging the execution time which impacts cost. In the event of network failures or an outage, the Observe lambda does not resubmit data.

CloudWatch Logs allows you to set up Subscription filters which forward logs within a Log Group to a destination.

  1. Install the Observe Lambda, either as a standalone forwarder or as part of the AWS integration.

  2. Navigate to CloudWatch and view your Log Groups.

  3. Select the Log Group you want to send to Observe.

  4. Click Actions and select Create Lambda subscription filter.

  5. Under Choose destination, select your Observe Lambda function.

  6. Under Configure log format and filters, select Log format Other. This forwards all logs.

  7. In Subscription filter name provide a name for this filter. The name identifies the subscription within the context of the log group.

  8. Click Start Streaming.

Video instructions

The observe_lambda Terraform module provides a helper submodule for subscribing log groups to the Observe Lambda forwarder.

Instantiate the Lambda forwarder by providing values for observe_customer and observe_token and the list of log groups you would like to forward:

module "observe_lambda" {
  source           = "github.com/observeinc/terraform-aws-lambda"
  observe_customer = var.observe_customer
  observe_token    = var.observe_token
}

module "observe_lambda_cloudwatch_logs_subscription" {
  source = "github.com/observeinc/terraform-aws-lambda//cloudwatch_logs_subscription"
  lambda = module.observe_lambda.lambda_function
  log_group_names = [
    "your-log-group-name"
  ]
}

AWS Kinesis Firehose

AWS Kinesis Firehose backs up data in S3 in the case of failed delivery. It is also more cost effective than Lambda functions for higher event volumes. Since Kinesis Firehose batches the data, there can be a significant buffering delay sending low volumes of data. For this reason, Observe recommends starting with the Lambda forwarder. Then evaluate your traffic profile and requirements to determine if AWS Kinesis Firehose is appropriate for your environment.

CloudWatch Logs uses Subscription filters to send logs to a Kinesis Firehose delivery stream, which then forwards to Observe.

  1. If needed, create a Kinesis Firehose delivery stream, either as a standalone forwarder or as part of the Observe AWS integration.

    Note

    If you created a delivery stream using one of our CloudFormation templates, you do not need to create an additional stream. The name of the role includes the stack that created it, for example ObserveAWSIntegration-CloudWatchLogsRole-1A2B3C4D5E.

  2. Configure the log group to send to this delivery stream:

    1. Navigate to CloudWatch Log Groups.

    2. Select the Log Group you want to export to Observe.

    3. From the Actions dropdown, select Subscription filters and Create Kinesis Firehose subscription filter.

    4. Under Choose destination, search for your desired Kinesis Firehose delivery stream. Leave the Destination account at Current account.

    5. Select a Kinesis Firehose delivery stream.

  3. Set the necessary permissions:

    1. Under Grant permission, select an IAM role that permits CloudWatch Logs to write to your delivery stream.

    2. Under Configure log format and filters, provide a name for this filter. The name identifies the subscription within the context of the log group.

  4. Click Start Streaming.

Video instructions

The observe_kinesis_firehose Terraform module provides a helper submodule for subscribing log groups to the Observe Lambda forwarder.

Instantiate a delivery stream by providing values for observe_customer and observe_token and the list of log groups you would like to forward:

module "observe_kinesis_firehose" {
  source           = "github.com/observeinc/terraform-aws-kinesis-firehose"
  observe_customer = var.observe_customer
  observe_token    = var.observe_token
}

module "observe_kinesis_firehose_cloudwatch_logs_subscription" {
  source           = "github.com/observeinc/terraform-aws-kinesis_firehose//cloudwatch_logs_subscription"
  kinesis_firehose = module.observe_kinesis_firehose
  log_group_names  = [
    "your-log-group-name"
  ]
}