AWS CloudWatch Logs

Use Amazon CloudWatch Logs to monitor, store, and access your log files from Amazon Elastic Compute Cloud (EC2) instances, AWS CloudTrail, Route 53, and other sources. Send those logs to Observe with one of the options described below.

Collection methods

Observe supports three methods of collecting AWS CloudWatch Logs.

../../../_images/cloudwatchlogs.png ../../../_images/cloudwatchlogs-dark.png

To choose the right method, consider the needs of your environment:

  1. For quick setup, use the Lambda forwarder, available as part of the Observe AWS Integration.

  2. For production traffic, evaluate AWS Kinesis Firehose. A pre-configured Firehose delivery stream is included with the AWS Integration.

  3. For higher throughput or sending data to multiple upstreams, use AWS Kinesis Data Streams.

Observe Lambda

Configuring Observe’s lambda function is the simplest way of getting started, incurs the lowest end-to-end latency, and is competitively priced for lower data volumes. Our lambda can handle multiple sources, and submits data to Observe using the http endpoint.

A downside of using our lambda is it will not retry on failure. This avoids prolonging execution time which would impact cost. In the event of network failures or an outage on our end, the data will not be resubmitted.

CloudWatch Logs allows you to set up Subscription filters which forward logs within a Log Group to a destination.

  1. Install the Observe Lambda, either as a standalone forwarder or as part of our AWS integration.

  2. Navigate to CloudWatch and view your Log Groups.

  3. Select the Log Group you would like to send to Observe.

  4. Click Actions and select Create Lambda subscription filter.

  5. Under Choose destination, select your Observe Lambda function.

  6. Under Configure log format and filters, select Log format Other. This forwards all logs.

  7. In Subscription filter name provide a name for this filter. The name is used to identify the subscription within the context of the log group.

  8. Click Start Streaming

Video instructions

The observe_lambda Terraform module provides a helper submodule for subscribing log groups to the Observe Lambda forwarder.

Instantiate the Lambda forwarder by providing values for observe_customer and observe_token and the list of log groups you would like to forward:

module "observe_lambda" {
  source           = "github.com/observeinc/terraform-aws-lambda"
  observe_customer = var.observe_customer
  observe_token    = var.observe_token
}

module "observe_lambda_cloudwatch_logs_subscription" {
  source = "github.com/observeinc/terraform-aws-lambda//cloudwatch_logs_subscription"
  lambda = module.observe_lambda.lambda_function
  log_group_names = [
    "your-log-group-name"
  ]
}

Kinesis Firehose

AWS Kinesis Firehose backs up data in S3 in the case of failed delivery. It is also more cost effective than Lambda functions for higher event volumes. Since data is batched, there can be a significant buffering delay sending low volumes of data. For this reason, we recommend starting with the Lambda forwarder. Then evaluate your traffic profile and requirements to determine if AWS Kinesis Firehose is appropriate for your environment.

CloudWatch Logs uses Subscription filters to send logs to a Kinesis Firehose delivery stream, which then forwards to Observe.

  1. If needed, create a Kinesis Firehose delivery stream, either as a standalone forwarder or as part of the Observe AWS integration.

    Note

    If you created a delivery stream using one of our CloudFormation templates, you do not need to create an additional stream. The name of the role includes the stack that created it, for example ObserveAWSIntegration-CloudWatchLogsRole-1A2B3C4D5E.

  2. Configure the log group to send to this delivery stream:

    1. Navigate to CloudWatch Log Groups.

    2. Select the Log Group you would like to export to Observe.

    3. From the Actions dropdown, select Subscription filters and Create Kinesis Firehose subscription filter.

    4. Under Choose destination, search for your desired Kinesis Firehose delivery stream. Leave the Destination account at Current account.

    5. Select a Kinesis Firehose delivery stream to send to.

  3. Set the necessary permissions:

    1. Under Grant permission, select an IAM role that permits CloudWatch Logs to write to your delivery stream.

    2. Under Configure log format and filters, provide a name for this filter. The name is used to identify the subscription within the context of the log group.

  4. Click Start Streaming

Video instructions

The observe_kinesis_firehose Terraform module provides a helper submodule for subscribing log groups to the Observe Lambda forwarder.

Instantiate a delivery stream by providing values for observe_customer and observe_token and the list of log groups you would like to forward:

module "observe_kinesis_firehose" {
  source           = "github.com/observeinc/terraform-aws-kinesis-firehose"
  observe_customer = var.observe_customer
  observe_token    = var.observe_token
}

module "observe_kinesis_firehose_cloudwatch_logs_subscription" {
  source           = "github.com/observeinc/terraform-aws-kinesis_firehose//cloudwatch_logs_subscription"
  kinesis_firehose = module.observe_kinesis_firehose
  log_group_names  = [
    "your-log-group-name"
  ]
}