Amazon CloudWatch Logs¶

Use Amazon CloudWatch Logs to monitor, store, and access your log files from Amazon Elastic Compute Cloud (EC2) instances, AWS CloudTrail, Amazon Route 53, and other sources. Send the logs to Observe using one of the options described below.

Collection methods¶

Observe supports three methods of collecting Amazon CloudWatch Logs.

Figure 1 - Collection methods for Amazon CloudWatch logs

To choose the right method, consider the needs of your environment:

For quick setup, use the Lambda forwarder, available as part of the Observe AWS Integration.
For production traffic, evaluate Amazon Kinesis Firehose. The AWS app includes a pre-configured Amazon Kinesis Firehose delivery stream.
For higher throughput or sending data to multiple streams, use Amazon Kinesis Data Streams.

Configuring the Observe Lambda¶

Configuring the Observe Lambda function provides the simplest way of getting started, incurs the lowest end-to-end latency, and provides competitive prices for lower data volumes. The Observe Lambda can handle multiple sources and submits data to Observe using the http endpoint.

Note

The Observe lambda does not retry on failure. This avoids prolonging the execution time which impacts cost. In the event of network failures or an outage, the Observe Lambda does not resubmit data.

AWS Console

Amazon CloudWatch Logs allows you to set up Subscription filters which forward logs within a Log Group to a destination.

Install the Observe Lambda, either as a standalone forwarder or as part of the AWS integration.
Navigate to Amazon CloudWatch and view your Log Groups.
Select the Log Group you want to send to Observe.
Click Actions and select Create Lambda subscription filter.
Under Choose destination, select your Observe Lambda function.
Under Configure log format and filters, select Log format Other. This forwards all logs.
In Subscription filter name provide a name for this filter. The name identifies the subscription within the context of the log group.
Click Start Streaming.

Terraform

The observe_lambda Terraform module provides a helper submodule for subscribing log groups to the Observe Lambda forwarder.

Instantiate the Lambda forwarder by providing values for observe_customer and observe_token and the list of log groups you would like to forward:

module "observe_lambda" {
  source           = "github.com/observeinc/terraform-aws-lambda"
  observe_customer = var.observe_customer
  observe_token    = var.observe_token
}

module "observe_lambda_cloudwatch_logs_subscription" {
  source = "github.com/observeinc/terraform-aws-lambda//cloudwatch_logs_subscription"
  lambda = module.observe_lambda.lambda_function
  log_group_names = [
    "your-log-group-name"
  ]
}

Configuring Amazon Kinesis Firehose¶

Amazon Kinesis Firehose backs up data in Amazon S3 in the case of failed delivery. It is also more cost effective than Lambda functions for higher event volumes. Since Amazon Kinesis Firehose batches the data, there can be a significant buffering delay when sending low volumes of data. For this reason, Observe recommends starting with the Lambda forwarder. Then evaluate your traffic profile and requirements to determine if Amazon Kinesis Firehose is appropriate for your environment.

AWS Console

Amazon CloudWatch Logs uses Subscription filters to send logs to an Amazon Kinesis Firehose delivery stream, which then forwards to Observe.

If needed, create an Amazon Kinesis Firehose delivery stream, either as a standalone forwarder or as part of the Observe AWS integration.

Note

If you created a delivery stream using one of the Observe CloudFormation templates, you do not need to create an additional stream. The name of the role includes the stack that created it, for example ObserveAWSIntegration-CloudWatchLogsRole-1A2B3C4D5E.
Configure the log group to send to this delivery stream:

a. Navigate to CloudWatch Log Groups.

b. Select the Log Group you want to export to Observe.

c. From the Actions dropdown, select Subscription filters and Create Kinesis Firehose subscription filter.

d. Under Choose destination, search for your desired Kinesis Firehose delivery stream. Leave the Destination account at Current account.

e. Select an Amazon Kinesis Firehose delivery stream.
Set the necessary permissions:

a. Under Grant permission, select an IAM role that permits Amazon CloudWatch Logs to write to your delivery stream.

b. Under Configure log format and filters, provide a name for this filter. The name identifies the subscription within the context of the log group.
Click Start Streaming.

Terraform

The observe_kinesis_firehose Terraform module provides a helper submodule for subscribing log groups to the Observe Lambda forwarder.

Instantiate a delivery stream by providing values for observe_customer and observe_token and the list of log groups to forward to Observe:

module "observe_kinesis_firehose" {
  source           = "github.com/observeinc/terraform-aws-kinesis-firehose"
  observe_customer = var.observe_customer
  observe_token    = var.observe_token
}

module "observe_kinesis_firehose_cloudwatch_logs_subscription" {
  source           = "github.com/observeinc/terraform-aws-kinesis_firehose//cloudwatch_logs_subscription"
  kinesis_firehose = module.observe_kinesis_firehose
  log_group_names  = [
    "your-log-group-name"
  ]
}