Amazon Elastic Container Log Collection¶

ECS Log Collection¶

There are two primary methods for gathering logs from ECS:

Utilizing CloudWatch to collect logs.
Using AWS FireLens to gather logs from AWS ECS Fargate.

ECS clusters can be either AWS-managed or self-managed, offering distinct options for each setup.

AWS-Managed ECS Clusters¶

Direct logs to CloudWatch and then use an Observe Lambda forwarder to forward them to Observe.
Set up FireLens to channel logs directly to Observe. AWS FireLens will then use the Observe Amazon Kinesis Forwarder to send logs to Observe

Self-Managed ECS Clusters¶

Capture logs using the Observe Agent.
These can be run directly on EC2 instances or within a container.

FireLens is an AWS service that acts as a log router for Amazon ECS. FireLens supports Fluent Bit and Fluentd, either can be used but Fluent Bit is preferred due to its smaller footprint and http compression. FireLens can be added to any task definition and runs as a container in the task when the task is deployed.

Here is an example FireLens configuration to send Amazon ECS logs to an AWS service or Observe.

{
  "essential": true,
  "image": "httpd",
  "name": "app",
  "logConfiguration": {
    "logDriver": "awsfirelens",
    "options": {
      "Name": "firehose",
      "region": "us-west-2",
      "delivery_stream": "ObserveCollection-DeliveryStream-IKQ5CMkA3swD",
      "log-driver-buffer-limit": "2097152"
      "time_key_format": "%Y-%m-%dT%H:%M:%S.%LZ",
      "time_key": "@timestamp"
    }
  },
  "dependsOn": [
    {
      "containerName": "log_router",
      "condition": "START"
    }
  ],
  "memoryReservation": 100
}

Adding PutRecordBatch to your role

Your task will need a Task Execution IAM Role to pull containers, and the AWS Console will create one for you if you do not specify one during task creation time. Assuming you are using the default you will need to access the Role in IAM and add a new permission along the lines of:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": "firehose:PutRecordBatch",
      "Resource": "arn:aws:firehose:<region>*:<account-id>:deliverystream/**"
    }
  ]
}

To configure Fluent Bit for Observe see Fluent Bit

Note: FireLens manages the match configuration. You do not specify the match configuration in your task definition. Ref - AWS Example Amazon ECS task definition: Route logs to FireLens