Home
User Guides
Start typing to search…
  1. Add Data
  2. Observe Agent
  3. Install on Amazon ECS

Install on Amazon ECS (Fargate - Sidecar Pattern)

This topic guides you through deploying the Observe Agent with a configuration file, updating your ECS task definition to use it, and finally running it as a service to collect metrics, traces, and logs from your ECS Tasks.

Prerequisites

Before you proceed, verify that the following requirements are met:

  • You have AWS CLI installed and configured.
  • You have an application running in ECS on Fargate
  • You have access to an ECS Cluster.
  • You have permissions to create and manage objects inside of an S3 bucket
  • Your account has the proper IAM roles for ECS tasks and execution, including permissions for CloudWatch logs and S3.

Configuration architecture

Given the following pre-configuration architecture, with an application running in ECS on an ECS cluster:

After you complete the steps on this page, the architecture looks like this:

Infrastructure setup

Perform the tasks in this section to create an S3 bucket and an IAM policy.

Create an S3 bucket

aws s3api create-bucket --bucket my-config-bucket --region us-west-2

Create an IAM Policy

Create an IAM policy that grants access to the bucket you just created.

aws iam create-policy \
  --policy-name MyConfigAccessPolicy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Action": [
          "s3:GetObject",
          "s3:ListBucket"
        ],
        "Effect": "Allow",
        "Resource": [
          "arn:aws:s3:::my-config-bucket",
          "arn:aws:s3:::my-config-bucket/*"
        ]
      }
    ]
  }'

The above command will output a policy ARN like so

arn:aws:iam::<ACCOUNT_ID>:policy/MyConfigAccessPolicy

Replace <POLICY_ARN> with the ARN from the previous command

aws iam attach-role-policy \
  --role-name my-ecs-task-role-name \
  --policy-arn <POLICY_ARN>

Your command will look like

aws iam attach-role-policy \
  --role-name my-ecs-task-role-name \
  --policy-arn arn:aws:iam::123456789012:policy/MyConfigAccessPolicy
📘

Note

You must ensure that the MyConfigAccessPolicy is attached to the IAM Role of your task IAM role and NOT your task execution IAM role .

Configure the Observe Agent

Your next step is to create and upload your observe-agent.yaml file to the my-config-bucket S3 bucket so that your service can download and use it.

Configuration delivery and rendering options

There are several ways to handle the rendering of container configuration and secrets at runtime. This approach is simplistic and should not be used in production environments. It is merely used to demonstrate the pattern and should be altered based on the needs and requirements of your organization.

Below is a non-exhaustive list of other possible approaches:

Create the observe-agent.yaml configuration file

📘

Note

You will want to replace <ingest_token>, <tenant>, <service-name>, <service-version>, and <environment with your tenant- and application-specific values.

# Observe data token (ex: a1b2c3d4e5f6g7h8i9k0:l1m2n3o4p5q6r7s8t9u0v1w2x3y4z5a6)
token: "<ingest_token>"

# Target Observe collection url (ex: https://123456789012.collect.observeinc.com/)
observe_url: "https://<tenant>.collect.observeinc.com/"

host_monitoring:
  enabled: false
  logs:
    enabled: false

forwarding:
  enabled: true
  metrics:
    output_format: otel

resource_attributes:
  service.name: <service-name>               # e.g. "my-app"
  service.version: <service-version>				 # e.g. "v.1.0"
  deployment.environment.name: <environment> # e.g. "dev"

otel_config_overrides:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

    awsecscontainermetrics:
      collection_interval: 20s
    
    filelog:
      include: [ /mnt/shared/logs/*.json ]
      operators:
        - type: json_parser
          timestamp:
            parse_from: attributes.asctime
            layout: '%Y-%m-%d %H:%M:%S,%f'

  processors:
    batch:
    resource:
      attributes:
        - key: service.name
          value: <service-name>    # Same as above
          action: insert
        - key: deployment.environment.name
          value: <environment>     # Same as above
          action: insert
        - key: service.version
          value: <service-version> # Same as above
          action: insert

  exporters:
    otlphttp/observelogs:
      endpoint: "https://<tenant>.collect.observeinc.com//v2/otel"
      headers:
        authorization: "Bearer <ingest_token>"
        x-observe-target-package: "Host Explorer"
      sending_queue:
        num_consumers: 4
        queue_size: 100
      retry_on_failure:
        enabled: true
      compression: zstd

  service:
    pipelines:
      logs:
        receivers: 
          - filelog
        processors: 
          - batch
          - resource
        exporters: 
          - otlphttp/observelogs
      
      metrics/ecs_fargate:
        receivers: 
          - awsecscontainermetrics
        processors: 
          - batch
          - memory_limiter
          - resourcedetection
          - resourcedetection/cloud
        exporters: 
          - otlphttp/observemetrics
📘

NOTE: The block below is specific to this application and will likely need to be modified for your structured log output. See the OTel filelogreceiver documentation for more information.

    filelog:
      include: [ /mnt/shared/logs/*.json ]
      operators:
        - type: json_parser
          timestamp:
            parse_from: attributes.asctime
            layout: '%Y-%m-%d %H:%M:%S,%f'

Upload your configuration to S3

Next, upload your observe-agent.yaml file to s3

aws s3 cp observe-agent.yaml s3://my-config-bucket/observe-agent/observe-agent.yaml

Alter the task definition

Next, you will need to alter your Application's Task Definition to include the new observe-agent and init container definitions.

Get the current task definition

export ECS_CLUSTER=<my-ecs-cluster>
export ECS_SERVICE_NAME=<my-app>
export ECS_SERVICE_ARN=$(aws ecs list-services --cluster $ECS_CLUSTER | jq -r --arg name "$ECS_SERVICE_NAME" '.serviceArns[] | select(endswith($name))')
export ECS_TASK_ARN=$(aws ecs describe-services  --cluster "${ECS_CLUSTER}" --services "${ECS_SERVICE_NAME}" --query "services[0].taskDefinition" --output text)

aws ecs describe-task-definition \
  --task-definition "${ECS_TASK_ARN}" \
  --query 'taskDefinition' \
  --output json > current-task-def.json

Make a copy of the current-task-def.json file

cp -pv current-task-def.json updated-task-def.json

Edit the updated-task-def.json and add two new shared volumes (shared-logs, and shared-config) to the task

{
    "taskDefinitionArn": "arn:aws:ecs:us-west-2:12345678910:task-definition/my-app:30",
    ...
    "volumes": [
        {
            "name": "shared-logs",
            "host": {}
        },
        {
            "name": "shared-config",
            "host": {}
        }
    ],
    ...
}

Container definitions

Update the my-app container definition to include the shared-logs mount point, a dependsOn clause for the init container, and most importantly set the correct OTEL_EXPORTER_OTLP_ENDPOINT environment variable.

{
    "taskDefinitionArn": "arn:aws:ecs:us-west-2:12345678910:task-definition/my-app:30",
    "volumes": [
        {
            "name": "shared-logs",
            "host": {}
        },
        {
            "name": "shared-config",
            "host": {}
        }
    ],      
    "containerDefinitions": [
        {
          "name": "my-app",
          "image": "<image_url>",
          "cpu": 256,
          "memory": 512,
          "mountPoints": [
               {
                  "sourceVolume": "shared-logs",
                  "containerPath": "/mnt/shared/",
                  "readOnly": false
                }
            ],
          "dependsOn": [
              {
                  "containerName": "init",
                  "condition": "SUCCESS"
                }
            ],
          "environment": [
          ...
            {
              "name": "OTEL_EXPORTER_OTLP_ENDPOINT",
              "value": "http://127.0.0.1:4318"
            },
          ...          
        },
        {
          // Init container definition goes here
        },
        {
          // Observe Agent container definition goes here
        } 
    ],
    ...
}

Update the Task Definition to include the following two container definitions

  • init
  • observe-agent

init container

{
  "name": "init",
  "image": "alpine:latest",
  "cpu": 128,
  "memory":  256,
  "essential": false,
  "command": [
    "sh",
    "-c",
    "set -euo pipefail; apk add --no-cache aws-cli; mkdir -p /mnt/shared-config/; mkdir -p /mnt/shared-logs/logs; aws s3 cp \"s3://my-config-bucket/observe-agent/observe-agent.yaml\" \"/mnt/shared-config/observe-agent.yaml\"; chmod 0644 \"/mnt/shared-config/observe-agent.yaml\"; chmod -R 0777 /mnt/shared-logs /mnt/shared-config; echo \"Init Completed Successfully\"; exit 0"
  ],
  "user": "root",
  "mountPoints": [
    {
      "sourceVolume": "shared-logs",
      "containerPath": "/mnt/shared-logs",
      "readOnly": false
    },
    {
      "sourceVolume": "shared-config",
      "containerPath": "/mnt/shared-config",
      "readOnly": false
    }
  ], 
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
        "awslogs-region": "us-west-2",
        "awslogs-group": "my-app",
        "awslogs-stream-prefix": "init"
    }
  }
}

observe-agent container

{
  "name": "observe-agent",
  "image": "observeinc/observe-agent:2.10.1",
  "cpu": 256,
  "memory":  512,
  "essential": false,
  "mountPoints": [
    {
      "sourceVolume": "shared-config",
      "containerPath": "/etc/observe-agent/",
      "readOnly": false
    },
    {
      "sourceVolume": "shared-logs",
      "containerPath": "/mnt/shared/",
      "readOnly": true
    }
  ], 
  "dependsOn": [
    {
      "containerName": "init",
      "condition": "SUCCESS"
    }
  ],
  "portMappings": [
    {
      "containerPort": 4317,
      "hostPort": 4317,
      "protocol" : "tcp"
    },
    {
      "containerPort": 4318,
      "hostPort": 4318,
      "protocol" : "tcp"
    }
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
        "awslogs-region": "us-west-2",
        "awslogs-group": "my-app",
        "awslogs-stream-prefix": "observe-agent"
    }
  }
}

Complete task definition

{
    "taskDefinitionArn": "arn:aws:ecs:us-west-2:12345678910:task-definition/my-app:30",
    "volumes": [
        {
            "name": "shared-logs",
            "host": {}
        },
        {
            "name": "shared-config",
            "host": {}
        }
    ],      
    "containerDefinitions": [
      {
          "name": "my-app",
          "image": "<image_url>",
          "cpu": 256,
          "memory": 512,
          "mountPoints": [
              {
                  "sourceVolume": "shared-logs",
                  "containerPath": "/mnt/shared/",
                  "readOnly": false
              }
          ],
          "dependsOn": [
              {
                  "containerName": "init",
                  "condition": "SUCCESS"
              }
          ],
          "environment": [
            ...
            {
                "name": "OTEL_EXPORTER_OTLP_ENDPOINT",
                "value": "http://127.0.0.1:4318"
            },
            ...
          ],
         ...          
      },
      {
        "name": "init",
        "image": "alpine:latest",
        "cpu": 128,
        "memory":  256,
        "essential": false,
        "command": [
          "sh",
          "-c",
          "set -euo pipefail; apk add --no-cache aws-cli; mkdir -p /mnt/shared-config/; mkdir -p /mnt/shared-logs/logs; aws s3 cp \"s3://my-config-bucket/observe-agent/observe-agent.yaml\" \"/mnt/shared-config/observe-agent.yaml\"; chmod 0644 \"/mnt/shared-config/observe-agent.yaml\"; chmod -R 0777 /mnt/shared-logs /mnt/shared-config; echo \"Init Completed Successfully\"; exit 0"
        ],
        "user": "root",
        "mountPoints": [
          {
            "sourceVolume": "shared-logs",
            "containerPath": "/mnt/shared-logs",
            "readOnly": false
          },
          {
            "sourceVolume": "shared-config",
            "containerPath": "/mnt/shared-config",
            "readOnly": false
          }
        ], 
        "logConfiguration": {
          "logDriver": "awslogs",
          "options": {
              "awslogs-region": "us-west-2",
              "awslogs-group": "my-app",
              "awslogs-stream-prefix": "init"
          }
        }
      }, 
      {
        "name": "observe-agent",
        "image": "observeinc/observe-agent:2.10.1",
        "cpu": 256,
        "memory":  512,
        "essential": false,
        "mountPoints": [
          {
            "sourceVolume": "shared-config",
            "containerPath": "/etc/observe-agent/",
            "readOnly": false
          },
          {
            "sourceVolume": "shared-logs",
            "containerPath": "/mnt/shared/",
            "readOnly": true
          }
        ], 
        "dependsOn": [
          {
            "containerName": "init",
            "condition": "SUCCESS"
          }
        ],
        "portMappings": [
          {
            "containerPort": 4317,
            "hostPort": 4317,
            "protocol" : "tcp"
          },
          {
            "containerPort": 4318,
            "hostPort": 4318,
            "protocol" : "tcp"
          }
        ],
        "logConfiguration": {
          "logDriver": "awslogs",
          "options": {
              "awslogs-region": "us-west-2",
              "awslogs-group": "my-app",
              "awslogs-stream-prefix": "observe-agent"
          }
        }
      }    
    ],
    ...
}

Update the ECS service

Register a new version of the task definition and update the service with it.

export NEW_ECS_TASK_DEFINITION_ARN=$(aws ecs register-task-definition --cli-input-json file://updated-task-def.json)
                                     
aws ecs update-service \
  --cluster ${ECS_CLUSTER} \
  --service ${ECS_SERVICE_NAME} \
  --task-definition ${NEW_ECS_TASK_DEFINITION_ARN}

Init container command

This script:

  1. Installs aws-cli
  2. Creates the /mnt/shared-config directory
  3. Creates the /mnt/shared-logs/logs directory
  4. Downloads and runs chmod 0644 on the observe-agent.yaml file from your S3 bucket
  5. Runs chmod 0777 on the /mnt/shared-logs/logs directory
set -euo pipefail; 
apk add --no-cache aws-cli; 
mkdir -p /mnt/shared-config/; 
mkdir -p /mnt/shared-logs/logs; 
aws s3 cp s3://my-config-bucket/observe-agent/observe-agent.yaml /mnt/shared-config/observe-agent.yaml || true; 
chmod 0644 /mnt/shared-config/observe-agent.yaml || true; 
chmod -R 0777 /mnt/shared-logs /mnt/shared-config || true; 
echo "Init Completed Successfully"; 
exit 0

Updated 2 days ago