Observe Lambda

The Observe Lambda forwarder is a general-purpose Lambda function that forwards data to Observe. It can handle multiple types of events, including those generated by S3, DynamoDB and CloudWatch Logs, as well as read objects in an S3 bucket.

Setup

Installation

To install the Observe Lambda, you must have a valid Observe customer ID and ingest token.

Use our CloudFormation template to automate creating the Lambda function and its permissions. To install via the AWS Console:

  1. Navigate to the CloudFormation console and view existing stacks.

  2. Click Create stack. If prompted, select With new resources.

  3. Provide the template details:

    1. Under Specify template, select Amazon S3 URL.

    2. In the Amazon S3 URL field, enter https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/lambda.yaml.

    3. Click Next to continue. (You may be prompted to view the function in Designer. Click Next again to skip.)

  4. Specify the stack details:

    1. In Stack name, provide a name for this stack. It must be unique within a region, and is used to name created resources.

    2. Under Required Parameters, provide your Customer ID in ObserveCustomer and ingest token in ObserveToken.

    3. Click Next

  5. Under Configure stack options, there are no required options to configure. Click Next to continue.

  6. Review your stack options:

    1. Under Capabilities, check the box to acknowledge that this stack may create IAM resources.

    2. Click Create stack

Video instructions

Alternatively, you may deploy the template with the awscli tool:

Caution

If you have multiple AWS profiles, make sure you configure the appropriate AWS_REGION and AWS_PROFILE environment variables in addition to OBSERVE_CUSTOMER and OBSERVE_TOKEN.

$ curl -Lo lambda.yaml https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/lambda.yaml
$ aws cloudformation deploy --template-file ./lambda.yaml \
	  --stack-name ObserveLambda \
	  --capabilities CAPABILITY_NAMED_IAM \
	  --parameter-overrides ObserveCustomer="${OBSERVE_CUSTOMER?}" ObserveToken="${OBSERVE_TOKEN?}"

You may also use our observe_lambda Terraform module hosted on github to configure the Lambda function.

module "observe_lambda" {
  source = "github.com/observeinc/terraform-aws-lambda"

  name                = "observe-lambda"
  observe_customer    = "${OBSERVE_CUSTOMER}"
  observe_token       = "${OBSERVE_TOKEN}"
}

We recommend you pin the module version to the latest tagged version.

Getting Started

Each time the Lambda function is triggered, it invokes a handler for the type of event ingested. This section describes how to configure triggers for different data sources.

S3 buckets

An S3 trigger sends an event to the Lambda function when an object is created in a bucket. The function then reads the file and uploads it to Observe.

Files are parsed according to their type. For example, .json files should contain a single JSON object or array, and .jsonl files should contain one or more newline delimited JSON objects.

Note

The S3 bucket must be in the same region as the Lambda function, and a bucket can only send data to one function. If you need to send to multiple functions, see Amazon Simple Notification Service.

  1. Navigate to the Lambda console and view your functions.

  2. Select your Observe Lambda function.

  3. Click Add trigger.

  4. Under Trigger configuration, search for S3.

  5. Select the bucket you wish to subscribe to your Lambda function.

  6. Optionally add a prefix or suffix filter.

  7. Under Recursive invocation check the box to acknowledge the function does not both read and write to the same bucket.

  8. Click Add.

Video instructions

We provide a submodule which subscribes S3 buckets to a Lambda configured via terraform-aws-lambda. The following is an example instantiation, assuming aws_s3_bucket.bucket references a bucket resource managed by Terraform:

module "observe_lambda" {
  source           = "github.com/observeinc/terraform-aws-lambda"
  observe_customer = var.observe_customer
  observe_token    = var.observe_token
  observe_domain   = var.observe_domain
  name             = var.name
}

module "observe_lambda_s3_subscription" {
  source = "github.com/observeinc/terraform-aws-lambda//s3_bucket_subscription"
  lambda = module.observe_lambda.lambda_function
  bucket = aws_s3_bucket.bucket
}

For more information, please visit the submodule documentation.

CloudWatch Logs

  1. Navigate to the Lambda console and view your functions.

  2. Select your Observe Lambda function.

  3. Click Add trigger.

  4. Under Trigger configuration, search for CloudWatch Logs.

  5. Select the desired source Log group from the dropdown.

  6. In Filter name, provide a name for this filter.

  7. Click Add.

Video instructions

We provide a submodule which subscribes CloudWatch Log Groups a Lambda configured via terraform-aws-lambda. The following is an example instantiation, assuming aws_cloudwatch_log_group.group references a log group resource managed by Terraform:

module "observe_lambda" {
  source           = "github.com/observeinc/terraform-aws-lambda"
  observe_customer = var.observe_customer
  observe_token    = var.observe_token
  observe_domain   = var.observe_domain
  name             = var.name
}

module "observe_lambda_cloudwatch_logs_subscription" {
  source = "github.com/observeinc/terraform-aws-lambda//cloudwatch_logs_subscription"
  lambda = module.observe_lambda.lambda_function
  log_group_names = [
    aws_cloudwatch_log_group.group.name
  ]
}

For more information, see the submodule documentation.

EventBridge

You may ingest EventBridge events with the Observe Lambda forwarder, although this method is no longer recommended. See the EventBridge ingest documentation for alternate methods.

FAQ

How are failures handled?

The Lambda forwarder does not retry on error. This reduces the risk of unexpected AWS charges from a long-running Lambda function.

What permissions does the Lambda forwarder need?

The Lambda forwarder requires an IAM Role with permission to be invoked.

The CloudFormation template allows the Lambda forwarder to be invoked from S3, SNS, or SQS, as well as read from S3. These permissions are not scoped to individual resources. For more fine grained control over permissions, we recommend the Terraform modules, which more strictly limited permissions.

What external entities does the Lambda forwarder interact with?

The Lambda forwarder only posts data over HTTPS to the Observe API. There is no mechanism for sending data from Observe to the Lambda forwarder.

Troubleshooting

  • CREATE_FAILED while creating the CloudFormation stack

    Check that you have the correct S3 URL, customer ID, and ingest token. The template verifies the connection to Observe as part of the install process, and will fail if it is not able to authenticate.