Observe Lambda¶

The Observe Lambda forwarder provides a general-purpose Lambda function that forwards data to Observe. It can handle multiple types of events, including those generated by Amazon S3, Amazon DynamoDB and Amazon CloudWatch Logs, as well as read objects in an Amazon S3 bucket.

Setting up the Observe Lambda forwarder¶

Installation¶

To install the Observe Lambda, you must have a valid Observe customer ID and data stream token.

AWS Console

Use the Observe CloudFormation template to automate creating the Lambda function and permissions. To install using the AWS Console, and the following steps:

Navigate to the CloudFormation console and view existing stacks.
Click Create stack. If prompted, select With new resources.
Provide the template details:

a. Under Specify template, select Amazon S3 URL.

b. In the Amazon S3 URL field, enter the URL for the Observe Lambda CloudFormation template: https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/lambda-latest.yaml.
- Observe recommends you pin the template version to a tagged version of the Observe Lambda template.

To do this, replace latest in the template URL with the desired version tag:

    https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/lambda-v0.3.0.yaml

For information about available versions, see the Observe Lambda CF template change log in GitHub.

4. Click Next to continue. You may be prompted to view the function in Designer. Click Next again to skip.

5. In Stack name, provide a name for this stack. It must be unique within a region, and used to name created resources.

6. Under Required Parameters, provide your Observe Customer ID in ObserveCustomer and ingest token in ObserveToken.

Note

Some Observe instances may optionally use a name instead of Customer ID; if this is the case for your instance, contact your Observe Data Engineer to discuss implementation. A stem name will work as is, but a DNS redirect name may require client configuration.

7. Click Next.

8. Under Configure stack options, there are no required options to configure. Click Next to continue.

9. Under Capabilities, check the box to acknowledge that this stack may create IAM resources.

10. Click Create stack.

Alternatively, you may deploy the template with the awscli tool:

Caution

If you have multiple AWS profiles, make sure you configure the appropriate AWS_REGION and AWS_PROFILE environment variables in addition to OBSERVE_CUSTOMER and OBSERVE_TOKEN.

$ curl -Lo lambda.yaml https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/lambda-latest.yaml
$ aws cloudformation deploy --template-file ./lambda.yaml \
	  --stack-name ObserveLambda \
	  --capabilities CAPABILITY_NAMED_IAM \
	  --parameter-overrides ObserveCustomer="${OBSERVE_CUSTOMER?}" ObserveToken="${OBSERVE_TOKEN?}"

Terraform

You may also use our observeinc/lambda/aws Terraform module hosted on github to configure the Lambda function.

module "observe_lambda" {
  source = "observeinc/lambda/aws"

  name                = "observe-lambda"
  observe_customer    = "${OBSERVE_CUSTOMER}"
  observe_token       = "${OBSERVE_TOKEN}"
}

Observe recommends that you pin the module version to the latest tagged version.

Getting Started¶

Each time Observe triggers the Lambda function, it invokes a handler for the type of event ingested. This section describes how to configure triggers for different data sources.

Amazon S3 buckets¶

An Amazon S3 trigger sends an event to the Lambda function when an object is created in a bucket. The function then reads the file and uploads it to Observe.

Files are parsed according to their type. For example, .json files should contain a single JSON object or array, and .jsonl files should contain one or more newline delimited JSON objects.

Note

The Amazon S3 bucket must be in the same region as the Lambda function, and a bucket can only send data to one function. If you need to send to multiple functions, see Amazon Simple Notification Service.

AWS Console

Navigate to the Lambda console and view your functions.
Select your Observe Lambda function.
Click Add trigger.
Under Trigger configuration, search for S3.
Select the bucket you wish to subscribe to your Lambda function.
Optionally add a prefix or suffix filter.
Under Recursive invocation check the box to acknowledge the function does not both read and write to the same bucket.
Click Add.

Terraform

Observe provides a submodule that subscribes S3 buckets to a Lambda function configured using terraform-aws-lambda. The following provides an example instantiation, assuming aws_s3_bucket.bucket references a bucket resource managed by Terraform:

module "observe_lambda" {
  source           = "observeinc/lambda/aws"
  observe_customer = var.observe_customer
  observe_token    = var.observe_token
  observe_domain   = var.observe_domain
  name             = var.name
}

module "observe_lambda_s3_subscription" {
  source = "observeinc/lambda/aws//modules/s3_bucket_subscription"
  lambda = module.observe_lambda.lambda_function
  bucket = aws_s3_bucket.bucket
}

For more information, visit the submodule

Amazon CloudWatch Logs¶

AWS Console

Navigate to the Lambda console and view your functions.
Select your Observe Lambda function.
Click Add trigger.
Under Trigger configuration, search for CloudWatch Logs.
Select the desired source Log group from the dropdown.
In Filter name, provide a name for this filter.
Click Add.

Terraform

Observe provides a submodule that subscribes Amazon CloudWatch Log Groups to a Lambda function configured using terraform-aws-lambda. The following is an example instantiation, assuming aws_cloudwatch_log_group.group references a log group resource managed by Terraform:

module "observe_lambda" {
  source           = "observeinc/lambda/aws"
  observe_customer = var.observe_customer
  observe_token    = var.observe_token
  observe_domain   = var.observe_domain
  name             = var.name
}

module "observe_lambda_cloudwatch_logs_subscription" {
  source = "observeinc/lambda/aws//modules/cloudwatch_logs_subscription"
  lambda = module.observe_lambda.lambda_function
  log_group_names = [
    aws_cloudwatch_log_group.group.name
  ]
}

For more information, see the submodule documentation

EventBridge¶

You may ingest EventBridge events with the Observe Lambda forwarder, although this method is no longer recommended. See the EventBridge ingest documentation for alternate methods.

How does the Observe Lambda forwarder handle failures?¶

The Lambda forwarder does not retry on error. This reduces the risk of unexpected AWS charges from a long-running Lambda function.

What permissions does the Observe Lambda forwarder need?¶

The Observe Lambda forwarder requires an IAM Role with permission to invoke the forwarder.

The AWS CloudFormation template allows the Lambda forwarder to be invoked from Amazon S3, Amazon SNS, or Amazon SQS and read from Amazon S3. These permissions are not scoped to individual resources. For more fine-grained control over permissions, Observe recommends the Terraform modules, and configures more strictly limited permissions.

What external entities does the Observe Lambda forwarder interact with?¶

The Observe Lambda forwarder only posts data over HTTPS to the Observe API. There is no mechanism for sending data from Observe to the Lambda forwarder.

Troubleshooting¶

CREATE_FAILED while creating the AWS CloudFormation stack

Check that you have the correct Amazon S3 URL, customer ID, and data stream token. The template verifies the connection to Observe as part of the install process and fails if it cannot authenticate.