Amazon Kinesis Firehose

Amazon Kinesis Data Firehose allows you to reliably deliver streaming data from multiple sources within AWS. Observe supports ingesting data through the Amazon Kinesis HTTP endpoint.

Note

If you want to ingest an Amazon Kinesis Data Stream, see Amazon Kinesis Data Stream to Observe for information about configuring a Data Stream source using Terraform.

Setup

Installation

Use the Observe CloudFormation template to automate creating an Amazon Kinesis Firehose delivery stream to send data to Observe. To install using the AWS Console, follow these steps:

  1. Navigate to the CloudFormation console and view existing stacks.

  2. Click Create stack. If prompted, select With new resources.

  3. Provide the template details:

    a. Under Specify template, select Amazon S3 URL.

    b. In the Amazon S3 URL field, enter the URL for the Amazon Kinesis Firehose CloudFormation template: https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/firehose-latest.yaml

Observe recommends that you pin the template version to a tagged version of the Amazon Kinesis Firehose template.

To do this, replace latest in the template URL with the desired version tag:

        https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/firehose-v0.2.0.yaml

  1. Click Next to continue. You may be prompted to view the function in Designer mode. Click Next again to skip.

  2. Specify the stack details:

    a. In Stack name, provide a name for this stack. It must be unique within a region and used to name created resources.

    b. Under Required Parameters, provide your Customer ID in ObserveCustomer and ingest token in ObserveToken.

Note

Some Observe instances may optionally use a name instead of Customer ID; if this is the case for your instance, contact your Observe Data Engineer to discuss implementation. A stem name will work as is, but a DNS redirect name may require client configuration.

c. Click Next.

  1. Under Configure stack options, there are no required options to configure. Click Next to continue.

  2. Review your stack options:

    a. Under Capabilities, check the box to acknowledge that this stack may create IAM resources.

    b. Click Create stack.

Video instructions

Alternatively, you can deploy the AWS CloudFormation template using the awscli utility:

Caution

If you have multiple AWS profiles, make sure you configure the appropriate AWS_REGION and AWS_PROFILE environment variables in addition to OBSERVE_CUSTOMER and OBSERVE_TOKEN.

$ curl -Lo firehose.yaml https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/firehose-latest.yaml
$ aws cloudformation deploy --template-file ./firehose.yaml \
	  --stack-name ObserveLambda \
	  --capabilities CAPABILITY_NAMED_IAM \
	  --parameter-overrides ObserveCustomer="${OBSERVE_CUSTOMER?}" ObserveToken="${OBSERVE_TOKEN?}"

You may also use the Observe observe_kinesis_firehose Terraform module to create an Amazon Kinesis Firehose delivery stream.

The following is an example instantiation of this module:

module "observe_kinesis_firehose" {
  source = "github.com/observeinc/terraform-aws-kinesis-firehose"

  name                = "observe-kinesis-firehose"
  observe_customer    = "${OBSERVE_CUSTOMER}"
  observe_token       = "${OBSERVE_TOKEN}"
}

Observe recommends that you pin the module version to the latest tagged version.

Getting Started

You can now use your Amazon Kinesis Firehose delivery stream to collect a variety of sources:

Retry on Failure

Amazon Kinesis Firehose supports retries with the Retry duration time period. If a request fails repeatedly, the contents are stored in a pre-configured S3 bucket. See the Amazon Kinesis Firehose data delivery documentation for more information.