Amazon Kinesis Firehose

Amazon Kinesis Data Firehose allows you to reliably deliver streaming data from multiple sources within AWS. Observe supports ingesting data through the Kinesis HTTP endpoint.

Note

If you would like to ingest a Kinesis Data Stream, see Kinesis Data Stream to Observe for information about configuring a Data Stream source using Terraform.

Setup

Installation

Use our CloudFormation template to automate creating a Kinesis Firehose delivery stream to send data to Observe. To install using the AWS Console, follow these steps:

  1. Navigate to the CloudFormation console and view existing stacks.

  2. Click Create stack. If prompted, select With new resources.

  3. Provide the template details:

    1. Under Specify template, select Amazon S3 URL.

    2. In the Amazon S3 URL field, enter https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/firehose.yaml.

    3. Click Next to continue. (You may be prompted to view the function in Designer. Click Next again to skip.)

  4. Specify the stack details:

    1. In Stack name, provide a name for this stack. It must be unique within a region, and is used to name created resources.

    2. Under Required Parameters, provide your Customer ID in ObserveCustomer and ingest token in ObserveToken.

    3. Click Next

  5. Under Configure stack options, there are no required options to configure. Click Next to continue.

  6. Review your stack options:

    1. Under Capabilities, check the box to acknowledge that this stack may create IAM resources.

    2. Click Create stack

Video instructions

Alternatively, you can deploy the CloudFormation template using the awscli utility:

Caution

If you have multiple AWS profiles, make sure you configure the appropriate AWS_REGION and AWS_PROFILE environment variables in addition to OBSERVE_CUSTOMER and OBSERVE_TOKEN.

$ curl -Lo firehose.yaml https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/firehose.yaml
$ aws cloudformation deploy --template-file ./firehose.yaml \
	  --stack-name ObserveLambda \
	  --capabilities CAPABILITY_NAMED_IAM \
	  --parameter-overrides ObserveCustomer="${OBSERVE_CUSTOMER?}" ObserveToken="${OBSERVE_TOKEN?}"

You may also use the Observe observe_kinesis_firehose Terraform module to create a Kinesis Firehose delivery stream.

The following is an example instantiation of this module:

module "observe_kinesis_firehose" {
  source = "github.com/observeinc/terraform-aws-kinesis-firehose"

  name                = "observe-kinesis-firehose"
  observe_customer    = "${OBSERVE_CUSTOMER}"
  observe_token       = "${OBSERVE_TOKEN}"
}

We recommend that you pin the module version to the latest tagged version.

Getting Started

You can now use your Kinesis Firehose delivery stream to collect a variety of sources:

Retry on failure

Amazon Kinesis Firehose supports retries with the Retry duration time period. If a request fails repeatedly, the contents are stored in a pre-configured S3 bucket. See the Amazon Kinesis Firehose data delivery documentation for more information.