Amazon Kinesis Firehose¶

Amazon Kinesis Data Firehose allows you to reliably deliver streaming data from multiple sources within AWS. Observe supports ingesting data through the Kinesis HTTP endpoint.

Note

If you want to ingest a Kinesis Data Stream, see Kinesis Data Stream to Observe for information about configuring a Data Stream source using Terraform.

Setup¶

Installation¶

AWS Console

Use the Observe CloudFormation template to automate creating a Kinesis Firehose delivery stream to send data to Observe. To install using the AWS Console, follow these steps:

Navigate to the CloudFormation console and view existing stacks.
Click Create stack. If prompted, select With new resources.
Provide the template details: a. Under Specify template, select Amazon S3 URL. b. In the Amazon S3 URL field, enter the URL for the Kinesis Firehose CloudFormation template: https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/firehose-latest.yaml.
- We recommend you pin the template version to a tagged version of the Kinesis Firehose template.
  
  To do this, replace latest in the template URL with the desired version tag:
```
https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/firehose-v0.2.0.yaml
```
- For information about available versions, see the Kinesis Firehose CF template change log in GitHub.
Click Next to continue. You may be prompted to view the function in Designer. Click Next again to skip.
Specify the stack details:

a. In Stack name, provide a name for this stack. It must be unique within a region, and is used to name created resources.

b. Under Required Parameters, provide your Customer ID in ObserveCustomer and ingest token in ObserveToken.

c. Click Next.
Under Configure stack options, there are no required options to configure. Click Next to continue.
Review your stack options:

a. Under Capabilities, check the box to acknowledge that this stack may create IAM resources.

b. Click Create stack.

Alternatively, you can deploy the CloudFormation template using the awscli utility:

Caution

If you have multiple AWS profiles, make sure you configure the appropriate AWS_REGION and AWS_PROFILE environment variables in addition to OBSERVE_CUSTOMER and OBSERVE_TOKEN.

$ curl -Lo firehose.yaml https://observeinc.s3-us-west-2.amazonaws.com/cloudformation/firehose-latest.yaml
$ aws cloudformation deploy --template-file ./firehose.yaml \
	  --stack-name ObserveLambda \
	  --capabilities CAPABILITY_NAMED_IAM \
	  --parameter-overrides ObserveCustomer="${OBSERVE_CUSTOMER?}" ObserveToken="${OBSERVE_TOKEN?}"

Terraform

You may also use the Observe observe_kinesis_firehose Terraform module to create a Kinesis Firehose delivery stream.

The following is an example instantiation of this module:

module "observe_kinesis_firehose" {
  source = "github.com/observeinc/terraform-aws-kinesis-firehose"

  name                = "observe-kinesis-firehose"
  observe_customer    = "${OBSERVE_CUSTOMER}"
  observe_token       = "${OBSERVE_TOKEN}"
}

We recommend that you pin the module version to the latest tagged version.

Getting Started¶

You can now use your Kinesis Firehose delivery stream to collect a variety of sources:

Retry on failure¶

Amazon Kinesis Firehose supports retries with the Retry duration time period. If a request fails repeatedly, the contents are stored in a pre-configured S3 bucket. See the Amazon Kinesis Firehose data delivery documentation for more information.