Amazon CloudFront

Amazon CloudFront is a “content distribution network” that speeds up distribution of your static and dynamic web content to your users. CloudFront delivers your content through a worldwide network of data centers called edge locations. When a user requests content you’re serving with CloudFront, the request is routed to the edge location with the lowest latency (time delay.) This allows your content to be delivered with the best possible performance.

  • If the content is already in the edge location with the lowest latency, CloudFront delivers it immediately.

  • If the content is not in that edge location, CloudFront retrieves it from an origin that you’ve defined—such as an Amazon S3 bucket, a MediaPackage channel, or an HTTP server (for example, a web server) you designate as the definitive version of your content.

Observe helps you monitor the health and activity of CloudFront with:

  • A dataset that represents the core abstraction in CloudFront: a “Distribution”. This dataset helps you maintain an inventory of all of your Distributions, and inspect their configurations over time.

  • A monitoring Board that gives you at-a-glance insight into the health of your distributions (for example, request error rates and origin request latencies).

  • Additional datasets to inspect real-time Access Logs and CloudTrail Events related to your Distributions.

  • Sample Monitor configurations you can configure to enable proactive alerting upon degradation of any CloudFront Distribution’s health.

The sections below describe how to configure CloudFront ingest and view details about your distributions in Observe. If you have already configured CloudFront ingest, jump to View CloudFront activity in Observe for more about exploring CloudFront data.

Setup

Ingest CloudFront data with the AWS Integration

CloudFront ingest is a component of the Observe AWS integration. If you aren’t using the latest version, contact us about updating your ingest configuration.

Because CloudFront is a global service (not configured separately for each AWS region), the AWS Integration must be installed in the us-east-1 region. This allows it to collect CloudFront data from all regions.

Basic CloudFront collection includes configuration snapshots and CloudWatch Metrics for AWS CloudFront. You may also enable these optional data sources:

Observe does not currently support collecting CloudFront “Standard Logs” via an S3 bucket. Instead we recommend real-time logs, as log delivery is more timely and more configurable (for example, only log certain cache behaviors). If real-time logs do not meet your needs, please contact us to discuss your infrastructure details.

Important

To ingest CloudFront data, you must install the AWS Integration in us-east-1.

For more about multi-region deployments, see How do I collect data from multiple regions? in the AWS integration documentation.

Optional: enable real-time access logs

To collect real-time access logs, you will need the following resources in AWS:

  • A Kinesis Data Stream, to serve as the destination for your real-time log configuration

  • A Kinesis Data Delivery Stream, to deliver Kinesis Data Stream records to Observe

  • A CloudFront real-time log configuration, attached to your CloudFront distribution

Block diagram of ingestion path. A CloudFront distribution has a real-time log configuration attached, which pushes to a Kinesis Data Stream. A Kinesis Data Delivery Stream pulls from that Kinesis Data Stream and sends the data to Observe.

The instructions below summarize how to create these resources in the AWS console. For additional information, see [Real-time logs](Real-time logs) in the AWS documentation.

Create a Kinesis Data Stream

  1. Visit https://console.aws.amazon.com/kinesis/home?region=us-east-1#/streams/create to create a Kinesis Data Stream.

    Note

    Because CloudFront is in the us-east-1 region, also create your Kinesis Data Stream and Data Delivery Stream in us-east-1.

  2. Give the Data Stream a descriptive name, like “cloudfront-to-observe”. You may leave other settings at their default values.

  3. Click “Create data stream”.

Create a Kinesis Data Delivery Stream

  1. Visit https://console.aws.amazon.com/firehose/home?region=us-east-1#/create to create a Kinesis Data Delivery Stream (also known as “Firehose”).

  2. Under Choose source and destination, select “Amazon Kinesis Data Streams” as the source and “HTTP Endpoint” as the destination.

    Kinesis Data Delivery Stream configuration, "Choose source and destination"
  3. Under “Source settings”, click “Browse”, and select the Kinesis Data Stream you created in the previous section (“cloudfront-to-observe”.)

    Kinesis Data Delivery Stream configuration, "Choose Kinesis data stream" with "cloudfront-to-observe" selected
  4. Under Delivery stream name, provide a descriptive delivery stream name, such as cloudfront-to-observe. Leave “Transform records” disabled.

    Kinesis Data Delivery Stream configuration, "Delivery stream name" and "Transform records."
  5. Under Destination settings, use the following settings:

    1. HTTP endpoint URL: https://kinesis.collect.observeinc.com

    2. Access key: <CUSTOMER ID> <INGEST TOKEN>.

      For example, if your customer id is 123456 and your ingest token is aBcDeFg, you would use the following Access key: 123456 aBcDeFg

    3. Content encoding: GZIP

    4. Retry duration: 300 seconds

    Kinesis Data Delivery Stream configuration, "Destination settings"
  6. Under Backup settings, select Failed data only.

  7. If you don’t already have a backup S3 bucket, you may create one now by clicking Create. (You may wish to do this in a new browser tab.)

    1. Give the new bucket a unique name. For example, if your Observe Customer ID is 123456, you might name it cloudfront-to-observe-123456.

    2. Optional, but strongly encouraged: enable server-side encryption (SSE-S3 key type.)

    3. Return to the previous page or browser tab, where you were setting up the Kinesis Data Delivery Stream.

  8. Under Backup settings, click Browse to select your backup bucket.

    1. Search for the Bucket you want to use. You may need to click the “refresh” icon to get it to appear.

      Kinesis Data Delivery Stream configuration, "Choose a bucket in Amazon S3." "cloudfront-to-observe" is selected.
  9. Click “Create delivery stream”.

Create a real-time log configuration

  1. Visit https://console.aws.amazon.com/cloudfront/v3/home?region=us-east-1#/logs/create to create a CloudFront real-time log configuration.

  2. Give it a descriptive name (e.g. cloudfront-to-observe).

  3. Enter a sampling rate.

    100% is fine for distributions with modest usage, but we recommend that you start with 1% for busy distributions if you are concerned about additional AWS costs for real-time logging. You can increase the sampling rate later.

  4. Under “Fields”, select every single field.

    Important

    You must include every field to enable Observe to correctly parse real-time logs. AWS does not include a header with real-time logs, so Observe uses the relative position of each field to identify them in the incoming data.

    Real-time logs configuration, "Create configuration," "Settings" section.
  5. Select one Distribution to attach this configuration to. If you have multiple CloudFront Distributions you would like to monitor, you can attach them in the next step.

    Real-time logs configuration, "Distributions"
  6. To attach additional Distributions to this real-time logs configuration, navigate back to “CloudFront → Logs → Real-time log configurations”, and click the Configuration you just created.

    CloudFront Logs console, Real-time configurations. The "cloudfront-to-observe" configuration is attached.
  7. Scroll to the bottom and click “Attach to distribution” to attach additional distributions.

    Note

    A distribution may only have one real-time log configuration at a time. You may not attach more than one configuration to the same distribution.

    List of distributions attached to this real-time logs configuration.

Verify real-time log forwarding

To verify the real-time logs are being sent to Observe, open the “CloudFront Access Logs” dataset in Observe. If everything is working correctly, you should start to see CloudFront real-time logs in this dataset within a few minutes.

If no records appear within 5-10 minutes, inspect your Kinesis Data Delivery Stream configuration in the AWS Management Console. Verify that the “HTTP endpoint delivery success” metric indicates 100% successful delivery.

HTTP endpoint delivery success graph, showing 100% success for real-time log delivery.

If the success rate is significantly less than 100%, check the Kinesis Data Delivery Stream settings to ensure your access key is correct.

If no data is plotted (the Kinesis Data Delivery Stream isn’t receiving any data), check your real-time logging configuration to confirm it attached to the CloudFront Distribution in active use.

View CloudFront activity in Observe

To see details of your distributions, go to the “CloudFront Distributions” dataset in Observe. Select the “Monitoring” board to view an overview of the activity and health of your CloudFront distributions.

The CloudFront Distribution landing page in Observe.

This board includes several metrics useful for performance and health monitoring, including:

  • Requests: How many requests your CloudFront Distribution has served over time.

  • TotalErrorRate: What fraction of requests resulted in an HTTP error (4xx or 5xx).

  • OriginLatency: The average latency to fill cache misses from origin servers.

Access the real-time logs associated with CloudFront Distributions by either (a) opening the “CloudFront Access Logs” dataset, or (b) selecting the “CloudFront Access Logs” dataset under the “Activity” tab of the “CloudFront Distributions” page (pictured).

The CloudFront Distribution dataset in Observer, showing CloudFront real-time access logs for requests for robots.txt.

Similarly, inspect configuration events related to your distributions by inspecting the “CloudFront Distribution Events” dataset.

The CloudFront Distribution dataset in Observer, showing observations where "EventName" is "UpdateDistribution".

A handful of sample Monitor configurations are provided out-of-the-box. To create a Monitor based on a sample one, go to the “Monitors” list page and use the “Duplicate” option for your desired sample.

List of threshold monitors in Observe, showing several disabled "(template)" monitors. The menu for one of the monitors is open, with options for "Disabled", "Duplicate", and "Delete."