Data Ingestion

This section describes the ingesting process to add data into Observe. Observe accepts data in any format.

Observe ingests data using Datastreams. Datastreams are associated with a number of tokens used for authorization and routing.

Using a Datastream token to authenticate, any data can be sent directly to Observe through our HTTP Endpoints from a client’s app.

Observe provides Sources and Forwarders to simplify the integration of many known formats and systems with Observe. Please contact Observe Support to discuss new integrations.

More information about Ingest concepts and available integrations can be found in the corresponding section below and concept specific pages.

Further information about relationship to other core Observe concepts and how they relate to Ingest and each other can be found here.

Datastreams

Observe uses Datastreams as a flexible way to manage data ingestion. Each datastream sends a dataset, managed with unique, revokable, tokens.

Sources

Sources may send data to Observe directly through an outgoing Webhook, a forwarder, or other types of agents. The documentation for each source describes the recommended method and any additional installations required.

Examples: AWS CloudWatch logs, Jenkins build logs

Forwarders

Forwarders collect data from a source and send it to Observe. They offer additional features, such as the ability to aggregate data from multiple sources or perform lightweight transformations. Forwarders can be useful when the original source does not have a way to send data to Observe, such as a process that only generates a local log file.

Examples: FluentBit, Prometheus Server

Endpoints

Endpoints support various wire protocols that Observe can ingest. All of the source and forwarder instructions ultimately send data to an endpoint. If you have a custom or highly customized source, you may configure it to use the appropriate endpoint directly.

Example: JSON through an HTTP POST

Troubleshooting Data Ingestion

When you ingest different sources of data into Observe, you may encounter issues during the ingestion. Refer to Troubleshooting Data Ingestion for possible causes and solutions.

Configurable Data Retention

Observe retains your data for thirteen (13) months by default. You can configure the data retention period for less or more than 13 months by contacting Observe Support. Data older than the retention period automatically deletes from the Dataset.

Downstream Datasets inherit the retention period from the Datastream as the minimum value for all input source Datastreams. Observe enforces data retention once a day.

As an additional benefit, shortening the data retention period can reduce your storage costs.

Data Immutability

From the point of ingestion, data received by Observe cannot be changed. It is stored in compressed Snowflake tables, accessed via a Datastream and used in Datasets. Data is retained until the retention period or customer contract expires. To have specific data removed, contact Observe Support.