Data Ingestion

Observe can accept pushed data in any format via HTTP endpoints, as well as using forwarders and pollers to pull specific types of data. To ingest data you must first create an ingestion token, which can be done manually by following these instructions, or by installing an integration.

Endpoints

Endpoints support various wire protocols that Observe can ingest. All of the source and forwarder instructions ultimately send data to an endpoint. If you have a custom or highly customized source, you may configure it to use the appropriate endpoint directly.

Example: JSON through an HTTP POST

Sources

Data Sources may send data to Observe directly through an outgoing Webhook, a forwarder, or other types of agents. The documentation for each source describes the recommended method and any additional installations required. Please contact your Observe Data Engineer to discuss new integrations.

Examples: AWS CloudWatch logs, Jenkins build logs

Forwarders

Observe provides Forwarders to simplify the integration of many known formats and systems with Observe. Forwarders collect data from a source and send it to Observe. They offer additional features, such as the ability to aggregate data from multiple sources or perform lightweight transformations. Forwarders can be useful when the original source does not have a way to send data to Observe, such as a process that only generates a local log file. Please contact your Observe Data Engineer to discuss new integrations.

Examples: FluentBit, Prometheus Server

Datastreams

Observe uses Datastreams as a flexible way to manage data ingestion. Datastreams are associated with tokens, which are used for authorization and data routing.

Using a Datastream token to authenticate, any data can be sent directly to Observe through our HTTP Endpoints from a client’s app.

Troubleshooting Data Ingestion

When you ingest different sources of data into Observe, you may encounter issues during the ingestion. Refer to Troubleshooting Data Ingestion for possible causes and solutions.

Data Immutability

From the point of ingestion, data received by Observe cannot be changed. It is stored in compressed Snowflake tables, accessed via a Datastream and used in Datasets. Data is retained until the retention period or customer contract expires. To have specific data removed, contact your Observe Data Engineer.

Further information about relationship to other core Observe concepts and how they relate to Ingest and each other can be found here.