Observe can accept data from endpoints, sources, forwarders, and Datastreams.

Endpoints

Endpoints support various wire protocols that Observe can ingest. All of the source and forwarder instructions ultimately send data to an endpoint. If you have a custom or highly customized source, you may configure it to use the appropriate endpoint directly.

Data such as JSON can be pushed to Observe using an HTTP POST operation.

Sources

Data sources such as AWS CloudWatch logs and Jenkins build logs may send data to Observe directly through an outgoing Webhook, a forwarder, or other types of agents. The documentation for each source describes the recommended method and any additional installations required.

Contact your Observe data engineer to discuss new integrations.

Forwarders

Observe provides forwarders such as FluentBit and Prometheus Server to simplify the integration of many known formats and systems with Observe. Forwarders collect data from a source and send it to Observe. They offer additional features, such as the ability to aggregate data from multiple sources or perform lightweight transformations. Forwarders can be useful when the original source does not have a way to send data to Observe, such as a process that only generates a local log file.

Contact your Observe data engineer to discuss new integrations.

Datastreams

Observe uses Datastreams as a flexible way to manage data ingestion. Datastreams are associated with tokens, which are used for authorization and data routing. Once you create a token to authenticate, any data can be sent directly to Observe through our HTTP endpoints from a client’s app.

Data immutability

From the point of ingestion, data received by Observe cannot be changed. It is stored in compressed Snowflake tables, accessed via a data stream and used in data sets. Data is retained until the retention period or customer contract expires. To have specific data removed, contact your Observe data engineer.

Further information about relationship to other core Observe concepts and how they relate to Ingest and each other can be found here.