Data ingestion

This section describes how to ingest data into Observe. Our goal is to accept data in any format: if your source or forwarder isn’t documented yet, let us know!

This documentation is divided into several sections:

Sources

Sources may send data to Observe directly via an outgoing webhook, a forwarder, or another type of agent. The documentation for each source describes the recommended method and any additional installations required.

Examples: AWS CloudWatch logs, Jenkins build logs

Forwarders

Forwarders collect data from a source and send it to Observe. They often offer additional features, such as the ability to aggregate data from multiple sources or perform lightweight transformations. Forwarders are useful when the original source does not have a way to send data elsewhere, such as a process that only generates a local log file.

Examples: FluentBit, Prometheus Server, Google Cloud Pub/Sub

Data streams

Data streams are a flexible way to manage data ingestion. Each datastream sends to its own dataset, managed with unique, revokable, tokens

Endpoints

Endpoints support various wire protocols by which data can be ingested. All of our source and forwarder instructions ultimately send data to an endpoint. If you have a custom or highly customized source, you may configure it to use the appropriate endpoint directly.

Example: JSON via HTTP POST