Data Ingestion

This section describes the ingesting process to add data into Observe. Observe accepts data in any format. More sources or forwarders may be available in future releases.

Sources

Sources may send data to Observe directly through an outgoing Webhook, a forwarder, or other types of agents. The documentation for each source describes the recommended method and any additional installations required.

Examples: AWS CloudWatch logs, Jenkins build logs

Forwarders

Forwarders collect data from a source and send it to Observe. They offer additional features, such as the ability to aggregate data from multiple sources or perform lightweight transformations. Forwarders can be useful when the original source does not have a way to send data to Observe, such as a process that only generates a local log file.

Examples: FluentBit, Prometheus Server, Google Cloud Pub/Sub

Datastreams

Observe uses Datastreams as a flexible way to manage data ingestion. Each datastream sends a dataset, managed with unique, revokable, tokens.

Endpoints

Endpoints support various wire protocols that Observe can ingest. All of the source and forwarder instructions ultimately send data to an endpoint. If you have a custom or highly customized source, you may configure it to use the appropriate endpoint directly.

Example: JSON through an HTTP POST

Troubleshooting Data Ingestion

When you ingest different sources of data into Observe, you may encounter issues during the ingestion. Refer to Troubleshooting Data Ingestion for possible causes and solutions.