Data ingestion¶
This section describes how to ingest data into Observe. Our goal is to accept data in any format: if your source or forwarder isn’t documented yet, let us know!
This documentation is divided into four categories:
Integrations¶
Integrations streamline the process of collecting data from multiple sources. Where possible, we recommend starting with an integration and adding additional sources as needed.
Example: Kubernetes
Sources¶
Sources may send data to Observe directly via an outgoing webhook, a forwarder, or another type of agent. The documentation for each source describes the recommended method and any additional installations required.
Examples: AWS CloudWatch logs, Jenkins build logs
Forwarders¶
Forwarders collect data from a source and send it to Observe. They often offer additional features, such as the ability to aggregate data from multiple sources or perform lightweight transformations. Forwarders are useful when the original source does not have a way to send data elsewhere, such as a process that only generates a local log file.
Examples: FluentBit, Prometheus Server, Google Cloud Pub/Sub
Endpoints¶
Endpoints support various wire protocols by which data can be ingested. All of our source and forwarder instructions ultimately send data to an endpoint. If you have a custom or highly customized source, you may configure it to use the appropriate endpoint directly.
Example: JSON via HTTP POST