Observe by Snowflake can accept data from endpoints, sources, forwarders, and Datastreams.

Endpoints

Endpoints support various wire protocols that Observe by Snowflake can ingest. All of the source and forwarder instructions ultimately send data to an endpoint. If you have a custom or highly customized source, you may configure it to use the appropriate endpoint directly.

Data such as JSON can be pushed to Observe by Snowflake using an HTTP POST operation.

Sources

Data sources such as AWS CloudWatch logs and Jenkins build logs may send data to Observe by Snowflake directly through an outgoing Webhook, a forwarder, or other types of agents. The documentation for each source describes the recommended method and any additional installations required.

Contact your Observe by Snowflake data engineer to discuss new integrations.

Forwarders

Observe by Snowflake provides forwarders such as FluentBit and Prometheus Server to simplify the integration of many known formats and systems with Observe by Snowflake. Forwarders collect data from a source and send it to Observe by Snowflake. They offer additional features, such as the ability to aggregate data from multiple sources or perform lightweight transformations. Forwarders can be useful when the original source does not have a way to send data to Observe, such as a process that only generates a local log file.

Contact your Observe by Snowflake data engineer to discuss new integrations.

Datastreams

Observe by Snowflake uses Datastreams as a flexible way to manage data ingestion. Datastreams are associated with tokens, which are used for authorization and data routing. Once you create a token to authenticate, any data can be sent directly to Observe by Snowflake through our HTTP endpoints from a client’s app.

Data immutability

From the point of ingestion, data received by Observe by Snowflake cannot be changed. It is stored in compressed Snowflake tables, accessed via a data stream and used in data sets. Data is retained until the retention period or customer contract expires. To have specific data removed, contact your Observe by Snowflake data engineer.

Further information about relationship to other core Observe by Snowflake concepts and how they relate to Ingest and each other can be found in Key terminology and Important concepts.