Observe by Snowflake doesn't have a preferred format for exchanging data. Any data format is accepted by natively supporting existing wire protocols.

Endpoints support various wire protocols that Observe by Snowflake can ingest. All of the source and forwarder instructions ultimately send data to an endpoint. If you have a custom or highly customized source, you may configure it to use the appropriate endpoint directly.

Before sending data, create a compatible source dataset and an ingest token (or reuse existing source datasets / ingest tokens), then configure routing. Existing Datastream clients can continue using their associated tokens.

Copy your collection URL from the account's setup instructions. Regional accounts can use a different hostname from the examples below. Do not construct a collection hostname solely from the account ID.

Observe by Snowflake maintains multiple collection endpoints, each implementing a concrete protocol or API.

The currently supported endpoints include the following:

• Datadog Metrics
• Elasticsearch
• HTTP
• Kinesis
• OpenTelemetry
• Prometheus

Endpoint behavior

While each endpoint implements a different API, all endpoints behave consistently.

• Data must be exchanged over TLS.
  Data emitted using protocols that do not support TLS, such as collectd and statsd, must be proxied through a forwarder.

• Use the account-specific collection hostname. For example, ${OBSERVE_CUSTOMER_ID}.collect.observeinc.com. Preserve any regional domain from your setup instructions. Observe by Snowflake uses the account-specific hostname to route and load balance traffic within its ingest architecture.

• Each endpoint has a unique path under the base domainhttps://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com/
  This path identifies the type of ingested data.

• The endpoint used dictates the observation kind.
  Data ingested through endpoint $server1 creates observations of the kind $server1.

• Query parameters are encoded as tags.
  Query parameters, except those part of the endpoint protocol, are encoded as tags on the ingested data. This provides a simple, generic mechanism for tagging data without modifying sources.

• A single observation cannot exceed 4MB.
  Independently of incoming endpoint data, a single, uncompressed observation cannot exceed 4MB. This is a hard limit of the stream processing pipeline.

• A single request cannot exceed 10 MB.
  You can batch multiple observations into a single HTTP request, but no endpoint accepts a payload larger than 10 MB. When uncompressed, the payload cannot exceed 50 MB. If you expect to submit large volumes of data, Observe by Snowflake recommends keeping request sizes under 4 MB and increasing the number of concurrent uploads until you achieve your desired throughput. The API responds with a 413 status code if you exceed your size limit. Contact your Observe by Snowflake data engineer if you want your size limit adjusted. You can compress your file and submit it as a single HTTP request.

• Rate limit
  The API responds with a 429 status code (Too Many Requests) if you exceed your rate limit. Please contact your Observe by Snowflake data engineer if you want your rate limit adjusted. Any request that receives a 429 status code can be retried.

Authentication

All requests to the Observe by Snowflake collection APIs must be authenticated.

Authentication requires the following attributes:

The collection hostname identifies the account. Send the complete credential in an Authorization header as a bearer token. Missing or invalid credentials result in 401 Unauthorized.

Authorization: Bearer <token-id>:<secret>

For clients already configured with destination routing, the following request checks authentication:

curl --fail-with-body --silent --show-error "${OBSERVE_COLLECT_URL:?}/" \
  -H "Authorization: Bearer ${OBSERVE_TOKEN:?}"

$ curl https://123456789012.collect.observeinc.com -H Authorization:"Bearer ds1mIe3aT7ayQllyFKLT:v7IQpFgDhwBan3WPsFOaTi2uunE-29Ei"
{"ok":true,"message":"💯"}

Some protocols carry the credential in their own field instead of an Authorization header. The Kinesis and Datadog endpoints document those fields. Do not place the credential in a URL or query parameter, where client, proxy, and application logs can capture it.

An authentication check does not send telemetry or verify its destination. To direct ingest observations to particular source datasets, configure routing and test the actual signal endpoint.

Example: Ingesting CSV data using cURL

Use a cURL statement to ingest Comma-Separated Values (CSV) data into Observe by Snowflake. Before ingesting CSV data, be sure to review the following Prerequisites:

• The CSV file contains headers.
• Has a field or column that can be used as a primary key.
• Create a Generic - Any Data source dataset and an ingest token (or reuse existing source datasets / ingest tokens), and review the routing for the Generic - Any Data type.
• The file size must be less than 10MB.
• Ensure that you specify a path for each CSV file. For example, a customer.csv file uses the path https://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com/v1/http/csv/customers.csv.

Ingestion Example

curl --fail-with-body --silent --show-error "${OBSERVE_COLLECT_URL:?}/v1/http/csv/customers.csv" \
    -H "Authorization: Bearer ${OBSERVE_TOKEN:?}" \
    -H 'Content-type: text/csv' \
    --data-binary @customers.csv

Verify the CSV records in Observe by Snowflake after a successful response.

Friendly stem names

It is possible to add an alias that replaces your 12 digit ${OBSERVE_CUSTOMER_ID} with a friendly name such as my-observe-instance. The name must be unique within its Observe by Snowflake cluster, between 3 and 48 characters, and composed of lowercase alphanumeric and hyphen characters only. Contact your Observe by Snowflake data engineer to learn more.