Prerequisites: Configure a Generic - Any Data source dataset, an ingest token (or reuse existing source datasets / ingest tokens), and routing. Existing Datastream clients can continue using their associated tokens.

The http API is unique among the supported endpoints because it does not implement an existing specification. It is a generic endpoint and a convenient method of ingesting data over HTTP.

• Endpoint: /http
• URL: https://${OBSERVE_CUSTOMER}.collect.observeinc.com/v1/http
• Maximum request size: 10MB compressed (50mb uncompressed)

Prerequisites

You must configure a Datastream and token before proceeding.

Ingest data over HTTP

You can send a POST request to Observe by Snowflake, which makes sense of the POST request based on the following principles:

• Observe by Snowflake parses the request body according to the content type header.
• The path component of the URL encodes as a tag.
• Query parameters encoded as tags.
• Up to 4 MB per observation, 10 MB per payload, or 50 MB after decompression per payload.
• Depending on the type of payload, further limits may apply.

Set OBSERVE_COLLECT_URL to your account's HTTPS collection origin without a trailing slash. Load OBSERVE_TOKEN from your secret manager. Before sending data, configure an enabled route that matches your observations and verify the default destination.

For example, send this POST:

curl --fail-with-body --silent --show-error "${OBSERVE_COLLECT_URL:?}/v1/http/first/example?key=value" \
  -H "Authorization: Bearer ${OBSERVE_TOKEN:?}" \
  -H "Content-Type: application/json" \
  -d '{"message":"Hello World!"}'

Returns in an observation with the following values:

The observation fields are based on the request body, while path and query parameters are encoded in EXTRA. All observations in a payload are given the same EXTRA metadata. HTTP headers determine how the content is parsed.

See Endpoint authentication for supported credential formats. For integrations that require a destination header, see Choose a routing method.

Supported content types

The HTTP endpoint supports the following content type values:

CSV ingestion example

Use curl to ingest Comma-Separated Values (CSV) data into Observe by Snowflake. For instance to ingest a example.csv file, use the following curl:

curl --fail-with-body --silent --show-error "${OBSERVE_COLLECT_URL:?}/v1/http" \
  -H "Authorization: Bearer ${OBSERVE_TOKEN:?}" \
  -H "Content-type: text/csv" \
  --data-binary @./example.csv

After ingestion, open the destination source dataset and verify the CSV records. If they are missing, check ingest activity and routing.

JSON ingestion examples

Object

You can submit an object by setting the Content-Type header to application/json.
The object results in a single observation.

{
"message": "Hello World!"
}

$ curl -X POST https://${OBSERVE_CUSTOMER?}.collect.observeinc.com/v1/http/example \
-H "Authorization: Bearer ${OBSERVE_TOKEN:?}" --data-binary @payload \
-H 'Content-Type: application/json'

Array of objects

You can submit an array of objects by setting the Content-Type header to application/json.
Each object results in a separate observation.

[
{
"id": 1
},
{
"id": 2
}
]

$ curl -X POST https://${OBSERVE_CUSTOMER?}.collect.observeinc.com/v1/http/example \
-H "Authorization: Bearer ${OBSERVE_TOKEN:?}" --data-binary @payload \
-H 'Content-Type: application/json'

Newline delimited objects

If you have a stream of newline delimited objects, you can submit your data as content type application/x-ndjson. Each object results in a separate observation.

{
"id": 1
}
{
"id": 2
}
{
"id": 3
}

$ curl -X POST https://${OBSERVE_CUSTOMER?}.collect.observeinc.com/v1/http/example \
-H "Authorization: Bearer ${OBSERVE_TOKEN:?}" --data-binary @payload \
-H 'Content-Type: application/x-ndjson'

Compression

The Observe by Snowflake HTTP Endpoint supports gzip compressed data, by adding an encoding header. For example:

curl https://123456789012.collect.observeinc.com/v1/http \
 -H "Authorization: Bearer ab1cdE2FGhiJKlmnop34Q:rstUv5w6Xy7z8AB_CdeFg9h0iJK1mnOPqr" \
 -H "Content-type: text/csv" \
 -H "Content-encoding: gzip" \
 --data-binary @./example.csv.gz