HTTP¶

The http API is unique among the supported endpoints because it does not implement an existing specification. It is a generic endpoint and a convenient method of ingesting data over HTTP.

Endpoint	http
URL	`https://${OBSERVE_CUSTOMER}.collect.observeinc.com/v1/http`
Maximum Request Size	10MB compressed (50mb uncompressed)

Ingesting data over HTTP¶

Prerequisite: You must configure a datastream and token before proceeding.

You can send a POST request to Observe. Observe makes sense of the POST request based on the following principles:

Observe parses the request body according to the content type header.
The path component of the URL encodes as a tag.
Query parameters encoded as tags.
Up to 4 MB per observation, 10 MB per payload, or 50 MB after decompression per payload.
Depending on the type of payload, further limits may apply.

As an example, the following POST:

$ curl -X POST https://${OBSERVE_CUSTOMER?}.collect.observeinc.com/v1/http/first/example?key=value \
	-H "Authorization: Bearer ${OBSERVE_DATASTREAM_TOKEN}" \
	-H "Content-Type: application/json" \
	-d '{"message":"Hello World!"}'

Note

${OBSERVE_DATASTREAM_TOKEN} is the Token created when you added a Datastream. It consists of 20 random alphanumeric digits, a colon, and then 32 random alphanumeric digits.

For example, the Token may be generated as follows: ab12C3de0FG4hIjklmn5:A123Bc_D4e5fghIjkl_MNO67PQ8Rstuv

Returns in an observation with the following values:

Column	Value
OBSERVATION_KIND	http
FIELDS	{"message": "Hello World!"}
EXTRA	{"key": "value", "path": "/first/example"}

The observation fields are based on the request body, while path and query parameters are encoded in EXTRA. All observations in a payload are given the same EXTRA metadata. HTTP headers determine how the content is parsed.

For more about authentication, including Bearer token, see Authentication.

Supported content types¶

The HTTP endpoint supports the following content type values:

Content-Type	Description
application/json	Parses a single JSON object or array of objects. Each object is a unique observation.
application/x-ndjson	Parses a stream of newline delimited JSON objects. Each object is a unique observation.
application/xml	Internally converts XML object to JSON, and processes it according to `application/json` handling.
application/msgpack	Parses an array of objects.
text/csv	Generates one observation per CSV record, using the fields defined in the header row. Empty values are omitted.
application/x-csv	The default CSV delimiter is `,` (comma). In order to specify alternative delimiters, use `application/x-csv` with a delimiter HTTP content-type parameter. For example, set `content-type` to `application/x-csv;delimiter=space` for space delimited CSV data. Supported values for the delimiter are comma, space, and tab.
text/plain	Generates one observation per line.

Note

Several additional headers also control data parsing.

CSV Ingestion Example¶

Use curl to ingest Comma-Separated Values (CSV) data into Observe. For instance to ingest a example.csv file, use the following curl:

curl https://123456789012.collect.observeinc.com/v1/http \
 -H "Authorization: Bearer ab1cdE2FGhiJKlmnop34Q:rstUv5w6Xy7z8AB_CdeFg9h0iJK1mnOPqr" \
 -H "Content-type: text/csv" \
 --data-binary @./example.csv

After a minute or so, you can see the data in the appropriate Datastream.

Note

CSV is limited to 1 MB per row

JSON Ingestion Examples¶

Object¶

You can submit an object by setting the Content-Type header to application/json. The object results in a single observation.

Payload

{
    "message": "Hello World!"
}

Curl

$ curl -X POST https://${OBSERVE_CUSTOMER?}.collect.observeinc.com/v1/http/example \
	--user ${OBSERVE_TOKEN?} --data-binary @payload \
	-H 'Content-Type: application/json'

Array of objects¶

You can submit an array of objects by setting the Content-Type header to application/json. Each object results in a separate observation.

Payload

[
  {
    "id": 1
  },
  {
    "id": 2
  }
]

Curl

$ curl -X POST https://${OBSERVE_CUSTOMER?}.collect.observeinc.com/v1/http/example \
	--user ${OBSERVE_TOKEN?} --data-binary @payload \
	-H 'Content-Type: application/json'

Newline delimited objects¶

If you have a stream of newline delimited objects, you can submit your data as content type application/x-ndjson. Each object results in a separate observation.

Payload

{
  "id": 1
}
{
  "id": 2
}
{
  "id": 3
}

Curl

$ curl -X POST https://${OBSERVE_CUSTOMER?}.collect.observeinc.com/v1/http/example \
	--user ${OBSERVE_TOKEN?} --data-binary @payload \
	-H 'Content-Type: application/x-ndjson'

Compression¶

The Observe HTTP Endpoint supports gzip compressed data, by adding an encoding header. For example:

curl https://123456789012.collect.observeinc.com/v1/http \
 -H "Authorization: Bearer ab1cdE2FGhiJKlmnop34Q:rstUv5w6Xy7z8AB_CdeFg9h0iJK1mnOPqr" \
 -H "Content-type: text/csv" \
 -H "Content-encoding: gzip" \
 --data-binary @./example.csv.gz