Unlike most legacy systems, Observe does not have a preferred format over which data must be exchanged. Instead, Observe strives to accept any data format by natively supporting existing wire protocols.

Observe maintains multiple collection endpoints, each implementing a concrete protocol or API. The currently supported endpoints are:

Endpoint behavior

While each endpoint implements a different API, all endpoints behave in a consistent manner.

  • Data must be exchanged over TLS. Data emitted using protocols that do not support TLS (such as collectd and statsd) must be proxied through a forwarder.

  • Data should be posted to the customer-specific hostname ${OBSERVE_CUSTOMER}.collect.observeinc.com. We use the customer-specific hostname to route and load balance traffic within our ingest architecture.

  • Each endpoint has a unique path under the base domain https://${OBSERVE_CUSTOMER}.collect.observeinc.com/ This path identifies the type of data being ingested.

  • The endpoint used dictates the observation kind. Data ingested through endpoint $foo creates observations of kind $foo.

  • Query parameters are encoded as tags. Query parameters, except for those that are part of the endpoint protocol, are encoded as tags on the ingested data. This provides a simple, generic mechanism for tagging data without the need to modify sources.

  • A single observation cannot exceed 1MB. Independently of what endpoint data comes in through, a single, uncompressed observation cannot exceed 1MB. This is a hard limit of our stream processing pipeline.

  • A single request cannot exceed 10MB. You can batch multiple observations into a single HTTP request, but no endpoint will accept a payload larger than 10MB. When uncompressed, the payload cannot exceed 50MB. If you expect to submit large volumes of data, we recommend keeping request sizes under 4MB, and increasing the number of concurrent uploads until you achieve your desired throughput.The API will respond with a 429 status code if you exceed your rate limit. Please contact support if you need your rate limit to be adjusted.


All requests to Observe’s collection API must be authenticated.

Authentication requires the following attributes:




A 12 digit number which identifies your account, present in the URL you use to log into Observe (https://${OBSERVE_CUSTOMER}.observeinc.com)


A token with write-access to a datastream in your customer account. A datastream token contains a colon separating the token identifier from the token secret.

Requests must provide these values in an HTTP header. If the Authentication header is missing, any request to endpoints under the https://${OBSERVE_CUSTOMER?}.collect.observeinc.com base domain will return 401 Unauthorized.

$ curl -f https://${OBSERVE_CUSTOMER?}.collect.observeinc.com
$ curl -f https://123456789012.collect.observeinc.com
curl: (22) The requested URL returned error: 401

Observe supports three authentication methods: Basic auth, Bearer token, and Endpoint-specific auth.

Bearer token


Observe recommends using a bearer token in the authorization header.

You can use your token as a bearer token in the authorization header. The following curl command verifies the provided credentials are valid:

$ curl https://${OBSERVE_CUSTOMER?}.collect.observeinc.com -H Authorization:"Bearer ${OBSERVE_TOKEN?}"
$ curl https://123456789012.collect.observeinc.com -H Authorization:"Bearer ds1mIe3aT7ayQllyFKLT:v7IQpFgDhwBan3WPsFOaTi2uunE-29Ei"

Basic authentication

Observe also supports basic authentication. You will need to provide the token identifier as a username, and the token secret as the password.

Most tools that support basic authentication expect the username and password to be separated by a single colon :. In such cases, you can reference the Observe token directly, e.g:

$ curl https://${OBSERVE_CUSTOMER?}.collect.observeinc.com --user ${OBSERVE_TOKEN}
$ curl https://123456789012.collect.observeinc.com --user ds1mIe3aT7ayQllyFKLT:v7IQpFgDhwBan3WPsFOaTi2uunE-29Ei

Alternatively, you can embed the token directly in the URL:

$ curl https://${OBSERVE_TOKEN}@${OBSERVE_CUSTOMER?}.collect.observeinc.com
$ curl https://ds1mIe3aT7ayQllyFKLT:[email protected].com

Query parameter

Some clients do not support configuring HTTP headers or configuring basic authentication. In such cases, you can embed the token as a query parameter:

$ curl https://${OBSERVE_CUSTOMER?}.collect.observeinc.com\?observe_token=${OBSERVE_TOKEN?}
$ curl https://123456789012.collect.observeinc.com\?observe_token=ds1mIe3aT7ayQllyFKLT:v7IQpFgDhwBan3WPsFOaTi2uunE-29Ei

This method should be used as a last resort, since query parameters are not typically expected to contain credentials and are therefore more likely to be leaked via logs when configured on third party vendors.