Endpoints

Unlike most legacy systems, Observe does not have a preferred format over which data must be exchanged. Instead, Observe accepts any data format by natively supporting existing wire protocols.

Observe maintains multiple collection endpoints, each implementing a concrete protocol or API. The currently supported endpoints include the following:

Endpoint behavior

While each endpoint implements a different API, all endpoints behave consistently.

  • Data must be exchanged over TLS. Data emitted using protocols that do not support TLS, such as collectd and statsd, must be proxied through a forwarder.

  • Data should be posted to the customer-specific hostname ${OBSERVE_CUSTOMER}.collect.observeinc.com. Observe uses the customer-specific hostname to route and load balance traffic within our ingest architecture.

  • Each endpoint has a unique path under the base domain https://${OBSERVE_CUSTOMER}.collect.observeinc.com/ This path identifies the type of data being ingested.

  • The endpoint used dictates the observation kind. Data ingested through endpoint $server1 creates observations of the kind $server1.

  • Query parameters are encoded as tags. Query parameters, except those part of the endpoint protocol, are encoded as tags on the ingested data. This provides a simple, generic mechanism for tagging data without modifying sources.

  • A single observation cannot exceed 1MB. Independently of incoming endpoint data, a single, uncompressed observation cannot exceed 1MB. This is a hard limit of the stream processing pipeline.

  • A single request cannot exceed 10MB. You can batch multiple observations into a single HTTP request, but no endpoint accepts a payload larger than 10MB. When uncompressed, the payload cannot exceed 50MB. If you expect to submit large volumes of data, Observe recommends keeping request sizes under 4MB and increasing the number of concurrent uploads until you achieve your desired throughput. The API responds with a 429 status code if you exceed your rate limit. Please contact Support if you want your rate limit adjusted.

Authentication

All requests to Observe’s collection API must be authenticated.

Authentication requires the following attributes:

Variable

Description

${OBSERVE_CUSTOMER}

A 12-digit number that identifies your account, and is displayed in the URL you use to log into Observe (https://${OBSERVE_CUSTOMER}.observeinc.com)

${OBSERVE_TOKEN}

A token with write-access to a datastream in your customer account. A datastream token contains a colon separating the token identifier from the token secret.

Requests must provide these values in an HTTP header. If the Authentication header is missing, any request to endpoints under the https://${OBSERVE_CUSTOMER?}.collect.observeinc.com base domain returns 401 Unauthorized.

$ curl -f https://${OBSERVE_CUSTOMER?}.collect.observeinc.com
$ curl -f https://123456789012.collect.observeinc.com
curl: (22) The requested URL returned an error: 401

Observe supports three authentication methods: Basic auth, Bearer token, and Endpoint-specific auth.

Bearer token

Note

Observe recommends using a bearer token in the authorization header.

You can use your token as a bearer token in the authorization header. The following curl command verifies the provided credentials are valid:

$ curl https://${OBSERVE_CUSTOMER?}.collect.observeinc.com -H Authorization:"Bearer ${OBSERVE_TOKEN?}"
$ curl https://123456789012.collect.observeinc.com -H Authorization:"Bearer ds1mIe3aT7ayQllyFKLT:v7IQpFgDhwBan3WPsFOaTi2uunE-29Ei"
{"ok":true,"message":"💯"}

Basic authentication

Observe also supports basic authentication. You need to provide the token identifier as a username and the token secret as the password.

Most tools that support basic authentication expect the username and password to be separated by a single colon, :. In such cases, you can reference the Observe token directly, for example:

$ curl https://${OBSERVE_CUSTOMER?}.collect.observeinc.com --user ${OBSERVE_TOKEN}
$ curl https://123456789012.collect.observeinc.com --user ds1mIe3aT7ayQllyFKLT:v7IQpFgDhwBan3WPsFOaTi2uunE-29Ei
{"ok":true,"message":"💯"}

Alternatively, you can embed the token directly in the URL:

$ curl https://${OBSERVE_TOKEN}@${OBSERVE_CUSTOMER?}.collect.observeinc.com
$ curl https://ds1mIe3aT7ayQllyFKLT:[email protected].com
{"ok":true,"message":"💯"}

Query parameter

Some clients do not support configuring HTTP headers or configuring basic authentication. In such cases, you can embed the token as a query parameter:

$ curl https://${OBSERVE_CUSTOMER?}.collect.observeinc.com\?observe_token=${OBSERVE_TOKEN?}
$ curl https://123456789012.collect.observeinc.com\?observe_token=ds1mIe3aT7ayQllyFKLT:v7IQpFgDhwBan3WPsFOaTi2uunE-29Ei
{"ok":true,"message":"💯"}

This method should be used as a last resort since query parameters are not typically expected to contain credentials and may be more likely to be leaked through logs when configured on third-party vendors.