Endpoints

Unlike most legacy systems, Observe does not have a preferred format over which data must be exchanged. Instead, Observe strives to accept any data format by natively supporting existing wire protocols.

Observe maintains multiple collection endpoints, each implementing a concrete protocol or API. The currently supported endpoints are:

Endpoint behavior

While each endpoint implements a different API, all endpoints behave in a consistent manner.

  • Data must be exchanged over TLS. Data emitted using protocols that do not support TLS (such as collectd and statsd) must be proxied through a forwarder.

  • Each endpoint has a unique path under the base domain https://collect.observeinc.com/v1 This path identifies the type of data being ingested, and is the preferred method.

  • The endpoint used dictates the observation kind. Data ingested through endpoint $foo creates observations of kind $foo.

  • Query parameters are encoded as tags. Query parameters, except for those that are part of the endpoint protocol, are encoded as tags on the ingested data. This provides a simple, generic mechanism for tagging data without the need to modify sources.

  • A single observation cannot exceed 1MB. Independently of what endpoint data comes in through, a single, uncompressed observation cannot exceed 1MB. This is a hard limit of our stream processing pipeline.

  • A single request cannot exceed 10MB. You can batch multiple observations into a single HTTP request, but no endpoint will accept a payload larger than 10MB. When uncompressed, the payload cannot exceed 50MB. If you expect to submit large volumes of data, we recommend keeping request sizes under 4MB, and increasing the number of concurrent uploads until you achieve your desired throughput.The API will respond with a 429 status code if you exceed your rate limit. Please contact support if you need your rate limit to be adjusted.

Authentication

All requests to Observe’s collection API must be authenticated.

Authentication requires the following attributes:

Variable

Description

${OBSERVE_CUSTOMER}

A 12 digit number which identifies your account, present in the URL you use to log into Observe (https://${OBSERVE_CUSTOMER}.observeinc.com)

${OBSERVE_TOKEN}

A token with write-access to the collector service in your customer account. Please contact support if you are missing this information.

Requests must provide these values in an HTTP header. If the Authentication header is missing, any request to endpoints under the https://collect.observeinc.com base domain will return 401 Unauthorized.

$ curl -f https://collect.observeinc.com
curl: (22) The requested URL returned error: 401

Observe supports three authentication methods: Basic auth, Bearer token, and Endpoint-specific auth.

Basic auth

For Basic authentication, provide your customer ID as the username, and your token as the password. In curl, this can be done through the --user argument. You can verify your credentials by issuing a GET against https://collect.observeinc.com.

$ curl https://collect.observeinc.com --user ${OBSERVE_CUSTOMER?}:${OBSERVE_TOKEN?}

Bearer token

For Bearer authentication, append your customer ID and token to the authorization header as a bearer token. The following curl command verifies the provided credentials are valid:

$ curl https://collect.observeinc.com -H Authorization:"Bearer ${OBSERVE_CUSTOMER?} ${OBSERVE_TOKEN?}"

Endpoint-specific

All requests to https://collect.observeinc.com require either Basic auth or a Bearer token.

Some endpoints support protocols which encode credentials in locations other than the Authorization header. For example, the AWS Kinesis Firehose protocol encodes credentials in the X-Amz-Firehose-Access-Key HTTP header. For such cases, we rewrite the authorization header at our ingress load balancer. The documentation for each type of endpoint describes which authentication methods it accepts.