Endpoints

Unlike most legacy systems, Observe does not have a preferred format over which data must be exchanged. Instead, Observe strives to accept any data format by natively supporting existing wire protocols.

Observe maintains multiple collection endpoints, each implementing a concrete protocol or API. The currently supported endpoints are:

Endpoint behavior

While each endpoint implements a different API, all endpoints behave in a consistent manner.

  • Data must be exchanged over TLS. Data emitted using protocols that do not support TLS (such as collectd and statsd) must be proxied through a forwarder.

  • Each endpoint has a unique path under the base domain https://collect/observeinc.com/v1 This path identifies the type of data being ingested, and is the preferred method.

  • For legacy clients, endpoints also have a subdomain under collect.observeinc.com. Many clients only support configuring the domain name of the endpoint data is sent to. For this reason, each endpoint is available through a subdomain of collect.observeinc.com. An endpoint $foo will be available on https://$foo.collect.observeinc.com.

    Internally, requests to https://$foo.collect.observeinc.com/ are rewritten to https://collect.observeinc.com/v1/$foo/. While subdomains are practical for legacy clients, paths can be simpler for new integrations or SDKs. Both methods implement the same underlying API, but may differ in how they handle authentication.

  • The endpoint used dictates the observation kind. Data ingested through endpoint $foo creates observations of kind $foo.

  • Query parameters are encoded as tags. Query parameters, except for those that are part of the endpoint protocol, are encoded as tags on the ingested data. This provides a simple, generic mechanism for tagging data without the need to modify sources.

  • A single observation cannot exceed 1MB. Independently of what endpoint data comes in through, a single, uncompressed observation cannot exceed 1MB. This is a hard limit of our stream processing pipeline.

Authentication

All requests to Observe’s collection API must be authenticated.

Authentication requires the following attributes:

Variable

Description

${OBSERVE_CUSTOMER}

A 12 digit number which identifies your account, present in the URL you use to log into Observe (https://${OBSERVE_CUSTOMER}.observeinc.com)

${OBSERVE_TOKEN}

A token with write-access to the collector service in your customer account. Please contact support if you are missing this information.

Requests must provide these values in an HTTP header. If the Authentication header is missing, any request to endpoints under the https://collect.observeinc.com base domain will return 401 Unauthorized.

$ curl -f https://collect.observeinc.com
curl: (22) The requested URL returned error: 401

Observe supports three authentication methods: Basic auth, Bearer token, and Endpoint-specific auth.

Basic auth

For Basic authentication, provide your customer ID as the username, and your token as the password. In curl, this can be done through the --user argument. You can verify your credentials by issuing a GET against https://collect.observeinc.com.

$ curl https://collect.observeinc.com --user ${OBSERVE_CUSTOMER?}:${OBSERVE_TOKEN?}

Bearer token

For Bearer authentication, append your customer ID and token to the authorization header as a bearer token. The following curl command verifies the provided credentials are valid:

$ curl https://collect.observeinc.com -H Authorization:"Bearer ${OBSERVE_CUSTOMER?} ${OBSERVE_TOKEN?}"

Endpoint-specific

Requests to the base domain https://collect.observeinc.com require either Basic auth or a Bearer token. Endpoint subdomains, on the other hand, are designed to mimic third party protocols. They may need to conform to a concrete specification.

For example, the AWS Kinesis Firehose protocol encodes credentials in the X-Amz-Firehose-Access-Key HTTP header. This header is recognized by https://kinesis.collect.observeinc.com, but not by the more specific https://collect.observeinc.com/v1/kinesis path. The subdomain version respects external requirements defined by the protocol specification, while the path version behaves as part of Observe’s API, and therefore expects credentials to be encoded in the Authorization header. The documentation for each type of endpoint describes which authentication methods it accepts.