Endpoints

Unlike most legacy systems, Observe does not have a preferred format for exchanging data. Instead, Observe accepts any data format by natively supporting existing wire protocols.

Endpoints support various wire protocols that Observe can ingest. All of the source and forwarder instructions ultimately send data to an endpoint. If you have a custom or highly customized source, you may configure it to use the appropriate endpoint directly.

Observe maintains multiple collection endpoints, each implementing a concrete protocol or API. The currently supported endpoints include the following:

Endpoint behavior

While each endpoint implements a different API, all endpoints behave consistently.

  • Data must be exchanged over TLS. Data emitted using protocols that do not support TLS, such as collectd and statsd, must be proxied through a forwarder.

  • Data should be posted to the customer-specific hostname ${OBSERVE_CUSTOMER_ID}.collect.observeinc.com. Observe uses the customer-specific hostname to route and load balance traffic within Observe ingest architecture.

  • Each endpoint has a unique path under the base domain https://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com/ This path identifies the type of ingested data.

  • The endpoint used dictates the observation kind. Data ingested through endpoint $server1 creates observations of the kind $server1.

  • Query parameters are encoded as tags. Query parameters, except those part of the endpoint protocol, are encoded as tags on the ingested data. This provides a simple, generic mechanism for tagging data without modifying sources.

  • A single observation cannot exceed 4MB. Independently of incoming endpoint data, a single, uncompressed observation cannot exceed 4MB. This is a hard limit of the stream processing pipeline.

  • A single request cannot exceed 10 MB. You can batch multiple observations into a single HTTP request, but no endpoint accepts a payload larger than 10 MB. When uncompressed, the payload cannot exceed 50 MB. If you expect to submit large volumes of data, Observe recommends keeping request sizes under 4 MB and increasing the number of concurrent uploads until you achieve your desired throughput. The API responds with a 429 status code if you exceed your rate limit. Please contact your Observe Data Engineer if you want your rate limit adjusted. You can compress your file and submit it as a single HTTP request.

  • Rate limit You might see the API respond with a 429 status code or Too many requests, internal quote exceeded. If you see this status code or the error message while using one of Observe integrations, this status code is purely informational since Observe integrations retry their requests. If you see this status code or the error message while using your own code or clients, be sure you have retry logic in place to work with the 429 status code.

Authentication

All requests to the Observe collection APIs must be authenticated.

Authentication requires the following attributes:

Variable

Description

${OBSERVE_CUSTOMER}

A 12-digit number that identifies your account and is displayed in the URL you use to log into Observe (https://${OBSERVE_CUSTOMER_ID}.observeinc.com)

${OBSERVE_TOKEN}

A token with write-access to a Datastream in your customer account. A Datastream token contains a colon separating the token identifier from the token secret.

Requests must provide these values in an HTTP header. If the Authentication header is missing, any request to endpoints under the https://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com base domain returns 401 Unauthorized.

$ curl -f https://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com

$ curl -f https://123456789012.collect.observeinc.com
curl: (22) The requested URL returned an error: 401

Observe supports three authentication methods: Basic authentication, Bearer token, and Endpoint-specific authentication.

Bearer token

Datastream authentication via bearer tokens is the preferred method.

Note

The bearer token format used for datastreams is different than that used by authentication tokens.

Note

Observe recommends using a bearer token in the authorization header.

You can use your token as a bearer token in the authorization header. The following cURL command verifies the provided credentials:

$ curl https://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com -H Authorization:"Bearer ${OBSERVE_TOKEN?}"

$ curl https://123456789012.collect.observeinc.com -H Authorization:"Bearer ds1mIe3aT7ayQllyFKLT:v7IQpFgDhwBan3WPsFOaTi2uunE-29Ei"
{"ok":true,"message":"💯"}

Basic authentication

Observe also supports basic authentication. You need to provide the token identifier as a username and the token secret as the password.

Most tools that support basic authentication expect the username and password to be separated by a single colon, :. In such cases, you can reference the Observe token directly, for example:

$ curl https://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com --user ${OBSERVE_TOKEN}

$ curl https://123456789012.collect.observeinc.com --user ds1mIe3aT7ayQllyFKLT:v7IQpFgDhwBan3WPsFOaTi2uunE-29Ei
{"ok":true,"message":"💯"}

Alternatively, you can embed the token directly in the URL:

$ curl https://${OBSERVE_TOKEN}@${OBSERVE_CUSTOMER_ID}.collect.observeinc.com

$ curl https://ds1mIe3aT7ayQllyFKLT:v7IQpFgDhwBan3WPsFOaTi2uunE-29Ei@123456789012.collect.observeinc.com
{"ok":true,"message":"💯"}

Query parameter

Some clients do not support configuring HTTP headers or configuring basic authentication. In such cases, you can embed the token as a query parameter:

$ curl https://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com\?observe_token=${OBSERVE_TOKEN?}

$ curl https://123456789012.collect.observeinc.com\?observe_token=ds1mIe3aT7ayQllyFKLT:v7IQpFgDhwBan3WPsFOaTi2uunE-29Ei
{"ok":true,"message":"💯"}

This method should be used as a last resort since query parameters are not typically expected to contain credentials and may be more likely to be leaked through logs when configured on third-party vendors.

Ingesting CSV data using cURL

Use a cURL statement to ingest Comma-Separated Values (CSV) data into Observe. Before ingesting CSV data into Observe, be sure to review the following Prerequisites:

  • The CSV file contains headers.

  • Has a field or column that can be used as a primary key.

  • You must create a Datastream and an ingest token.

  • The file size must be less than 10MB.

  • Ensure that you specify a path for each CSV file. For example, a customer.csv file uses the path https://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com/v1/http/csv/customers.csv.

Ingestion Example

curl https://123456789012.collect.observeinc.com/v1/http/csv/customers.csv \
    -H 'Authorization: Bearer ${OBSERVE_TOKEN?}' \
    -H 'Content-type: text/csv' \
    --data-binary @customers.csv

{"ok":"true"}%

Friendly Stem Names

It is possible to add an alias that replaces your 12 digit ${OBSERVE_CUSTOMER_ID} with a friendly name such as my-observe-instance. The name must be unique within its Observe cluster, between 3 and 48 characters, and composed of lowercase alphanumeric and hyphen characters only. Contact your Observe Data Engineer to learn more.