Prometheus

Prometheus provides open source metric collection agent software. The software contains many components, but at the core is the pull-based Prometheus server. Ingest Prometheus data with either a POST request to an API endpoint or with a Kubernetes agent. Observe implements the Prometheus remote write API for both of these methods.

Installation

Please see the Prometheus documentation for installation instructions.

Configuration

The examples below describe several types of configurations for sending Prometheus data to Observe. Add the appropriate remote_write block to your prometheus.yml configuration file.

API Endpoint

remote_write:
- url: https://${OBSERVE_CUSTOMER?}.collect.observeinc.com/v1/prometheus
  bearer_token: ${OBSERVE_TOKEN?}
  remote_timeout: "30s"
  queue_config:
    min_backoff: "1s"
    max_backoff: "30s"
    max_shards: 20
    max_samples_per_send: 5000
    capacity: 15000
    retry_on_http_429: true

Note

To add additional labels to every observation, append one or more query parameters to the ingest URL:

- url: https://${OBSERVE_CUSTOMER?}.collect.observeinc.com/v1/prometheus?alt_host=this_host&alt_namespace=that_namespace

Hot/Hot Cluster

If you run two instances as a hot/hot cluster (using Prometheus Operator), you may want to ingest only one set of metrics. Use writeRelabelConfigs to drop data from one of the instances. For example, the configuration below looks for instance names in the prometheus_replica field and drops metrics from prometheus-k8s-1:

remote_write:
  - url: https://${OBSERVE_CUSTOMER?}.collect.observeinc.com/v1/prometheus
    bearer_token: ${OBSERVE_TOKEN?}
    remote_timeout: 30s
    queue_config:
      min_backoff: 1s
      max_backoff: 30s
      max_shards: 20
      max_samples_per_send: 5000
      capacity: 15000
      retry_on_http_429: true
    writeRelabelConfigs:
      - action: drop
    regex: prometheus-k8s-1
    sourceLabels:
      - prometheus_replica

Observation Format

Observations contain fields with a timestamp in milliseconds, and a float64 metric value. Any additional tags get added to the JSON object in the EXTRA field.

Column	Value
OBSERVATION_KIND	prometheus
FIELDS	{"timestamp": ,"value": 0.1}
EXTRA	{"_name_": "my_metric", "other_tags": "foo"}

FAQ

Retry on failure

Prometheus supports exponential backoff for retries. See the Prometheus Remote Write Tuning documentation for more information.

By omission, Prometheus treats 429 HTTP status codes as unrecoverable errors. The retry_on_http_429 option was introduced as an experimental feature in version 2.26.0. Observe recommends enabling it wherever possible in order to back off when submitting a large backlog of data.