Prometheus¶

Prometheus is an open source metric collection agent. It has a lot of different components, but at its heart is the pull-based Prometheus server. Ingest Prometheus data with either a POST request to an API endpoint or with a Kubernetes agent. Observe implements the Prometheus remote write API for both of these methods.

Installation¶

Please see the Prometheus documentation for installation instructions.

Configuration¶

The examples below describe several types of configurations for sending Prometheus data to Observe. Add the appropriate remote_write block to your prometheus.yml configuration file.

API Endpoint¶

remote_write:
- url: https://collect.observeinc.com/v1/prometheus
  bearer_token: "my_customer_id my_ingest_token"
  remote_timeout: "30s"
  queue_config:
    min_backoff: "1s"
    max_backoff: "30s"
    max_shards: 20
    max_samples_per_send: 5000
    capacity: 15000

bearer_token is your customer ID and ingest token as a single space-delimited string:

    bearer_token: "1234567890 my_ingest_token"

Note

To add additional labels to every observation, append one or more query parameters to the ingest URL:

- url: https://collect.observeinc.com/v1/prometheus?alt_host=this_host&alt_namespace=that_namespace

Kubernetes Agent¶

remote_write:
- url: http://proxy.observe.svc.cluster.local:2001/
  remote_timeout: "30s"
  queue_config:
    min_backoff: "1s"
    max_backoff: "30s"
    max_shards: 20
    max_samples_per_send: 5000
    capacity: 15000

This method does not require bearer_token, as the daemonset already has it.

Hot/Hot Cluster¶

If you run two instances as a hot/hot cluster (using Prometheus Operator), you may wish to ingest only one set of metrics. Use writeRelabelConfigs to drop data from one of the instances. For example, the configuration below looks for instance names in the prometheus_replica field and drops metrics from prometheus-k8s-1:

remote_write:
  - url: 'http://proxy.observe.svc.cluster.local:2001/'
    remote_timeout: 30s
    queue_config:
      min_backoff: 1s
      max_backoff: 30s
      max_shards: 20
      max_samples_per_send: 5000
      capacity: 15000
    writeRelabelConfigs:
      - action: drop
    regex: prometheus-k8s-1
    sourceLabels:
      - prometheus_replica

Observation Format¶

Observations are of kind prometheus. Fields contains a timestamp in milliseconds, and a float64 metric value. Any additional tags are added to the JSON object in the EXTRA field.

Column	Value
OBSERVATION_KIND	prometheus
FIELDS	{"timestamp": ,"value": 0.1}
EXTRA	{"_name_": "my_metric", "other_tags": "foo"}

FAQ¶

Retry on failure¶

Prometheus supports exponential backoff for retries. See the Prometheus Remote Write Tuning documentation for more information.