Prometheus¶
Prometheus is an open source metric collection agent. It has a lot of different components, but at its heart is the pull-based Prometheus server. Ingest Prometheus data with either a POST request to an API endpoint or with a Kubernetes agent. Observe implements the Prometheus remote write API for both of these methods.
Installation¶
Please see the Prometheus documentation for installation instructions.
Configuration¶
The examples below describe several types of configurations for sending Prometheus data to Observe. Add the appropriate remote_write
block to your prometheus.yml
configuration file.
API Endpoint¶
remote_write:
- url: https://collect.observeinc.com/v1/prometheus
bearer_token: "my_customer_id my_ingest_token"
remote_timeout: "30s"
queue_config:
min_backoff: "1s"
max_backoff: "30s"
max_shards: 20
max_samples_per_send: 5000
capacity: 15000
bearer_token
is your customer ID and ingest token as a single space-delimited string:
bearer_token: "1234567890 my_ingest_token"
Note
To add additional labels to every observation, append one or more query parameters to the ingest URL:
- url: https://collect.observeinc.com/v1/prometheus?alt_host=this_host&alt_namespace=that_namespace
Kubernetes Agent¶
remote_write:
- url: http://proxy.observe.svc.cluster.local:2001/
remote_timeout: "30s"
queue_config:
min_backoff: "1s"
max_backoff: "30s"
max_shards: 20
max_samples_per_send: 5000
capacity: 15000
This method does not require bearer_token
, as the daemonset already has it.
Hot/Hot Cluster¶
If you run two instances as a hot/hot cluster (using Prometheus Operator), you may wish to ingest only one set of metrics. Use writeRelabelConfigs
to drop data from one of the instances. For example, the configuration below looks for instance names in the prometheus_replica
field and drops metrics from prometheus-k8s-1
:
remote_write:
- url: 'http://proxy.observe.svc.cluster.local:2001/'
remote_timeout: 30s
queue_config:
min_backoff: 1s
max_backoff: 30s
max_shards: 20
max_samples_per_send: 5000
capacity: 15000
writeRelabelConfigs:
- action: drop
regex: prometheus-k8s-1
sourceLabels:
- prometheus_replica
Observation Format¶
Observations are of kind prometheus
. Fields contains a timestamp
in milliseconds, and a float64
metric value. Any additional tags are added to the JSON object in the EXTRA field.
Column |
Value |
---|---|
OBSERVATION_KIND |
prometheus |
FIELDS |
{"timestamp": ,"value": 0.1}
|
EXTRA |
{"_name_": "my_metric", "other_tags": "foo"}
|
FAQ¶
Retry on failure¶
Prometheus supports exponential backoff for retries. See the Prometheus Remote Write Tuning documentation for more information.