Prometheus autodiscovery

Perform the tasks on this page to configure Prometheus autodiscovery.

Expose Prometheus metrics from your pod

Use one of the following annotations for your pod to expose Prometheus metrics from your pod:

  • Set the port in the annotations. This method has the disadvantage that it can only surface one HTTP endpoint for collection.

     annotations:
       prometheus.io/port: "9999"
  • Expose the metrics endpoint through a port with a name ending with metrics. For example, use the following:

    ports:
    - containerPort: 9999
      name: metrics
    - containerPort: 12345
      name: sidecar-metrics

You can use the following annotations to further influence the discovery process:

  • prometheus.io/scheme: set to http or https
  • prometheus.io/path: defaults to /metrics

To disable collecting metrics on a pod, use one of the following annotations:

  • prometheus.io/scrape: if set to false, the pod is ignored.
  • observeinc_com_scrape: 'false': used only by the Observe collection process.

Scrape Prometheus metrics from your pods

Create prometheus-scrape.yaml with the following configuration:

application:
  # use this option to scrape prometheus metrics from pods
  prometheusScrape:
    enabled: true
    interval: 60s
    # namespaces to exclude from scraping
    namespaceDropRegex: (.*istio.*|.*ingress.*|kube-system)
    # namespaces to explicity include for scraping - can use or (ns1|ns2)
    namespaceKeepRegex: (default)
    # port names to scrape from - can use or .*metrics|otherportname
    portKeepRegex: .*metrics|web
    # metrics to keep
    metricKeepRegex: (.*)

Redeploy the Observe Agent

Run the following command to redeploy the Observe Agent in the observe namespace with the prometheusScrape configuration.

helm upgrade --reuse-values observe-agent observe/agent -n observe --values prometheus-scrape.yaml

Restart the pods

Run the following commands to restart the pods with the updated configuration.

kubectl rollout restart deployment -n observe
kubectl rollout restart daemonset -n observe

Run the following command to make sure the Observe Agent has been redeployed successfully.

kubectl get pods -o wide -n observe

For more examples, see Prometheus pod metrics scrape example in the Helm charts documentation.