Prometheus Autodiscovery

  1. Use the following annotations for your pod

If you want to expose Prometheus metrics from your pod, you have two options:

  • Set the port in the annotations. This method has the disadvantage that it can only surface one HTTP endpoint for collection.

     annotations:
   prometheus.io/port: "9999"

  • Expose the metrics endpoint through a port with a name ending with metrics. For example, use the following:

  • ports:
- containerPort: 9999
  name: metrics
- containerPort: 12345
  name: sidecar-metrics

You can use the following annotations to further influence the discovery process:

  • prometheus.io/scheme: set to http or https

  • prometheus.io/path: defaults to /metrics

To disable collecting metrics on a pod, use one of the following annotations:

  • prometheus.io/scrape: if set to false, the pod is ignored.

  • observeinc_com_scrape: 'false': used only by the Observe collection process.

  1. Create prometheus-scrape.yaml with the following configuration.

application:
  # use this option to scrape prometheus metrics from pods
  prometheusScrape:
    enabled: true
    interval: 60s
    # namespaces to exclude from scraping
    namespaceDropRegex: (.*istio.*|.*ingress.*|kube-system)
    # namespaces to explicity include for scraping - can use or (ns1|ns2)
    namespaceKeepRegex: (default)
    # port names to scrape from - can use or .*metrics|otherportname
    portKeepRegex: .*metrics|web
    # metrics to keep
    metricKeepRegex: (.*)

  1. Redeploy the Observe Agent

Run the following command to redeploy the Observe Agent in the observe namespace with the prometheusScrape configuration.

helm upgrade --reuse-values observe-agent observe/agent -n observe --values prometheus-scrape.yaml

  1. Restart pods

Run the following commands to restart the pods with the updated configuration.

kubectl rollout restart deployment -n observe
kubectl rollout restart daemonset -n observe

Run the following command to make sure the Observe Agent has been redeployed successfully.

kubectl get pods -o wide -n observe

For more examples, see Prometheus pod metrics scrape example