Prometheus autodiscovery
Perform the tasks on this page to configure Prometheus autodiscovery.
Expose Prometheus metrics from your pod
Use one of the following annotations for your pod to expose Prometheus metrics from your pod:
-
Set the port in the annotations. This method has the disadvantage that it can only surface one HTTP endpoint for collection.
annotations: prometheus.io/port: "9999" -
Expose the metrics endpoint through a port with a name ending with
metrics. For example, use the following:ports: - containerPort: 9999 name: metrics - containerPort: 12345 name: sidecar-metrics
You can use the following annotations to further influence the discovery process:
prometheus.io/scheme: set tohttporhttpsprometheus.io/path: defaults to/metrics
To disable collecting metrics on a pod, use one of the following annotations:
prometheus.io/scrape: if set tofalse, the pod is ignored.observeinc_com_scrape: 'false': used only by the Observe collection process.
Scrape Prometheus metrics from your pods
Create prometheus-scrape.yaml with the following configuration:
application:
# use this option to scrape prometheus metrics from pods
prometheusScrape:
enabled: true
interval: 60s
# namespaces to exclude from scraping
namespaceDropRegex: (.*istio.*|.*ingress.*|kube-system)
# namespaces to explicity include for scraping - can use or (ns1|ns2)
namespaceKeepRegex: (default)
# port names to scrape from - can use or .*metrics|otherportname
portKeepRegex: .*metrics|web
# metrics to keep
metricKeepRegex: (.*)Redeploy the Observe Agent
Run the following command to redeploy the Observe Agent in the observe namespace with the prometheusScrape configuration.
helm upgrade --reuse-values observe-agent observe/agent -n observe --values prometheus-scrape.yamlRestart the pods
Run the following commands to restart the pods with the updated configuration.
kubectl rollout restart deployment -n observe
kubectl rollout restart daemonset -n observeRun the following command to make sure the Observe Agent has been redeployed successfully.
kubectl get pods -o wide -n observeFor more examples, see Prometheus pod metrics scrape example in the Helm charts documentation.
Updated 10 days ago