Tutorial: Getting Started with Observe and Prometheus¶
Prometheus is a popular open source tool for capturing observability data, and you can quickly get started with Observe by configuring Prometheus to push metrics to Observe. This guide walks you through the process of installing Prometheus, pushing metrics to Observe, and visualizing your metrics in Observe.
By following this guide, you perform the following tasks:
Install the Observe Prometheus Metrics App.
Install Prometheus and Node Exporter on a virtual machine.
Use the Prometheus
remote_writefeature to push metrics to Observe.Explore your metrics in Observe and start creating dashboards and monitors.
Note
If you want to collect Prometheus data to monitor a Kubernetes environment, consider using the Observe Kubernetes App with Metric Discovery instead of the Prometheus Metrics App.
Note
The examples in this guide target installation with an Ubuntu Linux VM, but you can follow similar steps for your OS or distribution of choice.
Installing the Observe Prometheus Metrics App¶
Log into your Observe instance, and locate the Apps page using the Apps icon on the Navigation bar.
From there, select Install and manage apps.
Search for Prometheus Metrics using the search bar and select the app.
Select the most recent version and then click Install.
Select Advanced and then select the Datastream to send metrics. You can use the default value. Then select Continue.
After the installation completes, under Connections > Data connection requirements > Create Prometheus Token, select Get Started.
Create a name and description for your Prometheus Token. Click Continue.
Copy the token and save it for later use.
Now you have installed the Observe Prometheus Metrics app and ready to receive Prometheus Metrics.
Download and Run Node Exporter¶
You can find and download the versions of Node Exporter appropriate for your system from the Prometheus download page. Uncompress it to your working directory and run the binary:
Ubuntu Linux Example
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.0/node_exporter-1.6.0.linux-amd64.tar.gz
tar xvfz node_exporter-1.6.0.linux-amd64.tar.gz
cd node_exporter-1.6.0.linux-amd64
setsid ./node_exporter --collector.systemd --collector.processes >/dev/null 2>&1 < /dev/null &
Confirm the connection status:
curl http://localhost:9100/metrics
Download and Install Prometheus¶
You can find and download the version of Prometheus appropriate for your system from the Prometheus download page. Return to your original working directory, and uncompress it.
Ubuntu Linux Example
# downlaod prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xvfz prometheus-2.45.0.linux-amd64.tar.gz
Create a user and a series of directories for Prometheus to use for collection.
Ubuntu Linux Example
Run this script to create a prometheus user and download and install prometheus.
bash <(wget -nv -O - https://gist.githubusercontent.com/slobsv/a38adbe50e3cacab4407205370d49581/raw/03189517269e4a7d13222d36f7a439e4fcb723e4/prometheus_install_ubuntu.sh)
Running Prometheus as a Service¶
Configure Prometheus to run as a service by writing a configuration file to /etc/systemd/system/prometheus.service with the following content:
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
Run your Prometheus service using the following:
sudo systemctl daemon-reload
sudo systemctl start prometheus
Confirm the status of your Prometheus configuration:
sudo systemctl status prometheus
sudo curl http://localhost:9090/graph
Pushing Metrics to Observe¶
Prometheus uses a feature called remote_write to push metrics to Observe. Configure your Prometheus instance to capture and push metrics to Observe by writing a Prometheus configuration file to /etc/prometheus/prometheus.yml with the following content, replacing the ${OBSERVE_*} variables with your values:
global:
scrape_interval: 15s
evaluation_interval: 15s
###Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
scrape_configs:
- job_name: node
static_configs:
- targets: ['localhost:9100']
remote_write:
- url: https://${OBSERVE_CUSTOMER_ID}.collect.${OBSERVE_DOMAIN}/v1/prometheus
bearer_token: ${OBSERVE_INGEST_TOKEN}
remote_timeout: "30s"
queue_config:
min_backoff: "1s"
max_backoff: "30s"
max_shards: 20
max_samples_per_send: 5000
capacity: 15000
retry_on_http_429: true
The variable,
${OBSERVE_DOMAIN}, typically has the valuecollect.observeinc.com.You can find your
${OBSERVE_CUSTOMER_ID}in the subdomain of the URL you use to access Observe. For example in the URL123456789012.observeinc.com, the Customer ID is123456789012.Some Observe instances may optionally use a name instead of Customer ID; if this is the case for your instance, contact your Observe Data Engineer to discuss implementation. A stem name will work as is, but a DNS redirect name may require client configuration.
The Prometheus token you saved when you installed the Prometheus app replaces the
${OBSERVE_INGEST_TOKEN}variable.Restart your Prometheus Metrics
sudo systemctl restart prometheus
Alternatively, you can use the Grafana agent to collect data from a Prometheus endpoint and push metrics to Observe using the Grafana agent remote_write feature.
Install the Grafana Agent.
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana-agent
2. Write a configuration for the Grafana Agent to /etc/grafana-agent.yaml and replace the ${OBSERVE_*} variables with your values:
Ubuntu Linux Example
---
server:
log_level: info
metrics:
wal_directory: /tmp/grafana-agent-wal
global:
scrape_interval: 15s
configs:
- name: integrations
remote_write:
- url: https://${OBSERVE_CUSTOMER_ID}.collect.${OBSERVE_DOMAIN}/v1/prometheus
authorization:
credentials: ${OBSERVE_INGEST_TOKEN}
scrape_configs:
- job_name: "integrations"
static_configs:
- targets: ["localhost:9100"]
The variable,
${OBSERVE_DOMAIN}, typically has the valuecollect.observeinc.com.You can find your
${OBSERVE_CUSTOMER_ID}in the subdomain of the URL you use to access Observe. In the example of123456789012.observeinc.comthe Customer ID is123456789012.The Prometheus token you saved when you installed the Prometheus app replaces the
${OBSERVE_INGEST_TOKEN}variable.
3. Write a systemctl default file to /etc/default/grafana-agent to set the Grafana Agent http and grcp ports to a different server than the Prometheus server that you scrape in the CUSTOM_ARGS section. Grafana uses ports 9090 and 9091 by default:
## Path:
## Description: Grafana Agent monitoring agent settings
## Type: string
## Default: ""
## ServiceRestart: grafana-agent
#
# Command line options for grafana-agent
#
# The configuration file holding the agent config
CONFIG_FILE="/etc/grafana-agent.yaml"
# Any user defined arguments
CUSTOM_ARGS="-server.http.address=127.0.0.1:9999 -server.grpc.address=127.0.0.1:9998"
# Restart on system upgrade. Default to true
RESTART_ON_UPGRADE=true
4. Start the Grafana Agent with sudo systemctl start grafana-agent.
Capturing Custom Prometheus Metrics¶
Capturing custom Prometheus metrics involves configuring Prometheus or Grafana to scrape the Prometheus target that contains those custom metrics and and remote_write them to Observe. To set up an example, follow these steps:
Expose Custom Metrics over a Prometheus endpoint. Here we’ll provide examples for both Python and Go:
1. Install the prometheus-client library with pip install prometheus-client.
2. Create a simple Python script to generate metrics:
import time
from prometheus_client import start_http_server, Counter, Gauge, Summary, Histogram
prometheus_scrape_port = 9092
c = Counter('mytest_counter', 'testing a counter type metric', unit='seconds')
g = Gauge('mytest_gauge', 'testing a gauge type metric', unit='seconds')
s = Summary('mtest_summary', 'testing a summary type metric', unit='seconds')
h = Histogram('mytest_histogram', 'testing a histogram type metrichistogram', unit='seconds')
def gen_metrics():
c.inc()
g.set(42)
s.observe(4.7)
h.observe(9.7)
time.sleep(1)
if __name__ == '__main__':
start_http_server(prometheus_scrape_port)
while True:
gen_metrics()
3. Run the script in the background python3 [mytest.py](http://mytest.py) &.
4. Make note of which prometheus_scrape_port value you used to expose your metrics.
Create a go package directory, e.g,
./mytest/
2. Add a go.mod.
module mytest
go 1.20
3. Add a mytest.go.
package main
import (
"log"
"math/rand"
"net/http"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
type metrics struct {
testGauge prometheus.Gauge
testGaugeTotal prometheus.Gauge
testGaugeBucket prometheus.Gauge
testCounter *prometheus.CounterVec
testHistogram prometheus.Histogram
testSummary prometheus.Summary
}
func NewMetrics(reg prometheus.Registerer) *metrics {
m := &metrics{
testCounter: prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "mytest_go_counter",
Help: "testing a counter type metric in golang",
},
[]string{"device"},
),
testHistogram: prometheus.NewHistogram(
prometheus.HistogramOpts{
Name: "mytest_go_histogram",
Help: "testing a histogram type metrichistogram in golang",
},
),
testGauge: prometheus.NewGauge(
prometheus.GaugeOpts{
Name: "mytest_go_gauge",
Help: "testing a gauge type metric",
},
),
testSummary: prometheus.NewSummary(
prometheus.SummaryOpts{
Name: "mytest_go_summary",
Help: "testing a summary type metric",
},
),
}
reg.MustRegister(m.testCounter)
reg.MustRegister(m.testHistogram)
reg.MustRegister(m.testGauge)
reg.MustRegister(m.testSummary)
return m
}
func main() {
// Create a non-global registry.
reg := prometheus.NewRegistry()
prometheus_scrape_port:= ":9093"
// Create new metrics and register them using the custom registry.
m := NewMetrics(reg)
// Set values for the new created metrics.
m.testGauge.Set(65.3)
m.testHistogram.Observe(rand.Float64() * 10)
m.testSummary.Observe(rand.Float64() * 10)
m.testCounter.With(prometheus.Labels{"device": "/dev/sda"}).Inc()
m.testCounter.With(prometheus.Labels{"device": "/dev/sda"}).Inc()
go func() {
for {
m.testGauge.Set(rand.Float64()*60 - 5)
m.testGauge.Set(rand.Float64()*60 - 5)
m.testHistogram.Observe(rand.Float64() * 10)
m.testSummary.Observe(rand.Float64() * 10)
m.testCounter.With(prometheus.Labels{"device": "/dev/sda"}).Add(rand.Float64() * 5)
m.testCounter.With(prometheus.Labels{"device": "/dev/sdb"}).Add(rand.Float64() * 5)
time.Sleep(time.Second)
}
}()
// Expose metrics and custom registry via an HTTP server
// using the HandleFor function. "/metrics" is the usual endpoint for that.
http.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{Registry: reg}))
log.Fatal(http.ListenAndServe(prometheus_scrape_port, nil))
}
4. Install the client_golang packages:
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promauto
go get github.com/prometheus/client_golang/prometheus/promhttp
5. Run your script in the background go run mytest.go &.
6. Make note of which prometheus_scrape_port value you used to expose your metrics.
Next, update your Prometheus or Grafana agent to scrape the new target you added on the prometheus_scrape_port port.
Update your
/etc/prometheus/prometheus.yamlfile with the value you assigned toprometheus_scrape_port:
...
scrape_configs:
- job_name: node
static_configs:
- targets: ['localhost:9100', 'localhost:9092']
...
2. Restart your prometheus sudo systemctl restart prometheus
1. Update your /etc/grafana-agent.yaml file’s contents using the value you assigned to prometheus_scrape_port:
...
scrape_configs:
- job_name: "integrations"
static_configs:
- targets: ["localhost:9100", "localhost:9092"]
...
2. Restart your Prometheus sudo systemctl restart grafana-agent.
You now have your Prometheus data routed to Observe and you can visualize your Prometheus Metrics in Observe in the Metrics Explorer and your custom dashboards.
Viewing your Metrics¶
Note
When you first start sending new metrics into Observe, there can be up to a 30 minute delay for the new metric to register in the Metric Explorer.
From the left navigation bar, click Metrics.
The left panel displays the list of metrics matching your search term, displayed as the metric dataset name. To find all Prometheus metrics, you can search for
Prometheus Metrics., which focuses on the metrics within the Prometheus Metrics dataset.For example, you can then search for and select the metric
Prometheus Metrics.node_cpu_seconds_total.In the Where and by fields you can add type
labels. to start filtering or grouping by specific label values.After creating a graph with important visualizations, select Actions at the top-right and either save your visualization to a dashboard or use it to start drafting a monitor for alerts.
Troubleshooting¶
If you followed all the steps above and you don’t see the metrics that you expect in the Observe Metric Explorer, you can use the following steps to troubleshoot:
Validate that Prometheus metrics reach your Observe account:
a. Go to Observe Account Settings and then click Datastreams.
b. Select the Datastream you created when you installed the Prometheus Metrics app.
c. Select the token that you created to push your Prometheus metrics to your Observe account, and then click Open dataset to view the data Observe receives with that token. If you want to go one step further, filter the OBSERVATION_KIND to prometheus.
2. If no data reaches your Observe account, validate that you don’t have a network issue that would stop data from reaching your Observe account. Follow this doc to use cURL to push a sample event to your Observe account over the HTTP Collection Endpoint.
3. If you can send data to your Observe account, but you do not see your Prometheus metrics, that suggests that either the Prometheus or the Grafana agent hit some issue in either scraping or pushing data. Use the following steps to resolve it:
a. Confirm the availability of your Prometheus metrics area at the designated endpoint: curl localhost:9100/api/v1/metrics.
b. Check your Prometheus logs using sudo journalctl -u prometheus.
c. Check your Grafana agent logs if you use a Grafana agent sudo journalctl -u grafana-agent.