Tutorial: Getting Started with Observe and Prometheus

Prometheus is a popular open source tool for capturing observability data, and you can quickly get started with Observe by configuring Prometheus to push metrics to Observe. This guide walks you through the process of installing Prometheus, pushing metrics to Observe, and visualizing your metrics in Observe.

By following this guide, you perform the following tasks:

  1. Install the Observe Prometheus Metrics App.

  2. Install Prometheus and Node Exporter on a virtual machine.

  3. Use the Prometheus remote_write feature to push metrics to Observe.

  4. Explore your metrics in Observe and start creating dashboards and monitors.


If you want to collect Prometheus data to monitor a Kubernetes environment, consider using the Observe Kubernetes App with Metric Discovery instead of the Prometheus Metrics App.


The examples in this guide target installation with an Ubuntu Linux VM, but you can follow similar steps for your OS or distribution of choice.

Installing the Observe Prometheus Metrics App

  1. Log into your Observe instance, and locate the Apps page using the Apps icon on the Navigation bar.

  2. From there, select Install and manage apps.

  3. Search for Prometheus Metrics using the search bar and select the app.

  4. Select the most recent version and then click Install.

  5. Select Advanced and then select the Datastream to send metrics. You can use the default value. Then select Continue.

  6. After the installation completes, under Connections > Data connection requirements > Create Prometheus Token, select Get Started.

  7. Create a name and description for your Prometheus Token. Click Continue.

  8. Copy the token and save it for later use.

Now you have installed the Observe Prometheus Metrics app and ready to receive Prometheus Metrics.

Download and Run Node Exporter

You can find and download the versions of Node Exporter appropriate for your system from the Prometheus download page. Uncompress it to your working directory and run the binary:

Ubuntu Linux Example
    wget https://github.com/prometheus/node_exporter/releases/download/v1.6.0/node_exporter-1.6.0.linux-amd64.tar.gz
    tar xvfz node_exporter-1.6.0.linux-amd64.tar.gz
    cd node_exporter-1.6.0.linux-amd64
    setsid ./node_exporter --collector.systemd --collector.processes >/dev/null 2>&1 < /dev/null &

Confirm the connection status:

 curl http://localhost:9100/metrics

Download and Install Prometheus

You can find and download the version of Prometheus appropriate for your system from the Prometheus download page. Return to your original working directory, and uncompress it.

Ubuntu Linux Example
    # downlaod prometheus
    wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
    tar xvfz prometheus-2.45.0.linux-amd64.tar.gz

Create a user and a series of directories for Prometheus to use for collection.

Ubuntu Linux Example

Run this script to create a prometheus user and download and install prometheus.

    bash <(wget -nv -O - https://gist.githubusercontent.com/slobsv/a38adbe50e3cacab4407205370d49581/raw/03189517269e4a7d13222d36f7a439e4fcb723e4/prometheus_install_ubuntu.sh)

Running Prometheus as a Service

Configure Prometheus to run as a service by writing a configuration file to /etc/systemd/system/prometheus.service with the following content:


ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \


Run your Prometheus service using the following:

sudo systemctl daemon-reload
sudo systemctl start prometheus

Confirm the status of your Prometheus configuration:

sudo systemctl status prometheus
sudo curl http://localhost:9090/graph

Pushing Metrics to Observe

Prometheus uses a feature called remote_write to push metrics to Observe. Configure your Prometheus instance to capture and push metrics to Observe by writing a Prometheus configuration file to /etc/prometheus/prometheus.yml with the following content, replacing the ${OBSERVE_*} variables with your values:

  scrape_interval: 15s
  evaluation_interval: 15s

###Alertmanager configuration
    - static_configs:
        - targets:
          # - alertmanager:9093

- job_name: node
  - targets: ['localhost:9100']

- url: https://${OBSERVE_CUSTOMER_ID}.collect.${OBSERVE_DOMAIN}/v1/prometheus
  bearer_token: ${OBSERVE_INGEST_TOKEN}
  remote_timeout: "30s"
    min_backoff: "1s"
    max_backoff: "30s"
    max_shards: 20
    max_samples_per_send: 5000
    capacity: 15000
    retry_on_http_429: true
  • The variable, ${OBSERVE_DOMAIN}, typically has the value collect.observeinc.com.

  • You can find your ${OBSERVE_CUSTOMER_ID} in the subdomain of the URL you use to access Observe. For example in the URL 123456789012.observeinc.com, the Customer ID is 123456789012.

  • The Prometheus token you saved when you installed the Prometheus app replaces the ${OBSERVE_INGEST_TOKEN} variable.

  • Restart your Prometheus Metrics sudo systemctl restart prometheus

Alternatively, you can use the Grafana agent to collect data from a Prometheus endpoint and push metrics to Observe using the Grafana agent remote_write feature.

  1. Install the Grafana Agent.

   sudo mkdir -p /etc/apt/keyrings/
   wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
   echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
   sudo apt-get update
   sudo apt-get install -y grafana-agent

2. Write a configuration for the Grafana Agent to /etc/grafana-agent.yaml and replace the ${OBSERVE_*} variables with your values:

Ubuntu Linux Example

          log_level: info
          wal_directory: /tmp/grafana-agent-wal
            scrape_interval: 15s
            - name: integrations
                - url: https://${OBSERVE_CUSTOMER_ID}.collect.${OBSERVE_DOMAIN}/v1/prometheus
                    credentials: ${OBSERVE_INGEST_TOKEN}
                - job_name: "integrations"
                    - targets: ["localhost:9100"]   
  • The variable, ${OBSERVE_DOMAIN}, typically has the value collect.observeinc.com.

  • You can find your ${OBSERVE_CUSTOMER_ID} in the subdomain of the URL you use to access Observe. In the example of 123456789012.observeinc.com the Customer ID is 123456789012.

  • The Prometheus token you saved when you installed the Prometheus app replaces the ${OBSERVE_INGEST_TOKEN} variable.

3. Write a systemctl default file to /etc/default/grafana-agent to set the Grafana Agent http and grcp ports to a different server than the Prometheus server that you scrape in the CUSTOM_ARGS section. Grafana uses ports 9090 and 9091 by default:

        ## Path:
        ## Description: Grafana Agent monitoring agent settings
        ## Type:        string
        ## Default:     ""
        ## ServiceRestart: grafana-agent
        # Command line options for grafana-agent
        # The configuration file holding the agent config
        # Any user defined arguments
        CUSTOM_ARGS="-server.http.address= -server.grpc.address="
        # Restart on system upgrade. Default to true

4. Start the Grafana Agent with sudo systemctl start grafana-agent.

Capturing Custom Prometheus Metrics

Capturing custom Prometheus metrics involves configuring Prometheus or Grafana to scrape the Prometheus target that contains those custom metrics and and remote_write them to Observe. To set up an example, follow these steps:

Expose Custom Metrics over a Prometheus endpoint. Here we’ll provide examples for both Python and Go:

1. Install the prometheus-client library with pip install prometheus-client.

2. Create a simple Python script to generate metrics:

           import time
           from prometheus_client import start_http_server, Counter, Gauge, Summary, Histogram
           prometheus_scrape_port = 9092
           c = Counter('mytest_counter', 'testing a counter type metric', unit='seconds')
           g = Gauge('mytest_gauge', 'testing a gauge type metric', unit='seconds')
           s = Summary('mtest_summary', 'testing a summary type metric', unit='seconds')
           h = Histogram('mytest_histogram', 'testing a histogram type metrichistogram', unit='seconds')
           def gen_metrics():
           if __name__ == '__main__':
           	while True:

3. Run the script in the background python3 [mytest.py](http://mytest.py) &.

4. Make note of which prometheus_scrape_port value you used to expose your metrics.

  1. Create a go package directory, e.g, ./mytest/

2. Add a go.mod.

   module mytest
   go 1.20

3. Add a mytest.go.

           package main
           import (
           type metrics struct {
               testGauge       prometheus.Gauge
               testGaugeTotal  prometheus.Gauge
               testGaugeBucket prometheus.Gauge
               testCounter     *prometheus.CounterVec
               testHistogram   prometheus.Histogram
               testSummary     prometheus.Summary
           func NewMetrics(reg prometheus.Registerer) *metrics {
               m := &metrics{
                   testCounter: prometheus.NewCounterVec(
                           Name: "mytest_go_counter",
                           Help: "testing a counter type metric in golang",
                   testHistogram: prometheus.NewHistogram(
                           Name: "mytest_go_histogram",
                           Help: "testing a histogram type metrichistogram in golang",
                   testGauge: prometheus.NewGauge(
                           Name: "mytest_go_gauge",
                           Help: "testing a gauge type metric",
                   testSummary: prometheus.NewSummary(
                           Name: "mytest_go_summary",
                           Help: "testing a summary type metric",
               return m
           func main() {
               // Create a non-global registry.
               reg := prometheus.NewRegistry()
               prometheus_scrape_port:= ":9093"
               // Create new metrics and register them using the custom registry.
               m := NewMetrics(reg)
               // Set values for the new created metrics.
               m.testHistogram.Observe(rand.Float64() * 10)
               m.testSummary.Observe(rand.Float64() * 10)
               m.testCounter.With(prometheus.Labels{"device": "/dev/sda"}).Inc()
               m.testCounter.With(prometheus.Labels{"device": "/dev/sda"}).Inc()
               go func() {
                   for {
                       m.testGauge.Set(rand.Float64()*60 - 5)
                       m.testGauge.Set(rand.Float64()*60 - 5)
                       m.testHistogram.Observe(rand.Float64() * 10)
                       m.testSummary.Observe(rand.Float64() * 10)
                       m.testCounter.With(prometheus.Labels{"device": "/dev/sda"}).Add(rand.Float64() * 5)
                       m.testCounter.With(prometheus.Labels{"device": "/dev/sdb"}).Add(rand.Float64() * 5)
               // Expose metrics and custom registry via an HTTP server
               // using the HandleFor function. "/metrics" is the usual endpoint for that.
               http.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{Registry: reg}))
               log.Fatal(http.ListenAndServe(prometheus_scrape_port, nil))

4. Install the client_golang packages:

     go get github.com/prometheus/client_golang/prometheus
     go get github.com/prometheus/client_golang/prometheus/promauto
     go get github.com/prometheus/client_golang/prometheus/promhttp

5. Run your script in the background go run mytest.go &.

6. Make note of which prometheus_scrape_port value you used to expose your metrics.

Next, update your Prometheus or Grafana agent to scrape the new target you added on the prometheus_scrape_port port.

  1. Update your /etc/prometheus/prometheus.yaml file with the value you assigned to prometheus_scrape_port:

           - job_name: node
             - targets: ['localhost:9100', 'localhost:9092']

2. Restart your prometheus sudo systemctl restart prometheus

1. Update your /etc/grafana-agent.yaml file’s contents using the value you assigned to prometheus_scrape_port:

              - job_name: "integrations"
                  - targets: ["localhost:9100", "localhost:9092"]

2. Restart your Prometheus sudo systemctl restart grafana-agent.

You now have your Prometheus data routed to Observe and you can visualize your Prometheus Metrics in Observe in the Metrics Explorer and your custom dashboards.

Viewing your Metrics


When you first start sending new metrics into Observe, there can be up to a 30 minute delay for the new metric to register in the Metric Explorer.

  1. From the left navigation bar, click Metrics.

  2. The left panel displays the list of metrics matching your search term, displayed as the metric dataset name. To find all Prometheus metrics, you can search for Prometheus Metrics., which focuses on the metrics within the Promethus Metrics dataset.

  3. For example, you can then search for and select the metric Prometheus Metrics.node_cpu_seconds_total.

  4. In the Where and by fields you can add type labels. to start filtering or grouping by specific label values.

  5. After creating a graph with important visualizations, select Actions at the top-right and either save your visualization to a dashboard or use it to start drafting a monitor for alerts.


If you followed all the steps above and you don’t see the metrics that you expect in the Observe Metric Explorer, you can use the following steps to troubleshoot:

  1. Validate that Prometheus metrics reach your Observe account:

    a. Go to Observe Account Settings and then click Datastreams.

    b. Select the Datastream you created when you installed the Prometheus Metrics app.

    c. Select the token that you created to push your Prometheus metrics to your Observe account, and then click Open dataset to view the data Observe receives with that token. If you want to go one step further, filter the OBSERVATION_KIND to prometheus.

2. If no data reaches your Observe account, validate that you don’t have a network issue that would stop data from reaching your Observe account. Follow this doc to use cURL to push a sample event to your Observe account over the HTTP Collection Endpoint.

3. If you can send data to your Observe account, but you do not see your Prometheus metrics, that suggests that either the Prometheus or the Grafana agent hit some issue in either scraping or pushing data. Use the following steps to resolve it:

a. Confirm the availability of your Prometheus metrics area at the designated endpoint: curl localhost:9100/api/v1/metrics.

b. Check your Prometheus logs using sudo journalctl -u prometheus.

c. Check your Grafana agent logs if you use a Grafana agent sudo journalctl -u grafana-agent.