Collecting and Using Metrics¶

A metric is any value you can measure over time. It can be blocks used on a filesystem, the number of nodes in a cluster, or a temperature reading. Observe ingests this time series data into event datasets. Any event dataset with a numerical value column can be interpreted as a metric dataset in Observe, in Metrics Explorer, a dashboard, a worksheet, or a monitor. Each point in a time series represents a measurement from a single resource, with its name, value, and tags.

A metric has three major metadata attributes:

Metric Type: The metric type is used by Observe to interpret the reported values. It is critical to set the Metric Type correctly. For instance, a cumulativeCounter’s normal reset could look like a service failure if that metric is mistakenly typed as a gauge or delta.
Measurement Type: The metric’s measurement type describes the data type reported in each point. Currently Observe only supports the float64 data type.
Unit: The metrics’ unit describes the unit of the measurements, such as kB or kB/s. Observe uses these attributes to display metrics properly in charts.

Types of Metrics Collected by Observe¶

Observe requires that a type is set on metrics data. If a metric type is not explicitly set in metadata, Observe will attempt to auto-assigned a metric type.

Metric type is used to drive default visualization behaviors in metric expression builders (such as Metrics Explorer or a Threshold Metrics Monitor. The metric type also affects the behavior of alignment function that are sensitive to metric types, such as rate and delta.

Note

The Metric type drives default visualization behavior. If the default is not the desired behavior, you can easily switch the alignment functions in the expression builder: for instance, change avg() to rate() or change rate() to delta().

You can view the type of an ingested metrics dataset by hovering over the name in Metrics Explorer. It is useful to confirm that the type is set correctly, as false positive or negative results could occur from incorrectly typed metrics.

Observe supports the following Metrics Types:

gauge

The gauge metric type represents a snapshot of the state at the time of collection. Use a gauge metric type to measure data that is reported continuously, such as available disk space or memory used. The value of a gauge metric can go up or down or stay the same.

Given a series of memory measurements, such as pod_memory_working_set_bytes from Kubernetes, an example data series might look like 31, 30.5, 31, 31, 31.5.

In a gauge metric of pod_memory_working_set_bytes, all reported values are retained. Query time parameters are then used to calculate and display what the user requires.

cumulativeCounter

The cumulativeCounter metric type represents a cumulative measurement that is measured since the creation of the time-series. The value can only increase (monotonically), or reset to zero when restarted.

Given a series of CPU measurements, such as pod_cpu_usage_seconds_total as used in Kubernetes, the CPU measurements accumulated since the pod started. An example data series might look like 36.33, 36.61, 40.96, 652.20, 1097.59.

In a cumulativeCounter metric of pod_cpu_usage_seconds_total, all of the values are reported (36.33, 36.61, 40.96, 652.20, 1097.59).

delta

The delta metric type represents the value change since the last collected report. For example, if CPU utilization increased from 50% to 80% during a metric reporting interval, the delta metric reported for it would be 30%. The value of a delta metric can go up or down or stay the same.

Given a series of memory measurements, such as pod_memory_working_set_bytes from Kubernetes, an example data series might look like 31, 30.5, 31, 31, 31.5.

In a delta metric of pod_memory_working_set_bytes, the first measurement is reported, followed by the delta at the end of the period (31, 0.5).

Metric types do not affect how metric data is collected or stored, but they are used at query time. A metric is tabulated to make a chart or test a monitored condition. This requires a time resolution, which Observe dynamically determines based on your query window size. For instance, a query window of four hours would have a resolution of one minute; while a query window of one day will have a resolution of five minutes.

The chart or table’s behavior is also established by use of an OPAL alignment function. This is a mathematical operation used on the values in each time resolution window to determine which value to show in the table or chart. For instance, a metric might have points at every thirty seconds, while our chart has a five minute resolution. This means each five minute window has ten measurements to evaluate. The avg() function will show the average of those ten values. See alignment functions for more options.

Metric types affect the run time behavior of some operations that use alignment functions.

delta: calculates the value difference of the argument in each time bin for each group. delta may produce negative values when the argument decreases over time.
- for gauge metrics, delta retains the default behavior. Negatives may be produced when the value is decreasing.
- for cumulativeCounter metrics, delta will assume the values to be monotonically increasing, and treats decreasing values as counter resets. Negatives will not be produced for cumulativeCounter type metrics.
- for delta metrics, delta will sum up the values in the time window to return the total sum. Negatives may be produced for negative input values.
delta_monotonic: calculates the amount of difference in a column in each time bin for each group. delta_monotonic by default assumes the argument to be monotonically increasing, and treats decreasing values as counter resets.
- for gauge and cumulativeCounter metrics, delta_monotonic retains the default behavior of assuming monotonic increases.
- for delta metrics, delta_monotonic will sum up the values in the time window to return the total sum. Negatives may be produced for negative input values.
deriv: calculates the average per-second derivative of the metric during each time frame
- for gauge metrics, deriv computes the value change over the time frame, allowing negative changes, and then divides the value change with time frame size.
- for cumulativeCounter metrics, deriv computes the value change over the time frame, treating value decreases as counter resets to prevent negative changes, and then divides it by the time frame size.
- for delta metrics, deriv computes the value change by summing up the deltas, and then divides it by the time frame size.
rate: calculates the average per-second rate of increase of the metric during each time frame
- for gauge and cumulativeCounter metrics, rate computes the value increase over the time frame, assuming monotonic increase in the value and treating decreasing values as counter resets, and then divides the value increase by the time frame size.
- for delta metrics, rate computes the value increase by sums of the deltas, and then divides it by the time frame size.

Understanding Alignment¶

An alignment is a computation that divides time series into regular time intervals, and aggregate the time series data into one single point for each time interval. Conceptually, alignment operations look like this:

../../_images/metrics-alignment-concept.png

Figure 1 - Metrics aligned by the avg function

A more advanced version of alignment is sliding window alignment, also known as hopping window alignment. It is used for computations like “rolling average”. In this query, for each time-series, we generate one output point for each 5 minute window, by computing the average of the input points in the prior 10 minutes. For example:

../../_images/metrics-alignment-sliding-window.png

Figure 2 - Metrics aligned by the avg function with a frame

A tag-dimension aggregation is an operation that aggregates multiple aligned time series into one. Tag-dimension aggregation takes in regularly aligned time series, and keep the timestamps unchanged. Conceptually, tag-dimension aggregation looks like this:

../../_images/metrics-alignment-aggregation.png

Figure 3 - Metrics aligned by the sum function

Additional Metrics Metadata¶

In addition to the metric type, the following metadata items can be set:

Measurement type - describes the type of data reported in each data point. Currently, Observe only supports the float64 type.
Unit - describes the unit of measurement such as kb or kB/s.
Description - detailed information about the metric.
Tags - For a time series, tags better describe and differentiate the measurements. You can use them to identify individual times series during metric computations such as align and aggregate.

Figure 4 - Pod memory usage metrics on the Pod dashboard

A Metrics Dataset contains metric data recognized by Observe. Observe optimizes the metric dataset for scalable ingestion and queries, supporting a large number of metrics. A metric dataset has the following properties:

Each row in the dataset table describes one point in a time series.
A metric dataset contains a string type metric value column named metric.
Contains a float64 metric value column named value.
Contains a valid_from column with the measurement time for each data point.
The metric interface OPAL language designates a dataset as a metric dataset.
All non-metric names, values, and non-valid_from columns contain metric tags.

A Metrics Dataset is always an Event Dataset and the data either inherited from an upstream Metrics Dataset or created using the OPAL interface verb.

interface "metric", metric:my_name, value:the_reading

Now that you understand the types of Metrics collected by Observe and the Observe apps, use the Metrics Explorer feature to easily view and model Metrics Datasets.

Note

Metrics use OPAL in a worksheet to transform the raw data, add metadata, and create relationships between datasets. If you are not familiar with OPAL, please see OPAL — Observe Processing and Analysis Language

A metric dataset contains one metric point per row - a single data point containing a timestamp, name, value, and zero or more tags. For example, the following table contains values for two metrics:

valid_from	metric	value	tags
00:00:00	disk_used_bytes	20000000	{“device”:”sda1”}
00:00:00	disk_total_bytes	50000000	{“device”:”sda1”}
00:01:00	disk_used_bytes	10000000	{“device”:”sda1”}
00:01:00	disk_total_bytes	50000000	{“device”:”sda1”}
00:02:00	disk_used_bytes	40000000	{“device”:”sda1”}
00:02:00	disk_total_bytes	50000000	{“device”:”sda1”}

Some systems generate this by default, or you can shape other data into the correct form with OPAL.

Note

Metric values must be float64. If you need to convert from another type, see the float64() function.

Metric Interfaces¶

In order to see your metric dataset in Metrics Explorer, it needs to be given a metrics interface. This is done with an OPAL verb.

The interface verb maps fields to a metric interface, so subsequent operations know which fields contain the metric names and values. This metadata-only operation prepares a dataset for use as metrics.

Example:

interface "metric"

The data you see doesn’t change, but registering or implementing the metric interface establishes the following conditions:

Each row represents one point in a time series
A field named metric contains the metric names
A field named value contains the metric values

If the metric names and values are already in fields called metric and value, interface discovers them automatically.

Modeling Metadata into Metric Datasets¶

Every metric in Observe can have a type, unit, and description associated with it. Defining a metric using set_metric, lets you set values for all three of these.

Observe uses the metadata of a metric to visualize metrics appropriately. For example, rate rollup is chosen for metrics of the type cumulativeCounter, and the Y-axis of metric visualizations selects the unit based on the unit of the chosen metric.

Some metric providers, such as Google Cloud Platform (GCP), AWS, and OTEL, provide metric metadata with every metric point. The metric dataset looks similar to the following table:

Metric	Value	Tag	Type	Unit	Description	Tags

Other providers, such as Prometheus, send metadata separately from the metric and do not contain any information about the tags.

A sample observation of a metric point sent by Prometheus with the OBSERVATION_KIND set to prometheus:

{

  "__name__": "collector_request_duration_seconds_bucket",  // <- metric name
  "clusterUid": "23e17bad-48da-427e-9585-ead2231bbcae",     //---+
  "container": "collector",                                 //   |
  "endpoint": "/chronicle/eu-west1/dmgprotect",             //   |
  "instance": "172.20.68.149:11000",                        //   |
  "job": "integrations/kubernetes/pods",                    //   |<- metric tags
  "le": "0.05",                                             //   |
  "namespace": "prod-eu-1",                                 //   |
  "node": "ip-172-20-83-237.eu-central-1.compute.internal", //   |
  "pod": "collector-6dbb6787f9-5bqdn",                      //   |
  "status": "202"                                           //---+
 
}

A sample observation of metric metadata sent by Prometheus with the OBSERVATION_KIND set to prom_md:

{
  "help": "HTTP request latency in seconds",                    
  "metric_family_name": "collector_request_duration_seconds",  // <- metric name
  "type": "HISTOGRAM",
  "unit": ""
}

Modeling Metadata¶

Observe provides two ways of modeling metric metadata:

GCP, AWS, and OTEL metrics where the metric includes the metadata
Prometheus where the metrics and metadata are sent separately.

For metrics with included metadata, use the following OPAL:

interface "metric", metric:metric, value:value, metricType:type, metricUnit:unit, metricDescription:description

metricType, metricUnit, and metricDescription are optional parameters for the metric interface. When defined, Observe can find the metric metadata in the columns for these fields.

To model metadata sent separately from metrics, store the metrics and metadata in separate datasets. Datasets containing metrics should implement metric interface and the dataset containing metadata should implement the metric_metadata interface.

Metric	Type	Unit	Description

Define the metric_metadata interface using the following OPAL:

interface "metric_metadata", metric:metric, metricType:type, metricUnit:unit, metricDescription:description