APM runtime metrics

Observe APM can automatically plot runtime metrics and infrastructure metrics for your services. This provides a fast way to answer questions like: "how is garbage collection affecting my CPU usage and service response times?" or "are we seeing an increase of goroutines associated with the service erroring?"

To view runtime metrics for your services, perform the following steps:

  1. Click APM in the left navigation.
  2. Select a service.
  3. Click the metric you want to investigate, such as Infrastructure metrics or Go metrics in this example:

Requirements to enable runtime metrics

The following requirements must be met in order for metrics to be plotted in APM:

  1. The metrics must be sent to Observe in OTel format and be written to the Metrics/OpenTelemetry Dataset.
    • If you are running the Observe Agent on a host or containerized environment, make sure the forwarding.metrics.output_format: otel option is set in the values.yaml file. This value is set automatically if you install the Observe Agent from the Add Data portal.
    • If you are running the Observe Agent in Kubernetes, make sure the node.forwarder.metrics.outputFormat=otel option is set in your values.yaml file when installing or upgrading the Observe Agent Helm chart. This value is set automatically if you install the Observe Agent from the Add Data portal.
  2. The metrics must have temporality delta. This can be configured using the standard OTel environment variable OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta. For compatibility, the Observe Agent can also be configured to convert all metrics to delta temporality by setting forwarding.metrics.convert_cumulative_to_delta: true in the agent config or node.forwarder.metrics.convertCumulativeToDelta=true in your Helm chart values.yaml file.

To associate your infra metrics with your services and display them alongside runtime metrics in APM, see Associate infrastructure metrics with services.

Supported languages and runtimes

Select a language for more information.

Supported versions are v0.55.0 and later, but starting with v0.62.0 the environment variable OTEL_GO_X_DEPRECATED_RUNTIME_METRICS must be set to true.

APM Runtime metrics plot data for the following metric names:

  • process.runtime.go.cgo.calls
  • process.runtime.go.gc.count
  • process.runtime.go.gc.pause_ns
  • process.runtime.go.gc.pause_ns.count
  • process.runtime.go.gc.pause_ns.max
  • process.runtime.go.gc.pause_ns.min
  • process.runtime.go.gc.pause_ns.sum
  • process.runtime.go.gc.pause_total_ns
  • process.runtime.go.goroutines
  • process.runtime.go.mem.heap_alloc
  • process.runtime.go.mem.heap_idle
  • process.runtime.go.mem.heap_inuse
  • process.runtime.go.mem.heap_objects
  • process.runtime.go.mem.heap_released
  • process.runtime.go.mem.heap_sys
  • process.runtime.go.mem.live_objects
  • process.runtime.go.mem.lookups