Breaking changes when upgrading to version 2.0.0

Observe Agent v2.0.0 includes the following breaking changes.

All metrics are now exported as Prometheus metrics by default

Observe Agent version 2.0.0 exports all metrics as Prometheus metrics by default. This is done by using the provided prometheusremotewrite/observe exporter instead of the otlphttp/observe exporter. If you want to export metrics as OTel metrics (for example, to use the new observe-agent version with an existing Datastream that's currently receiving OTel Metrics), you can add the following override to your observe-agent config file:

otel_config_overrides:
  service:
    pipelines:
      metrics/forward:
        receivers: [otlp]
        processors: [resourcedetection, resourcedetection/cloud]
        exporters: [otlphttp/observe]

      # If host_monitoring is enabled
      metrics/agent-filestats:
        receivers: [filestats/agent]
        processors: [resourcedetection, resourcedetection/cloud]
        exporters: [otlphttp/observe]

      # If host_monitoring::metrics or host_monitoring::metrics::host is enabled
      metrics/host_monitoring_host:
        receivers: [hostmetrics/host-monitoring-host]
        processors: [memory_limiter, resourcedetection, resourcedetection/cloud, batch]
        exporters: [otlphttp/observe]

      # If host_monitoring::metrics::process is enabled
      metrics/host_monitoring_process:
        receivers: [hostmetrics/host-monitoring-process]
        processors: [memory_limiter, resourcedetection, resourcedetection/cloud, batch]
        exporters: [otlphttp/observe]

      # If self_monitoring is enabled
      metrics/agent-internal:
        receivers: [prometheus/agent, count]
        processors: [memory_limiter, transform/truncate, resourcedetection, resourcedetection/cloud, batch]
        exporters: [otlphttp/observe]

This change improves the performance of the Observe metric monitoring. All previously collected metrics will still be collected going forward, but the names may be changed as follows:

Metric name normalization

Starting in v2.0.0, the Observe Agent exports all metrics using the Prometheus Remote Write exporter by default. Prometheus metric naming conventions require that metric names contain only ASCII letters, digits, and underscores. As a result, characters such as dot (.) and hyphen (-) in OTel metric names are automatically converted to underscore (_).

For example:

OTel metric name	Prometheus metric name
`http.server.request.duration.bucket`	`http_server_request_duration_bucket`
`rpc.server.duration`	`rpc_server_duration`
`process.runtime.go.mem.heap_alloc`	`process_runtime_go_mem_heap_alloc`

This normalization also applies to metrics generated by OTel auto-instrumentation. If you are migrating from a stock OpenTelemetry collector to the Observe Agent, existing dashboards, monitors, or OPAL queries that reference the original OTel-format metric names must be updated.

To preserve original OTEL metric names, configure the Observe Agent to export metrics using the OTLP exporter instead of Prometheus Remote Write. Add the following override to your observe-agent.yaml config file:

otel_config_overrides:
  service:
    pipelines:
      metrics/forward:
        receivers: [otlp]
        processors: [resourcedetection, resourcedetection/cloud]
        exporters: [otlphttp/observe]

📘
Note
If your Observe instance already has Datastreams configured to receive OTEL metrics, switching to OTLP export maintains compatibility with those Datastreams. If you are setting up a new deployment, Prometheus export (the default) is recommended for better performance.

Suffixed with Total

Any metric that is a cumulative (monotonic) counter is suffixed with _total. For example:

OTel metric name	Prometheus metric name
`http_client_request_size`	`http_client_request_size_total`

Suffixed with Unit

Any metric with a unit defined in the metadata has the unit added as a suffix to the name. For example:

OTel metric name	Prometheus metric name
`file.mtime`	`file_mtime_seconds`

Combinations of the above

Combinations of the above transformations may be applied to a metric. For example:

OTel metric name	Prometheus metric name
`process.cpu.time`	`process_cpu_time_seconds_total`

For more information, see the OTel Prometheus Remote Write Exporter design doc in the OTel documentation on GitHub.

CLI Changes renaming the config flags

The flag to specify the observe agent config has been changed from --config to --observe-config. The flag to specify OTel config has been changed from --otel-config to --config. This makes our start command fully compatible with the base otelcol command CLI. If you were using the default file location for the observe config and not specifying OTel config overrides via command line flag, no change is required.

Config file env var changes

The following changes are happening to the environment variables that the observe-agent automatically populates:

OBSERVE_ENDPOINT has been removed and replaced with:
- OBSERVE_COLLECTOR_URL (ex: https://198826828955.collect.observe-eng.com/)
- OBSERVE_OTEL_ENDPOINT (ex: https://198826828955.collect.observe-eng.com/v2/otel)
- OBSERVE_PROMETHEUS_ENDPOINT (ex: https://198826828955.collect.observe-eng.com/v1/prometheus)
OBSERVE_TOKEN has been renamed to OBSERVE_AUTHORIZATION_HEADER (this always had the "Bearer " prefix)

Updated 3 days ago