Breaking changes when upgrading to version 2.0.0

Observe Agent v2.0.0 includes the following breaking changes.

All metrics are now exported as Prometheus metrics by default

Observe Agent version 2.0.0 exports all metrics as Prometheus metrics by default. This is done by using the provided prometheusremotewrite/observe exporter instead of the otlphttp/observe exporter. If you want to export metrics as OTel metrics (for example, to use the new observe-agent version with an existing Datastream that's currently receiving OTel Metrics), you can add the following override to your observe-agent config file:

otel_config_overrides:
  service:
    pipelines:
      metrics/forward:
        receivers: [otlp]
        processors: [resourcedetection, resourcedetection/cloud]
        exporters: [otlphttp/observe]

      # If host_monitoring is enabled
      metrics/agent-filestats:
        receivers: [filestats/agent]
        processors: [resourcedetection, resourcedetection/cloud]
        exporters: [otlphttp/observe]

      # If host_monitoring::metrics or host_monitoring::metrics::host is enabled
      metrics/host_monitoring_host:
        receivers: [hostmetrics/host-monitoring-host]
        processors: [memory_limiter, resourcedetection, resourcedetection/cloud, batch]
        exporters: [otlphttp/observe]

      # If host_monitoring::metrics::process is enabled
      metrics/host_monitoring_process:
        receivers: [hostmetrics/host-monitoring-process]
        processors: [memory_limiter, resourcedetection, resourcedetection/cloud, batch]
        exporters: [otlphttp/observe]

      # If self_monitoring is enabled
      metrics/agent-internal:
        receivers: [prometheus/agent, count]
        processors: [memory_limiter, transform/truncate, resourcedetection, resourcedetection/cloud, batch]
        exporters: [otlphttp/observe]

This change improves the performance of the Observe metric monitoring. All previously collected metrics will still be collected going forward, but the names may be changed as follows:

Periods to Underscores

All periods in metric names are replaced with underscores. Ex: http_client_duration.bucket -> http_client_duration_bucket

Suffixed with Total

Any metric that is a cumulative counter (aka monotonic) will be suffixed with _total. Ex: http_client_request_size -> http_client_request_size_total

Suffixed with Unit

Any metric with a unit defined in the metadata will have the unit added as a suffix to the name. Ex: file.mtime -> file_mtime_seconds

Combinations of the above

Multiple of the above transformations may be applied to a metric. Ex: process.cpu.time -> process_cpu_time_seconds_total

For more information, see the OTel Prometheus Remote Write Exporter design doc.

CLI Changes renaming the config flags

The flag to specify the observe agent config has been changed from --config to --observe-config. The flag to specify OTel config has been changed from --otel-config to --config. This makes our start command fully compatible with the base otelcol command CLI. If you were using the default file location for the observe config and not specifying OTel config overrides via command line flag, no change is required.

Config file env var changes

The following changes are happening to the environment variables that the observe-agent automatically populates:

• OBSERVE_ENDPOINT has been removed and replaced with:
  • OBSERVE_COLLECTOR_URL (ex: https://198826828955.collect.observe-eng.com/)
  • OBSERVE_OTEL_ENDPOINT (ex: https://198826828955.collect.observe-eng.com/v2/otel)
  • OBSERVE_PROMETHEUS_ENDPOINT (ex: https://198826828955.collect.observe-eng.com/v1/prometheus)
• OBSERVE_TOKEN has been renamed to OBSERVE_AUTHORIZATION_HEADER (this always had the "Bearer " prefix)