The Observe Agent is built on the OpenTelemetry Collector. When a newer Collector version changes how a configuration behaves or what data it sends, this page lists the change, who it affects, and how to migrate.

How you're notified

• At startup. From at least one release before a change ships, the agent logs one warn message for each component in your configuration that the change affects. After the change ships, the message says what changed, until two releases later.
• In observe-agent config validate. The same messages appear as ⚠️ lines before the validation result.
• In the release notes for the Observe Agent and the Helm chart. On Kubernetes, the messages appear in the agent pod logs.

The messages fire only for configuration you add, such as otel_config_overrides, Helm values or extra --config files. The configuration the agent ships with never triggers them.

Observe Agent v2.20.0

The changes below take effect in Observe Agent v2.20.0 (OpenTelemetry Collector v0.164.1). Starting in v2.19.2, the agent warns at startup when your configuration is affected so you can migrate before upgrading. Unless a section says to act when you upgrade, you can make the change before you upgrade and keep it after you upgrade.

Tail sampling: invert_match policies

What will change (Collector v0.152.0): a string_attribute, numeric_attribute or boolean_attribute policy with invert_match: true will return an ordinary "not sampled" decision. It will no longer override other policies that sample the same trace, so traces you meant to exclude can be kept. A not policy around an inverted policy will negate it.

You're affected if a tail_sampling processor has a top-level inverted policy next to any other top-level policy, or a top-level not policy that wraps an inverted policy. A processor whose only policy is inverted is not affected. If you already run the agent with --feature-gates=processor.tailsamplingprocessor.disableinvertdecisions, you already have the new behavior; remove that flag when you upgrade, because the updated Collector no longer has the gate.

What to do: replace each inverted policy with a drop policy.

Before:

otel_config_overrides:
  processors:
    tail_sampling:
      policies:
        - name: drop-k8s-probes
          type: string_attribute
          string_attribute:
            key: http.user_agent
            values: [^kube-probe, ^kube-resource-report]
            enabled_regex_matching: true
            invert_match: true
        - name: keep-error-traces
          type: status_code
          status_code:
            status_codes: [ERROR]

After:

otel_config_overrides:
  processors:
    tail_sampling:
      policies:
        - name: drop-k8s-probes
          type: drop
          drop:
            drop_sub_policy:
              - name: k8s-probes
                type: string_attribute
                string_attribute:
                  key: http.user_agent
                  values: [^kube-probe, ^kube-resource-report]
                  enabled_regex_matching: true
        - name: keep-error-traces
          type: status_code
          status_code:
            status_codes: [ERROR]

httpcheck receiver: timing metrics in nanoseconds

What will change (Collector v0.153.0): httpcheck.dns.lookup.duration, httpcheck.client.connection.duration, httpcheck.tls.handshake.duration, httpcheck.client.request.duration and httpcheck.response.duration will report nanoseconds instead of milliseconds, so their values will be 1,000,000 times larger.

You're affected if your configuration has an http_check receiver, or one using its old name httpcheck.

What to do: when you upgrade, update dashboards and monitors that use these metrics, for example by dividing by 1,000,000 or raising thresholds.

mongodb receiver

What will change:

• The receiver will no longer connect to MongoDB 4.0 or 4.2 servers, so it will stop collecting metrics from them (Collector v0.155.0). The MongoDB Go driver it uses dropped support for these versions, which are past MongoDB's end of life.
• Metric units will follow the Unified Code for Units of Measure (UCUM) (Collector v0.157.0).
• mongodb.operation.latency.time will become a cumulative sum instead of a gauge (Collector v0.164.0).

You're affected if your configuration has a mongodb receiver.

What to do:

• When you upgrade, update dashboards and monitors that use these metrics.
• If the receiver monitors MongoDB 4.0 or 4.2 servers, keep the agents that monitor them on Observe Agent v2.19.x to keep collecting their metrics.

sqlserver receiver

What will change:

• Metric units will follow the Unified Code for Units of Measure (UCUM), and sqlserver.lock.timeout.rate, if enabled, will report one data point per sqlserver.lock.timeout.type (Collector v0.157.0).
• server.address and server.port will become resource attributes enabled by default, and will be removed from db.server.top_query log attributes (Collector v0.161.0).
• On direct database connections, counters from the Databases object will carry sqlserver.database.name and the unlabeled _Total series will be dropped, so sqlserver.database.backup_or_restore.rate will split per database (Collector v0.163.0).

You're affected if your configuration has a sqlserver receiver.

What to do: when you upgrade, update dashboards and monitors that use these metrics and attributes.

postgresql receiver: locks from every database

What will change (Collector v0.158.0): postgresql.database.locks will be collected from every configured database instead of only the postgres database.

You're affected if your configuration has a postgresql receiver.

What to do: when you upgrade, update dashboards and monitors that use this metric.

hostmetrics receiver: cpu attribute

What will change (Collector v0.157.0): the CPU scraper will report system.cpu.time and system.cpu.utilization without the cpu attribute unless you list it, so per-CPU series will be combined into one.

You're affected if you define your own hostmetrics (or host_metrics) receiver whose CPU scraper has either metric enabled without cpu in its attributes. The agent's built-in host monitoring already lists it.

What to do: list the attributes on system.cpu.time, and on system.cpu.utilization if you enable it.

otel_config_overrides:
  receivers:
    hostmetrics/custom:
      scrapers:
        cpu:
          metrics:
            system.cpu.time:
              attributes: [cpu, state]

prometheus exporter: metric name suffixes

What will change (Collector v0.154.0): add_metric_suffixes will be ignored, and type and unit suffixes such as _total will be added to metric names.

You're affected if a prometheus exporter sets add_metric_suffixes: false without translation_strategy.

What to do: replace add_metric_suffixes: false with translation_strategy: UnderscoreEscapingWithoutSuffixes.

otel_config_overrides:
  exporters:
    prometheus:
      endpoint: 0.0.0.0:8889
      translation_strategy: UnderscoreEscapingWithoutSuffixes

jmx receiver: removal

What will change (Collector v0.157.0): the jmx receiver will be removed. It runs a Java program as a subprocess of the agent, so it needs a Java runtime next to the agent. The agent has logged a deprecation warning for it since v2.15.0.

You're affected if your configuration has a jmx receiver. A configuration that still has one won't load after you upgrade.

What to do: remove the jmx receiver, and collect JMX metrics with one of these OpenTelemetry programs instead. Both send metrics to the agent over the OpenTelemetry Protocol (OTLP), the same way as application telemetry.

• The OpenTelemetry Java agent, for JVMs you can restart with -javaagent. It reads MBeans from inside the JVM, so you don't need a remote JMX port or another process. It can also send traces; see Send Java application data to Observe.
• The OpenTelemetry JMX Scraper, for JVMs you can't restart with an agent, such as a Kafka broker someone else runs. It connects to the JVM's remote JMX port. It's the program the jmx receiver runs as its subprocess when you use the JMX Scraper jar.

For example, this receiver:

otel_config_overrides:
  receivers:
    jmx:
      jar_path: /opt/opentelemetry-jmx-scraper.jar
      endpoint: service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi
      target_system: tomcat,jvm
      collection_interval: 60s

becomes either the Java agent, added to the application's command line:

java -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.service.name=my-service \
  -Dotel.exporter.otlp.endpoint=http://localhost:4318 \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.jmx.metrics.experimental.included=tomcat.* \
  -jar app.jar

or the JMX Scraper, run as its own process next to the agent:

java -Dotel.jmx.service.url=service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi \
  -Dotel.jmx.target.system=tomcat,jvm \
  -Dotel.service.name=my-service \
  -Dotel.exporter.otlp.endpoint=http://localhost:4318 \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.metric.export.interval=60000 \
  -jar opentelemetry-jmx-scraper.jar

localhost:4318 is the agent's OTLP endpoint on the same host. On Kubernetes, use observe-agent-forwarder.observe.svc.cluster.local:4318.

Notes:

• Java agent metrics: the Java agent collects JMX metrics by default, but only predefined metric sets marked stable, which today are some Tomcat metrics. Every other target, including Kafka, Jetty and the rest of Tomcat, needs an opt-in by name pattern with otel.jmx.metrics.experimental.included, for example tomcat.*,kafka.*. Don't use otel.jmx.target.system with the Java agent; it's deprecated there and will be removed in Java agent 3.0. JVM metrics come from the agent's runtime telemetry rather than JMX.
• Predefined targets: activemq, camel, jetty, kafka-broker, kafka-connect, tomcat, wildfly, hadoop and cassandra. For other MBeans, write YAML rules and pass them with otel.jmx.config.
• Metric names: for these targets, the Java agent and JMX Scraper send the same metric names, units and attributes as the jmx receiver does with the JMX Scraper jar. If your receiver runs the older JMX Metric Gatherer jar, names can differ; with the JMX Scraper, set otel.jmx.target.source=legacy to keep names close to the Gatherer's.
• Downloads: the Java agent and the JMX Scraper.