Trace Tail Sampling

Trace tail sampling allows you to reduce the span data sent to Observe while having full control of which traces are dropped.

To enable trace tail sampling, configure the following in your values.yaml file:

# Enable the gateway deployment and trace sampling
gatewayDeployment:
  enabled: true
  traceSampling:
    enabled: true
    # † Uncomment and edit the following section to override the default
    # tail sampling policies.
    #
    # config:
    #   policies:
    #     # Send all spans with status_code error to Observe
    #     - name: retain-all-errors
    #       type: status_code
    #       status_code: {status_codes: [ERROR]}
    #     # Send 10% of all other spans to Observe
    #     - name: sample-probabilistically-10-percent
    #       type: probabilistic
    #       probabilistic: {sampling_percentage: 10}

# Enable RED metrics generation in order to get accurate counts while sampling
application:
  REDMetrics:
    enabled: true

# To scale the gateway, increase the replicaCount
gateway:
  replicaCount: 3

# ‡ Decrease the resources required for the forwarder daemonset since the
# gateway now handles most of the work.
forwarder:
  resources:
    requests:
      cpu: 250m
      memory: 256Mi
    limits:
      memory: 256Mi

†: For more information on the configuration format, see the tail sampling processor documentation.

‡: For more information on resource requirements for the forwarder daemonset, see Tune service resource requests and limits.

Verification

Once tail sampling is enabled in your Observe Agent, you should see the metric otelcol_processor_tail_sampling_count_traces_sampled_total in your Kubernetes Explorer/Prometheus Metrics dataset. These metrics will have counts of the traces seen by the agent along with a sampled boolean label.

The full list of metrics generated by the tail sampling processor can be found here: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/documentation.md