Trace Tail Sampling¶
Trace tail sampling allows you to reduce the span data sent to Observe while having full control of which traces are dropped.
To enable trace tail sampling, configure the following in your values.yaml
file:
# Enable the gateway deployment and trace sampling
gatewayDeployment:
enabled: true
traceSampling:
enabled: true
# † Uncomment and edit the following section to override the default
# tail sampling policies.
#
# config:
# policies:
# # Send all spans with status_code error to Observe
# - name: retain-all-errors
# type: status_code
# status_code: {status_codes: [ERROR]}
# # Send 10% of all other spans to Observe
# - name: sample-probabilistically-10-percent
# type: probabilistic
# probabilistic: {sampling_percentage: 10}
# Enable RED metrics generation in order to get accurate counts while sampling
application:
REDMetrics:
enabled: true
# To scale the gateway, increase the replicaCount
gateway:
replicaCount: 3
# ‡ Decrease the resources required for the forwarder daemonset since the
# gateway now handles most of the work.
forwarder:
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
memory: 256Mi
†: For more information on the configuration format, see the tail sampling processor documentation.
‡: For more information on resource requirements for the forwarder daemonset, see Tune service resource requests and limits.
Verification¶
Once tail sampling is enabled in your Observe Agent, you should see the metric otelcol_processor_tail_sampling_count_traces_sampled_total
in your Kubernetes Explorer/Prometheus Metrics
dataset. These metrics will have counts of the traces seen by the agent along with a sampled
boolean label.
The full list of metrics generated by the tail sampling processor can be found here: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/documentation.md