APM reference

This page describes what data is provided by APM and how it is used.

Span schema

Spans are ingested to the Tracing/Span dataset.

Observe APM is OpenTelemetry-native, meaning it is powered entirely by OpenTelemetry data and is fully compatible with the OpenTelemetry semantic conventions. In order to make the product more intuitive for users who may not be intimately familiar with OpenTelemetry, we transform raw OpenTelemetry data into the following schema:

Span dataset field	Underlying OpenTelemetry field from which it is derived
`start_time`	Span `start_time_unix_nano`
`duration`	Span `start_time_unix_nano` and `send_time_unix_nano`
`response_status`	Span `status.code`
`service_name`	`service.name` in the span's resource attributes
`span_name`	Span `name`
`status_code`	derived from `http.status_code`, `rpc.status_code`, `grpc.status_code`, or `rpc.grpc.status_code` in the span's attributes
`error`	`True` if the Span Status is set to `Error`, false otherwise
`status_message`	Span Status object's `status.message`
`span_type`	Span `kind` (see below for the mapping between `span_type` and `kind`
`service_namespace`	`service.namespace` in the span's resource attributes
`service_instance_id`	`service.instance.id` in the span's resource attributes
`environment`	`deployment.environment` in the span's resource attributes
`service_version`	`service.version` in the span's resource attributes
`attributes`	The span's attributes
`resource_attributes`	The span's resource attributes
`instrumentation_library`	Instrumentation Scope object
`span_id`
`parent_span_id`
`trace_id`

The span_type field provides an intuitive categorization of OpenTelemetry span kinds, mapping 1:1 to the kind field in OpenTelemetry but using more user-friendly terms to describe the span’s role in the system:

Observe `span_type` value	OpenTelemetry Span `kind` value
`Service entry point`	`SERVER`
`Remote call`	`CLIENT`
`Async consumer`	`CONSUMER`
`Async producer`	`PRODUCER`
`Internal operation`	`INTERNAL`

Endpoint terminology

The "Endpoint" terminology in Observe has equivalents in other APM products:

Observe	Datadog	New Relic	AppDynamics / Dynatrace
Endpoint	Resource	Transaction	Business Transaction

Toggling between HTTP/RPC endpoints and message-based endpoints in Observe also has equivalents in other APM products:

Observe	Datadog	New Relic
Endpoint dropdown	Operation dropdown	Web/non-web transaction dropdown

Underlying OpenTelemetry data used to detect endpoints:

Spans with kind SERVER are used to detect HTTP/RPC endpoints, which are known as "Service entry points" in Observe. The endpoint name in Observe is the span name.
Spans with kind CONSUMER are used to detect message-based endpoints, which are known as "Async consumers" in Observe. The endpoint name in Observe is the span name.

The Service Inspector also allows you to toggle between synchronous (HTTP/RPC) and asynchronous (message-based) endpoints with the "operation type" dropdown.

Correlation tags on span data

The following correlation tags are added to the Span dataset and can be used to correlate other data with traces:

Correlation tag	Span dataset field on which it is defined
`deployment.environment.name`	`resource_attributes."deployment.environment"`
`k8s.cluster.uid`	`resource_attributes."k8s.cluster.uid"`
`k8s.namespace.name`	`resource_attributes."k8s.namespace.name"`
`k8s.pod.name`	`resource_attributes."k8s.pod.name"`
`service.name`	`resource_attributes."service.name"`
`service.namespace`	`resource_attributes."service.namespace"`
`service.version`	`resource_attributes."service.version"`
`service.name`	`service_name`
`service.namespace`	`service_namespace`
`operation.name`	`span_name`
`service.version`	`service_version`

Span event dataset schema

Span events are ingested to the Tracing/Span Event dataset.

Span Event dataset field	Underlying OTel field from which it is derived
`span_id`	the `span_id` of the span containing the event
`trace_id`	the `trace_id` of the span containing the event
`event_name`	The event's `name`
`attributes`	The event's attributes
`resource_attributes`	The resource attributes of the span containing the event

Span link dataset schema

Span links are ingested to the Tracing/Span Link dataset.

Span Link dataset field	Underlying OTel field from which it is derived
`span_id`	the `span_id` of the span containing the link
`trace_id`	the `trace_id` of the span containing the link
`linked_span_id`	the link's `span_id`
`linked_trace_id`	the link's `trace_id`
`attributes`	the link's attributes

APM metrics

The following metrics are derived from spans and are available in the Metric Explorer, and can be used for dashboarding and monitors.

Facets

The following dimensions are tracked by all the APM metrics:

service.name, peer.db.name, and peer.messaging.system: they are mutually exclusive and represent the system being instrumented -- microservice, database, or messaging system.
service.namespace
service.version
span.name: only the span names for spans that correspond to endpoints (i.e., spans of type Service entry point or Async consumer) are available in APM metrics
span.type
k8s.cluster.uid
k8s.namespace.name
k8s.pod.name

RED metrics

apm_service_call_count
apm_service_error_count
apm_service_duration (histogram)
The following metrics are the same as above but with extra dimensions, i.e., status.code and status.message:
- apm_service_call_count_with_status
- apm_service_error_count_with_status
- apm_service_duration_with_status
apm_service_duration_log: same as apm_service_duration but values are log base-2
apm_service_duration_sum_with_status: total duration (i.e., sum of latency across all invocations)
apm_service_exception_count: exception count per service
- Unique labels/dimensions:
  - exception.type
  - exception.stacktrace
  - exception.message.pattern.hash
apm_service_error_count_by_fingerprint: number of invocations per unique exceptions per service

Other metrics are also published beyond the basic RED metrics, all with the prefix apm_.

Updated about 2 months ago