APM observability reference

This page describes the data APM provides and how it is used.

APM is OpenTelemetry-native. It is powered by OpenTelemetry data and follows the OpenTelemetry semantic conventions. Raw OpenTelemetry fields are transformed into the schemas below so the same data can be queried without knowing every semantic-convention name.

Datasets

A trace datastream writes three source datasets. APM content then builds the datasets under Tracing/.

Source datasets

Dataset	What it holds
`Tracing/Span Raw`	Raw OpenTelemetry spans, as ingested.
`Tracing/Span Event`	Span events.
`Tracing/Span Link`	Span links.

APM content datasets

Dataset	What it holds
`Tracing/Span`	The modeled span view described in Span schema. Built from `Tracing/Span Raw`.
`Tracing/Trace`	One row per trace, built from `Tracing/Span`.
`Tracing/Canonical Trace`	A compact trace used by Service Explorer and the Service Inspector.
`Tracing/Service Explorer Spans`	Spans shaped for Service Explorer.
`Tracing/Service`	Discovered services.
`Tracing/Deployment`	Discovered deployments.
`Tracing/Service Metrics`	RED metrics derived from spans. See APM metrics. This dataset is omitted on tenants where those metrics are produced by the agent instead.
`Tracing/Service Edge Metrics`	Calls between services, and calls to databases and message brokers.
`Tracing/Service Inspector Metrics`	Metrics behind the Service Inspector.
`Tracing/Service Explorer Drilldown Metrics`	Metrics behind Service Explorer drilldown.
`Tracing/Error Tracking Metrics`	Exception and error-fingerprint metrics, from span events and Service Explorer spans.

Tracing/Service Explorer Spans carries the same correlation tags as Tracing/Span.

Span schema

Tracing/Span is a view of Tracing/Span Raw. Spans are ingested into Tracing/Span Raw. The columns below are the modeled view.

Customers can add further columns with span field extraction. Those columns are not part of this default schema.

Dataset field	Underlying OpenTelemetry field from which it is derived
`start_time`	Span `start_time_unix_nano`.
`end_time`	Span `end_time_unix_nano`. Hidden in the UI by default. It is the end of the span interval.
`duration`	`end_time_unix_nano` minus `start_time_unix_nano`, in nanoseconds.
`response_status`	`"Error"` when the span status code is `STATUS_CODE_ERROR` (2). `"Ok"` otherwise, including when the status is unset.
`service_name`	`service.name` in the span's resource attributes.
`span_name`	Span `name`.
`status_code`	First present span attribute among `rpc.grpc.status_code`, `grpc.status_code`, `rpc.status_code`, `http.status_code`, and `http.response.status_code`. Null when none are set.
`error`	`true` when the span status code is `STATUS_CODE_ERROR` (2). `false` otherwise.
`status_message`	Span status `status.message`.
`span_type`	Span `kind`. See span_type. Unrecognized kinds, including `UNSPECIFIED`, are `Unknown`.
`service_namespace`	`service.namespace` in the span's resource attributes. Empty string when unset.
`service_instance_id`	`service.instance.id` in the span's resource attributes. Empty string when unset.
`environment`	`deployment.environment.name` in the span's resource attributes, falling back to `deployment.environment`. Empty string when neither is set.
`service_version`	`service.version` in the span's resource attributes. Empty string when unset.
`attributes`	The span's attributes.
`resource_attributes`	The span's resource attributes.
`instrumentation_library`	The span's instrumentation scope.
`parent_span_id`	Span `parent_span_id`.
`span_id`	Span `span_id`.
`trace_id`	Span `trace_id`.
`fields`	Span fields that are not promoted to their own columns, including `status`, `trace_state`, `flags`, and dropped-count fields.
`meta`	Ingest metadata. This is not an OpenTelemetry field.

Span event Dataset schema

Span events are ingested to Tracing/Span Event. A span event is something that happened during a span, such as an exception.

Dataset field	Underlying OpenTelemetry field from which it is derived
`timestamp`	The event's `time_unix_nano`.
`span_id`	`span_id` of the span that contains the event.
`trace_id`	`trace_id` of the span that contains the event.
`event_name`	The event's `name`.
`attributes`	The event's attributes.
`resource_attributes`	Resource attributes of the span that contains the event.
`fields`	Event fields that are not promoted to their own columns. Includes `dropped_attributes_count`.
`meta`	Ingest metadata. This is not an OpenTelemetry field.

Span link Dataset schema

Span links are ingested to Tracing/Span Link. A span link connects spans that are related but do not have a parent-child relationship, often across traces.

Dataset field	Underlying OpenTelemetry field from which it is derived
`start_time`	`start_time_unix_nano` of the span that contains the link.
`end_time`	`end_time_unix_nano` of the span that contains the link.
`span_id`	`span_id` of the span that contains the link.
`trace_id`	`trace_id` of the span that contains the link.
`linked_span_id`	The link's `span_id`.
`linked_trace_id`	The link's `trace_id`.
`attributes`	The link's attributes.
`fields`	Link fields that are not promoted to their own columns: `trace_state`, `dropped_attributes_count`, and `flags`.
`meta`	Ingest metadata. This is not an OpenTelemetry field.

Correlation tags

Correlation tags join traces to other data. Tags on Tracing/Span are below. Tracing/Service Explorer Spans uses this same set.

`Tracing/Span`

Correlation tag	Field
`deployment.environment.name`	`environment`
`service.name`	`service_name`
`service.namespace`	`service_namespace`
`service.version`	`service_version`
`span.id`	`span_id`
`operation.name`	`span_name`
`trace.id`	`trace_id`
`host.name`	`resource_attributes."host.name"`
`host.uid`	`resource_attributes."host.uid"`
`k8s.cluster.name`	`resource_attributes."k8s.cluster.name"`
`k8s.cluster.uid`	`resource_attributes."k8s.cluster.uid"`
`k8s.namespace.name`	`resource_attributes."k8s.namespace.name"`
`k8s.pod.name`	`resource_attributes."k8s.pod.name"`
`k8s.pod.uid`	`resource_attributes."k8s.pod.uid"`
`k8s.cronjob.name`	`resource_attributes."k8s.cronjob.name"`
`k8s.daemonset.name`	`resource_attributes."k8s.daemonset.name"`
`k8s.deployment.name`	`resource_attributes."k8s.deployment.name"`
`k8s.ingress.name`	`resource_attributes."k8s.ingress.name"`
`k8s.job.name`	`resource_attributes."k8s.job.name"`
`k8s.node.name`	`resource_attributes."k8s.node.name"`
`k8s.node.uid`	`resource_attributes."k8s.node.uid"`
`k8s.replicaset.name`	`resource_attributes."k8s.replicaset.name"`
`k8s.statefulset.name`	`resource_attributes."k8s.statefulset.name"`
`peer.db.name`	`attributes."peer.db.name"`
`peer.messaging.system`	`attributes."peer.messaging.system"`
`messaging.destination.name`	`attributes."messaging.destination.name"`
`peer.server.address`	`attributes."peer.server.address"`

`Tracing/Span Event`

Correlation tag	Field
`span.id`	`span_id`
`trace.id`	`trace_id`
`k8s.pod.name`	`resource_attributes."k8s.pod.name"`
`k8s.namespace.name`	`resource_attributes."k8s.namespace.name"`
`k8s.cluster.uid`	`resource_attributes."k8s.cluster.uid"`
`deployment.environment.name`	`resource_attributes."deployment.environment"`
`service.name`	`resource_attributes."service.name"`

On span events, deployment.environment.name is read from resource_attributes."deployment.environment".

`Tracing/Service` and `Tracing/Deployment`

Dataset	Correlation tag	Field
`Tracing/Service`	`deployment.environment.name`	`environment`
`Tracing/Service`	`service.name`	`service_name`
`Tracing/Service`	`service.namespace`	`service_namespace`
`Tracing/Deployment`	`deployment.environment.name`	`environment`
`Tracing/Deployment`	`service.name`	`service_name`
`Tracing/Deployment`	`service.namespace`	`service_namespace`
`Tracing/Deployment`	`service.version`	`service_version`

Service Metrics, Service Edge Metrics, Service Inspector Metrics, Service Explorer Drilldown Metrics, and Error Tracking Metrics also publish correlation tags. Those tags are paths inside each dataset's tags column and follow the dimensions in APM metrics. peer.messaging.system and messaging.destination.name are registered on those metric datasets when message brokers are modeled as entities.

APM metrics

APM derives metrics from spans. They are available in Metric Explorer, and they can be used on dashboards and in monitors.

The RED metrics in this section are published on Tracing/Service Metrics. On tenants where that dataset is retired, rate, error, and duration are produced by the agent, and the metric names can differ.

Dimensions

Dimensions are the labels on these metrics. Filter and group by them to choose a service, version, endpoint, cluster, or pod.

service.name, peer.db.name, and peer.messaging.system name what the point is about. A point is for a service, a database, or a message broker, and only one of those three is set. Database points are always produced. Message-broker points, and messaging.destination.name, are produced when message brokers are modeled as entities.

The basic rate, error, and duration metrics carry service, namespace, environment, span name, and span type. apm_service_duration_log also carries service.version. Status, Kubernetes, host, container, and peer labels are populated on the _with_status metrics.

Dimension	What it tells you
`service.name`	The service that handled the call. Empty when the point is for a database or a message broker.
`peer.db.name`	The database the call went to. Set on database points.
`peer.messaging.system`	The message broker, such as Kafka. Set on message-broker points.
`messaging.destination.name`	The queue or topic on a message-broker point. Present when message brokers are modeled as entities.
`service.namespace`	A grouping of services, from the resource attribute `service.namespace`. Often a team or system name.
`service.version`	The deployed version of the service, from `service.version`.
`environment`	The deployment environment, from `deployment.environment.name`, falling back to `deployment.environment`.
`span.name`	The operation name. For a service this is the endpoint. For a database or message broker this is the call's span name.
`span.type`	The span's role, such as `Service entry point`, `Async consumer`, or `Remote call`.
`status.code`	The protocol status code, from HTTP, gRPC, or RPC. Populated on the `_with_status` metrics.
`status.message`	The span status message. Populated on the `_with_status` metrics.
`k8s.cluster.uid`	The Kubernetes cluster UID of the workload that emitted the span.
`k8s.namespace.name`	The Kubernetes namespace of that workload.
`k8s.pod.name`	The Kubernetes pod that emitted the span.
`host.name`	The host that emitted the span.
`container.id`	The container that emitted the span.

RED metrics

Rate, Errors, and Duration (RED) are the three health signals for a service. They count incoming work: calls that entered a service, and messages a service consumed. They also count calls to databases, and calls to message brokers when message brokers are modeled as entities.

Metric	What it tells you
`apm_service_call_count`	How many calls the service handled. This is the rate.
`apm_service_error_count`	How many of those calls failed. Divide by the call count for the error rate. A call is failed when `response_status` is `Error`.
`apm_service_duration`	How long those calls took, as a histogram in nanoseconds. Use it for percentiles such as p50, p95, and p99.
`apm_service_call_count_with_status`	Call count split by `status.code` and `status.message`.
`apm_service_error_count_with_status`	Error count split by `status.code` and `status.message`.
`apm_service_duration_with_status`	Latency histogram split by `status.code` and `status.message`.
`apm_service_duration_log`	The same latency histogram, with each duration stored as its log base 10. Very fast and very slow calls stay visible on one chart. The values are log10 of the duration in nanoseconds.
`apm_service_duration_sum_with_status`	Total time spent, in nanoseconds, split by status. Divide by `apm_service_call_count_with_status` for the average latency.

Error count metrics

These metrics are on Tracing/Error Tracking Metrics. They break failures down by exception or by status-message pattern. They are not the RED error signal.

Metric	What it tells you
`apm_service_exception_count`	How many exception events a service emitted. An exception event is a span event named `exception`.
`apm_service_error_count_by_fingerprint`	How many failed incoming calls share the same status-message pattern. Counts service entry points and async consumers whose span status is `Error`.
`apm_service_exception_count_by_fingerprint`	How many exceptions share a stack-trace fingerprint. Published when exception fingerprinting is enabled, for Go, Python, and Node.js.

apm_service_exception_count is split by these dimensions, in addition to service, version, environment, and Kubernetes identity:

Dimension	What it tells you
`exception.type`	The exception type.
`exception.stacktrace`	The exception stack trace.
`exception.message.pattern.hash`	A hash of the message shape. Letters are stripped and whitespace is collapsed, so similar messages group together.

apm_service_error_count_by_fingerprint is split by service.name, service.namespace, environment, span.name, span.type, status.code, and status.message.

apm_service_exception_count_by_fingerprint adds exception.fingerprint, exception.message, language, trace.id, and span.id.

Service edge metrics

These metrics are on Tracing/Service Edge Metrics. They describe a call from one service to another, or from a service to a database or message broker. Parent dimensions name the caller.

Metric	What it tells you
`apm_service_edge_call_count`	How many calls crossed this edge.
`apm_service_edge_error_count`	How many of those calls failed.
`apm_service_edge_duration`	Latency of those calls, as a histogram in nanoseconds.
`apm_service_edge_duration_sum`	Total time spent on those calls, in nanoseconds. Published when message brokers are modeled as entities.

Edge points also carry parent.service.name, parent.service.namespace, and parent.environment. When message brokers are modeled as entities, they also carry messaging.operation.name, messaging.operation.type, and messaging.consumer.group.name.

Service Inspector metrics

These metrics are on Tracing/Service Inspector Metrics. The for_service metrics are the service's own entry points. The edge metrics are calls that service made, and they include the parent service and parent span.

Metric	What it tells you
`apm_service_call_count_for_service`	Call count for the service's own entry points.
`apm_service_error_count_for_service`	Error count for those entry points.
`apm_service_duration_for_service`	Latency histogram for those entry points.
`apm_service_edge_call_count_for_service`	Call count for an edge from this service.
`apm_service_edge_error_count_for_service`	Error count for that edge.
`apm_service_edge_duration_for_service`	Latency histogram for that edge.

Inspector points can also carry parent.span.name, for.service.name, for.service.namespace, for.environment, and for.span.name. Database, external-call, and message-broker rows leave service.name empty and set peer.db.name, peer.server.address, or peer.messaging.system instead.

Service Explorer drilldown metrics

These metrics are on Tracing/Service Explorer Drilldown Metrics. They break an edge down by endpoint.

Metric	What it tells you
`apm_service_edge_call_count_by_endpoint`	Call count for an edge, split by endpoint.
`apm_service_edge_call_count_by_endpoint_unique`	Distinct-call count for that edge and endpoint.
`apm_service_edge_duration_sum_by_endpoint`	Total time spent on that edge and endpoint, in nanoseconds.

Drilldown points also carry service.kind, parent.service.name, parent.service.namespace, parent.environment, and parent.span.name.

Terminology

span_type is how Observe by Snowflake names the OpenTelemetry span kind. Endpoint is the product term for an incoming operation, and it lines up with the entry-point span types below.

`span_type`

OpenTelemetry records a span's role as kind, using SERVER, CLIENT, CONSUMER, PRODUCER, and INTERNAL. Observe by Snowflake keeps that split and shows it as span_type. Each label matches one kind. A kind Observe by Snowflake does not recognize, including UNSPECIFIED, is Unknown.

What you see in Observe by Snowflake	What the span is	OpenTelemetry span `kind`
Service entry point	A request entered this service. This is an incoming HTTP or RPC call, and it is what Observe by Snowflake treats as an endpoint. The endpoint name is the span name.	`SERVER`
Remote call	This service called something else: another service, a database, or another downstream system.	`CLIENT`
Async consumer	This service received a message from a queue or topic. Observe by Snowflake treats these as message-based endpoints, again named by the span name.	`CONSUMER`
Async producer	This service published a message to a queue or topic.	`PRODUCER`
Internal operation	Work inside the service that does not cross a service or messaging boundary.	`INTERNAL`
Unknown	The span kind was `UNSPECIFIED` or a value Observe by Snowflake does not map.	`UNSPECIFIED` or unrecognized

Endpoint terminology

The Endpoint term in Observe by Snowflake has equivalents in other APM products:

Observe by Snowflake	Datadog	New Relic	AppDynamics / Dynatrace
Endpoint	Resource	Transaction	Business Transaction

Toggling between HTTP/RPC endpoints and message-based endpoints also has equivalents in other APM products:

Observe by Snowflake	Datadog	New Relic
Endpoint dropdown	Operation dropdown	Web/non-web transaction dropdown

OpenTelemetry data used to detect endpoints:

Spans with kind SERVER are HTTP/RPC endpoints. In Observe by Snowflake these are Service entry points. The endpoint name is the span name.
Spans with kind CONSUMER are message-based endpoints. In Observe by Snowflake these are Async consumers. The endpoint name is the span name.

The Service Inspector toggles between synchronous (HTTP/RPC) and asynchronous (message-based) endpoints with the operation type dropdown.

Updated 5 days ago

Did this page help you?