Trace Explorer

Observe can help you analyze user requests and transactions and connect them to resources (e.g., services, infrastructure) that served those requests. By reviewing Observability data in a resource-oriented way, you can quickly prioritize and troubleshoot issues that arise.

Who uses traces and why?

A common scenario is: your customers are reporting that their experience is slow or erroring. But the user requests coming in are flowing through dozens or potentially hundreds of microservices running on a variety of architectures. If you’re a developer trying to debug a bad request, the questions you need to ask in order to make progress include:

  • Where in the request path did the request fail (i.e., which microservice?)

  • Are there issues upstream or downstream of the service I own?

  • What infrastructure was used to serve each part of the request?

  • Is my own code slow?

Tracing data can cut down the MTTR from hours to as little as a few minutes, because it brings together so much context in a single place.

What is a trace?

A trace is a representation of a single end-to-end user request flowing through a distributed system. For instance, a trace might represent a series of related transactions from login to browse to purchase on a storefront.

A trace is made up of spans, or intervals in an interval dataset, each of which has the trace ID. Each span represents some bit of work done by some part of some service, and it contains metadata describing what work it did, how long it took, and what resources were involved in the execution of that work.

Tracing instrumentation propagates the trace ID across the request, creating spans and adding attributes to those spans that make them searchable and useful for debugging.

How to perform common workflows with trace data in Observe

Common user need when looking at a trace

Observe Trace Viewer feature that supports this need

Inspect the call structure of a trace

Flame chart

View a summary of where time was spent in a trace

Service legend in flame chart, as well as inferring from the flame chart itself.

See what resources were used to serve an operation in a trace, or other attributes (e.g., business metadata) associated with the operation

Fields & Attributes tab underneath flame chart.

Inspect errors in a particular span & across a trace

Scan the flame chart for spans in red, then click into those spans for more inspection.

Inspect logs associated with a span or trace

Link to a Log Explorer search with trace id or span id pre-populated. Also, Span Events tab underneath flame chart.

Common user need when searching for traces

Observe Trace Explorer feature that supports this need

Filter a trace search to a particular service

Service facet filter

Filter a trace search to a particular endpoint

Operation facet filter

Filter a trace search to errors only

Status facet filter

Filter a trace search to a particular HTTP status code

status_code facet filter

Filter a trace search to a particular environment

environment facet filter

Filter a trace search to service entry spans only

Span type drop-down and facet filter

Filter a trace search to a particular span attribute

Extract the attribute from the attributes field, then search on it using the query bar.

Filter a trace search to a particular resource attribute

Extract the attribute from the resource_attributes field, then search on it using the query bar.

Exporting Data

To download the data displayed in Trace Explorer, click the Export button. You may select CSV or JSON format, and a maximum size limit (one thousand, ten thousand, or one hundred thousand rows). Note that hidden fields will be included. Use the pick_col OPAL verb to reduce the width of downloaded data.

For more information on where the underlying data comes from, see the following: