OpenTelemetry (Public Preview)

OpenTelemetry is a vendor-neutral open source observability framework for generating and forwarding telemetry data. Commonly collected data includes traces, metrics, and logs. OpenTelemetry also supports distributed tracing for tracing activity across multiple services. For more about OpenTelemetry distributed tracing, see the OpenTelemetry documentation.

Note

The Observe OpenTelemetry app supports OpenTelemetry traces and metrics. Since OpenTelemetry logging is under active development by the Cloud Native Computing Foundation, Observe recommends ingesting log data by another method, such as the Observe Kubernetes app.

The OpenTelemetry app ingests your OpenTelemetry data to monitor and troubleshoot your application performance and allows you to answer questions such as:

  • Which microservices contain an elevated error rate, and what are the errors?

  • Which microservice in a slow request took the longest to respond?

  • Which other microservices call a particular service?

There are two starting points in the OpenTelemetry app:

  • The Services Home dashboard

  • The Trace Search dashboard

Use the Services Home dashboard to get a health overview of all the microservices in an application. To investigate a specific trace, use the Trace Search dashboard.

Viewing Service Performance in Observe

The Services Home dashboard provides a high level overview of your application services performance. You can quickly get insights into span rates, error rates, and latency for each service.

Service Home dashboard showing span rate, error rate, and latency.

Figure 1 - The Service Home dashboard

Double-click on a Service n ame in the Service Summary table to see more detail in the Service Inspector such as:

  • Span Latency by Operation

  • A summary of span rate, error rate, and latency by Operation

  • Upstream Caller/Downstream Callee span rates

  • Slowest Spans

  • Spans with Errors

Service Inspector

Figure 2 - The Service Inspector

Double-click on an Operation name to inspect the metrics, including span rate, error rate, latency, slowest spans, and spans with errors:

Operations Inspector

Figure 3 - The Operations Inspector

Viewing Trace Performance in Observe

The Trace Search dashboard summarizes all traces and allows you to search for traces of interest. You can search based on the following criteria:

  • A range of minimum and maximum trace durations

  • Traces containing particular values, such as those with errors

Trace Search Dashboard

Figure 4 - The Trace Search dashboard

For example, to search for traces containing errors:

  1. Click Dataset filter.

  2. Select the field, error, the operator =, and the value true.

  3. Click Apply to filter the data.

Trace Search with filter for traces with errors

Figure 5 - Filtering for traces with errors

The dashboard displays details of your selected traces:

  • Trace rate, error rate, and trace duration for the matching traces

  • A Traces table with a list of the individual traces

  • A health overview of the Services and Operations for the matching traces

Double-click on the Trace name to inspect a trace to open the Trace Inspector. The Trace Inspector displays a waterfall view of all the services called from this trace.

Trace Inspector

Figure 6 - The Trace Inspector

Click on a span to show additional detail:

Trace Inspector with side rail details

Figure 7 - Trace Inspector details

Use the Spans and Events table to inspect errors, associated services, or operations within each span.

Setup

Installing the Observe OpenTelemetry app

Install the OpenTelemetry App for Observe from the App section in Workspace Setting.

OpenTelemetry app

Figure 8 - The OpenTelemetry app card

The default configuration handles many common configurations, but you may also specify options appropriate for your environment:

  • Datastream - the datastream receiving your OpenTelemetry data.

  • Extract Attributes - List of attributes to extract as columns for all spans.

  • Extract Resource Attributes - List of resource attributes to extract as columns for all spans.

  • Freshness Default - Default dataset freshness. May be overridden with freshness input.

  • Kubernetes App - If you also use the Kubernetes app, automatically link your Kubernetes resources to your OpenTelemetry datasets and enable GraphLink.

  • Max Expiry - Expiry time for OpenTelemetry traces.

  • Max Time Diff - Maximum time difference for the processing time window.

  • Name Format - The location of your OpenTelemetry resources which is, by default, the OpenTelemetry folder.

Creating a data connection

To configure the OpenTelemetry app to accept incoming data, create a data connection and a token. If you configured the app to use an existing datastream, this is the datastream token associated with the specified datastream. If not, then create a new connection:

  1. From the Apps tab, click Manage on the OpenTelemetry card to open the App settings page.

  2. Click the Connections tab.

  3. In the Data connection requirements section, click Create connection.

  4. Enter a name and description for the new connection token and click Continue.

  5. Copy the newly created token and store it in a safe location. You will need it later to configure OpenTelemetry data ingestion.

  6. Click Continue to open the OpenTelemetry collector instructions.

Note

You may also access the collector instructions by clicking View instructions in the Connections tab.

Configure the OpenTelemetry collector

The Observe OpenTelemetry collector provides a gateway process that collects span data and sends it to Observe.

To install the collector, you need the following items:

  • An Observe Customer ID.

  • The OpenTelemetry connection token you created in the previous section.

OpenTelemetry for Kubernetes

For Kubernetes environments, deploy the collector with our Kustomize manifest following the steps below.

  1. Create a secret for the OpenTelemetry collector.

    OBSERVE_CUSTOMER='customer_id'
    OBSERVE_TOKEN='connection_token_otel_app'
    kubectl -n observe create secret generic credentials \
            --from-literal=OBSERVE_CUSTOMER=${OBSERVE_CUSTOMER?} \
            --from-literal=OBSERVE_TOKEN=${OBSERVE_TOKEN?}
    
  2. Install the collector Kustomize stack using kubectl:

    kubectl apply -k github.com/observeinc/manifests/stack/otel
    

    For larger environments (100+ Kubernetes nodes), specify a larger deployment size:

    kubectl apply -k github.com/observeinc/manifests/stack/otel/l
    

For full details, see the manifest README in GitHub.

Testing OpenTelemetry ingest with Docker

You may also test your OpenTelemetry instrumentation in a development environment with Docker Desktop. To install the collector for testing, you need the following:

Note

This example is intended for testing and is unsuitable for production use.

In the first shell:

  1. Create a new directory for testing.

    mkdir otel-test
    
  2. Create a config.yaml configuration file with the following contents:

    Click to expand
    # Test OpenTelemetry collector configuration. Not for production use.
    exporters:
      logging:
        loglevel: "${OTEL_LOG_LEVEL}"
      otlphttp:
        endpoint: "https://${OBSERVE_CUSTOMER}.${OBSERVE_COLLECTOR_HOST}:${OBSERVE_COLLECTOR_PORT}/v1/otel"
        headers:
          authorization: "Bearer ${OBSERVE_TOKEN}"
    extensions:
      health_check: {}
      zpages: {}
    processors:
      probabilistic_sampler:
        hash_seed: 22
        sampling_percentage: 100
      batch:
      memory_limiter:
        # 80% of maximum memory up to 2G
        limit_mib: 2048
        # 25% of limit up to 2G
        spike_limit_mib: 512
        check_interval: "2s"
    receivers:
      zipkin:
      otlp:
        protocols:
          grpc:
          http:
    service:
      telemetry:
        logs:
          level: "${OTEL_LOG_LEVEL}"
      extensions: [health_check, zpages]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [probabilistic_sampler, memory_limiter, batch]
          exporters: [logging, otlphttp]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging]
    
  3. If not already defined in your shell environment, set the OBSERVE_CUSTOMER and OBSERVE_TOKEN environment variables to your Observe customer ID and OpenTelemetry connection token.

  4. Start the Docker process, configured to accept data on ports 4317 (gRPC) and 4318 (POST).

    export OBSERVE_TOKEN ; export OBSERVE_CUSTOMER
    docker run -p 4317:4317 -p 4318:4318 \
            --env OBSERVE_TOKEN="${OBSERVE_TOKEN}" \
            --env OBSERVE_CUSTOMER="${OBSERVE_CUSTOMER}" \
            --env OBSERVE_COLLECTOR_HOST="collect.observeinc.com" \
            --env OBSERVE_COLLECTOR_PORT="443" \
            --env OTEL_LOG_LEVEL="debug" \
        -v $(pwd -P)/config.yaml:/etc/otel-collector-config.yaml \
        otel/opentelemetry-collector-contrib:0.62.1 \
        --config=/etc/otel-collector-config.yaml
    
  5. Leave the Docker process running and open a second shell.

In the second shell:

  1. Create sample span data in test.json, using the payload below. You may use the same otel-test directory created previously.

    Click to expand
    {
     "resourceSpans": [
       {
         "resource": {
           "attributes": [
             {
               "key": "service.name",
               "value": {
                 "stringValue": "test-with-curl"
               }
             }
           ]
         },
         "scopeSpans": [
           {
             "instrumentationLibrary": {
               "name": "manual-test"
             },
             "spans": [
               {
                 "traceId": "71699b6fe85982c7c8995ea3d9c95df2",
                 "spanId": "3c191d03fa8be065",
                 "name": "spanitron",
                 "kind": 2,
                 "droppedAttributesCount": 0,
                 "events": [],
                 "droppedEventsCount": 0,
                 "status": {
                   "code": 1
                 }
               }
             ]
           }
         ]
       }
     ]
    }
    
  2. Send the test data to the collector with curl.

    curl -i http://localhost:4318/v1/traces -X POST -H "Content-Type: application/json" -d @test.json
    
  3. Confirm the collector accepted the test data.

    $ curl -i http://localhost:4318/v1/traces -X POST -H "Content-Type: application/json" -d @test.json
    HTTP/1.1 200 OK
    Content-Type: application/json
    Date: Mon, 14 Nov 2022 22:48:06 GMT
    Content-Length: 2
    
    {}
    

In Observe, view the ingested data:

  1. On the Apps tab, locate the OpenTelemetry app card from the list of installed apps.

  2. Click Manage, and then Connections.

  3. Under Existing connections, click on your token to view the details.

  4. Click Open dataset to view the data associated with this token.