Installing the Observe OpenTelemetry App#

Install the OpenTelemetry App for Observe from the App section in Workspace Setting.

OpenTelemetry app

Figure 1 - The OpenTelemetry app card

The default configuration handles many common needs, but you may also specify options appropriate for your environment:

  • Datastream - the datastream receiving your OpenTelemetry data.

  • Extract Attributes - List of attributes to be extracted as columns for all spans.

  • Extract Resource Attributes - List of resource attributes to be extracted as columns for all spans.

  • Freshness Default - Default dataset freshness. It may be overridden with freshness input.

  • Kubernetes App - If you also use the Kubernetes App, automatically link your Kubernetes resources to your OpenTelemetry datasets to enable GraphLink.

  • Max Expiry - Expiry time for OpenTelemetry traces.

  • Max Time Diff - Maximum time difference for the processing time window.

  • Name Format - The location of your OpenTelemetry resources is the OpenTelemetry folder by default.

Observe also has support for OpenTelemetry Logs and Metrics, however the bundled datasets are disabled by default. If you are sending Logs and Metrics to Observe via the OTEL ingestion endpoints, we recommend enabling these datasets. You can enable them via configuration page under the “Enable Logs” and “Enable Metrics” options.

OpenTelemetry app

Figure 2 - The OpenTelemetry Logs & Metrics dataset configuration

Creating a data connection#

To allow the OpenTelemetry app to accept incoming data, create a data connection and a token. If you configured the app to use an existing datastream, this is the datastream token associated with the specified datastream. If not, create a new connection:

  1. From the Apps tab, click Manage on the OpenTelemetry card to open the App settings page.

  2. Click the Connections tab.

  3. In the Data connection requirements section, click Create connection.

  4. Enter a name and description for the new connection token and click Continue.

  5. Copy the newly created token and store it in a safe location. You need it later to configure OpenTelemetry data ingestion.

  6. Click Continue to open the OpenTelemetry collector instructions.

Note

You may also access the Collector instructions by clicking View instructions in the Connections tab.

Configuring the OpenTelemetry collector#

The Observe OpenTelemetry collector is a gateway process that collects span data and sends it to Observe.

To install the collector, you need the following:

  • An Observe Customer collection endpoint.

  • The OpenTelemetry connection token you created in the previous section.

OpenTelemetry for Kubernetes#

For Kubernetes environments, deploy the collector with the Observe traces Helm chart using the following steps.

Note

While Helm is the preferred installation method, you can find alternate instructions using Kustomize.

Helm can be used to install the OpenTelemetry collector to an existing namespace. If it does not already exist, you must create an observe namespace with a human readable cluster name annotation. This can be done with your infrastructure management tool of choice (e.g. Terraform) or with the following command:

CLUSTER_NAME="My Cluster"
kubectl create namespace observe && \
kubectl annotate namespace observe observeinc.com/cluster-name="$CLUSTER_NAME"

See https://github.com/observeinc/helm-charts#namespace for more information on the namespace requirement.

To add the Observe Helm repository, run the following commands:

helm repo add observe https://observeinc.github.io/helm-charts
helm repo update

Then create a release for the traces chart in the observe namespace, using helm install:

helm install --namespace=observe observe-traces observe/traces \
  --set global.observe.collectionEndpoint="${OBSERVE_COLLECTION_ENDPOINT?}" \
  --set observe.token.value="${OBSERVE_TOKEN?}"

Save the values to a file, and keep it somewhere safe for further configuration and upgrades:

helm -n observe get values observe-traces -o yaml > observe-traces-values.yaml

Configuration#

See Helm chart configuration for more details, including setting custom resource limits for the agent.

Testing OpenTelemetry Ingestion with Docker#

You may also test your OpenTelemetry instrumentation in a development environment with Docker Desktop. To install the collector for testing, you need the following items:

Note

This example is intended for testing; it is not suitable for production use.

In the first shell, use the following steps:

  1. Create a new directory for testing.

    mkdir otel-test
    
  2. Create a config.yaml configuration file with the following contents:

    Click to expand
    exporters:
       logging:
         loglevel: "${OTEL_LOG_LEVEL}"
       otlphttp:
         endpoint: "https://${OBSERVE_CUSTOMER}.${OBSERVE_COLLECTOR_HOST}:${OBSERVE_COLLECTOR_PORT}/v2/otel"
         headers:
           authorization: "Bearer ${OBSERVE_TOKEN}"
     extensions:
       health_check: {}
       zpages: {}
     receivers:
       otlp:
         protocols:
           grpc:
           http:
     service:
       extensions: [health_check, zpages]
       pipelines:
         traces:
           receivers: [otlp]
           exporters: [otlphttp]
         metrics:
           receivers: [otlp]
           exporters: [otlphttp]
         logs:
           receivers: [otlp]
           exporters: [otlphttp, logging]
       telemetry:
         metrics:
           address: 0.0.0.0:8888
         logs:
           level: "${OTEL_LOG_LEVEL}"
    
  3. Start the Docker process, configured to accept data on ports 4317 (gRPC) and 4318 (POST).

    export OBSERVE_TOKEN ; export OBSERVE_CUSTOMER
    docker run -p 4317:4317 -p 4318:4318 \
            --env OBSERVE_TOKEN="${OBSERVE_TOKEN}" \
            --env OBSERVE_CUSTOMER="${OBSERVE_CUSTOMER}" \
            --env OBSERVE_COLLECTOR_HOST="collect.observeinc.com" \
            --env OBSERVE_COLLECTOR_PORT="443" \
            --env OTEL_LOG_LEVEL="debug" \
        -v $(pwd -P)/config.yaml:/etc/otel-collector-config.yaml \
        otel/opentelemetry-collector-contrib:0.62.1 \
        --config=/etc/otel-collector-config.yaml
    
  4. Leave the Docker process running and open a second shell.

In the second shell, perform the following steps:

  1. Create sample span data in test.json using the payload below. You may use the same otel-test directory created previously.

    Click to expand
    {
     "resourceSpans": [
       {
         "resource": {
           "attributes": [
             {
               "key": "service.name",
               "value": {
                 "stringValue": "test-with-curl"
               }
             }
           ]
         },
         "scopeSpans": [
           {
             "instrumentationLibrary": {
               "name": "manual-test"
             },
             "spans": [
               {
                 "traceId": "71699b6fe85982c7c8995ea3d9c95df2",
                 "spanId": "3c191d03fa8be065",
                 "name": "spanitron",
                 "kind": 2,
                 "droppedAttributesCount": 0,
                 "events": [],
                 "droppedEventsCount": 0,
                 "status": {
                   "code": 1
                 }
               }
             ]
           }
         ]
       }
     ]
    }
    
  2. Send the test data to the collector with curl.

   curl -i http://localhost:4318/v1/traces -X POST -H "Content-Type: application/json" -d @test.json
  1. Confirm the collector accepted the test data.

  $ curl -i http://localhost:4318/v1/traces -X POST -H "Content-Type: application/json" -d @test.json
  HTTP/1.1 200 OK
  Content-Type: application/json
  Date: Mon, 14 Nov 2022 22:48:06 GMT
  Content-Length: 2
  
  {}

You can now view the ingested data:

  1. In the Apps tab, locate the OpenTelemetry app card in the list of installed apps.

  2. Click Manage and then Connections.

  3. Under Existing connections, click on your token to view details.

  4. Click Open dataset to view the data associated with this token.