Auto-Instrumentation with OpenTelemetry Operator in Kubernetes

This guide walks you through setting up cert-manager, opentelemetry-operator, and observe-agent using Helm and configuring an Instrumentation custom resource to enable auto-instrumentation in your Kubernetes cluster. Note that this guide only applies to Kubernetes clusters.

Prerequisites

  • A Kubernetes cluster (v1.20+ recommended)

  • kubectl installed and configured

  • helm installed (v3.0+)

1. Install cert-manager

cert-manager is required to handle TLS certificates in your cluster and for the Operator to be able to communicate with the Kubernetes cluster control plane and modify resources as required. Install it using Helm:

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io --force-update

# Install cert-manager with CRDs
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.17.0 \
  --set crds.enabled=true

2. Install otel-operator

The OpenTelemetry Operator manages telemetry collection and auto-instrumentation. Install it using the official Helm chart:

# Add the OpenTelemetry Helm repository
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

# Install otel-operator
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
--namespace observe \
--set "manager.collectorImage.repository=observeinc/observe-agent" \
--set "manager.extraArgs={--enable-go-instrumentation}"

Check that the operator is running:

kubectl get pods -n observe

3. Configure OpenTelemetry Instrumentation Custom Resource

Create an Instrumentation resource to enable automatic telemetry for supported workloads. To work with the observe-agent helm chart, the instrumentation will use the forwarder daemonset cluster endpoint as the target for the exporter. Below is an example YAML that configures auto-instrumentation for multiple languages:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: observe-auto-instrumentation
  namespace: observe
spec:
  exporter:
    endpoint: http://observe-agent-forwarder.observe.svc.cluster.local:4318
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_always_on
  java:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://observe-agent-forwarder.observe.svc.cluster.local:4318
  nodejs:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://observe-agent-forwarder.observe.svc.cluster.local:4318
  python:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://observe-agent-forwarder.observe.svc.cluster.local:4318
  dotnet:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://observe-agent-forwarder.observe.svc.cluster.local:4318
  go:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://observe-agent-forwarder.observe.svc.cluster.local:4318

Apply the configuration:

kubectl apply -f instrumentation.yaml

4. Install observe-agent

Finally, install the observe-agent Helm chart to collect and forward telemetry data to your Observe backend. See Observe Agent Helm Chart for detailed instructions.

5. Deploy Your Instrumented Application

The operator will look for specific annotations on your workloads to enable auto-instrumentation and will automatically inject the sidecar or instrumentation code for any service with matching annotations. Note that the instrumentation annotation includes the observe namespace before the name of the instrumentation. Also note that in the case of golang since the autoinstrumentation is eBPF based, you’ll need to add an additional annotation that indicates the path of the main executable within the container.

Partial example golang deployment with instrumentation enabled:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
      annotations:
        instrumentation.opentelemetry.io/inject-go: "observe/observe-auto-instrumentation"
        instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/usr/local/bin/sample-app"

For more detail on the available annotations, see the OpenTelemetry Operator documentation. Once your workloads have been annotated, restart them in order for the operator to inject the necessary sidecar or instrumentation code.

References