Install Observe Agent on Kubernetes

Note

The Observe Agent on Kubernetes is designed to deliver the best-in-class user experience with Kubernetes Explorer, which is currently in private preview. For more details on Kubernetes Explorer, use the Contact Support button within the product. Navigate to Docs & Support on the left menu, select Contact Support, and click Send Us a Message to reach an Observe Data Engineer.

You may follow the instructions on this page to deploy the Observe Agent on Kubernetes, allowing you to gather raw events, logs, and metrics from Kubernetes.

If you’re interested in using the built-in dashboards, please refer to the Kubernetes App.

Warning

Amazon EKS Auto Mode, Azure Kubernetes Service (AKS) Automatic, or GKE Autopilot is not supported yet.

Add the Observe Helm repository

Run the following commands to add the Observe Helm repository to your local Helm setup, allowing you to access and deploy Observe’s charts.

helm repo add observe https://observeinc.github.io/helm-charts
helm repo update

Deploy the Observe Agent

Run the following commands to create a Kubernetes namespace called observe and generate a secret. Replace ${OBSERVE_TOKEN} with your instance’s token (ex: a1b2c3d4e5f6g7h8i9k0:l1m2n3o4p5q6r7s8t9u0v1w2x3y4z5a6).

kubectl create namespace observe
  kubectl -n observe create secret generic agent-credentials --from-literal=OBSERVE_TOKEN=${OBSERVE_TOKEN} 

kubectl annotate secret agent-credentials -n observe \
  meta.helm.sh/release-name=observe-agent \
  meta.helm.sh/release-namespace=observe

kubectl label secret agent-credentials -n observe \
  app.kubernetes.io/managed-by=Helm

Recommended

Run the following command to deploy the Observe Agent in the observe namespace using observe-agent-monitored-cluster as the name for your Kubernetes cluster. Replace ${OBSERVE_COLLECTION_ENDPOINT} with your instance’s collection endpoint (ex: https://123456789012.collect.observeinc.com/). For more information on configuring the agent, see values.yaml.

helm install observe-agent observe/agent -n observe \
--set observe.token.create="false" \
--set observe.collectionEndpoint.value="${OBSERVE_COLLECTION_ENDPOINT}" \
--set cluster.name="observe-agent-monitored-cluster" \
--set cluster.events.enabled="true" \
--set cluster.metrics.enabled="true" \
--set node.containers.logs.enabled="true" \
--set node.containers.metrics.enabled="true" \
--set application.prometheusScrape.enabled="false" \
--set agent.selfMonitor.enabled="true"

Run the following command to make sure the Observe Agent has been deployed successfully.

kubectl get pods -n observe

Run the following command to save the current configuration values to a file for future configuration adjustments and upgrades.

helm -n observe get values observe-agent -o yaml > observe-agent-values.yaml

Installation with a values file

Create observe-agent-values.yaml with the following configuration. Replace ${OBSERVE_COLLECTION_ENDPOINT} with your instance’s collection endpoint (ex: https://123456789012.collect.observeinc.com/). For more information on configuring the agent, see values.yaml.

agent:
  selfMonitor:
    enabled: true
application:
  prometheusScrape:
    enabled: false
cluster:
  events:
    enabled: true
  metrics:
    enabled: true
  name: observe-agent-monitored-cluster
node:
  containers:
    logs:
      enabled: true
    metrics:
      enabled: true
observe:
  collectionEndpoint:
    value: ${OBSERVE_COLLECTION_ENDPOINT}
  token:
    create: false

Run the following command to deploy the Observe Agent in the observe namespace

helm install observe-agent observe/agent -n observe --values observe-agent-values.yaml

Run the following command to make sure the Observe Agent has been deployed successfully.

kubectl get pods -n observe

Configure application instrumentation

Follow these steps to direct your application’s trace data to Observe:

1. Install the OpenTelemetry App if you have not already done so.

2. Create a token associated with OpenTelemetry App’s datastream.

3. Create the trace-values.yaml file with the following config. Replace <YOUR ANOTHER OBSERVE TOKEN> (your instance’s token) and <YOUR OBSERVE COLLECTION ENDPOINT> (your instance’s collection endpoint) on each host.

agent:
  config:
    nodeLogsMetrics:
      # This config specifies both a new receiver and exporter and sets up a new traces pipeline with them
      receivers:
        otlp/app-telemetry:
          protocols:
            grpc:
              endpoint: ${env:MY_POD_IP}:4317
            http:
              endpoint: ${env:MY_POD_IP}:4318
      processors:
        attributes/debug_source_app_traces:
          actions:
          - action: insert
            key: debug_source
            value: app_traces
        attributes/debug_source_app_logs:
          actions:
          - action: insert
            key: debug_source
            value: app_logs
        attributes/debug_source_app_metrics:
          actions:
          - action: insert
            key: debug_source
            value: app_metrics
      exporters:
        otlphttp/observe-traces:
          # (ex: https://123456789012.collect.observeinc.com/v2/otel)
          endpoint: "<YOUR OBSERVE COLLECTION ENDPOINT>/v2/otel"
          headers:
            # (ex: Bearer a1b2c3d4e5f6g7h8i9k0:l1m2n3o4p5q6r7s8t9u0v1w2x3y4z5a6)
            authorization: "Bearer <YOUR ANOTHER OBSERVE TOKEN>"
      service:
        pipelines:
          traces/observe-forward:
            receivers: [otlp/app-telemetry]
            processors: [memory_limiter, k8sattributes, batch, resourcedetection/cloud, resource/observe_common, attributes/debug_source_app_traces]
            exporters: [otlphttp/observe-traces]
          logs/observe-forward:
            receivers: [otlp/app-telemetry]
            processors: [memory_limiter, k8sattributes, batch, resourcedetection/cloud, resource/observe_common, attributes/debug_source_app_logs]
            exporters: [otlphttp/observe/base]
          metrics/observe-forward:
            receivers: [otlp/app-telemetry]
            processors: [memory_limiter, k8sattributes, batch, resourcedetection/cloud, resource/observe_common, attributes/debug_source_app_metrics]
            exporters: [prometheusremotewrite]
node-logs-metrics:
  service:
    enabled: true
    type: ClusterIP
  networkPolicy:
    enabled: true

5. Redeploy the Observe Agent with the updated config.

helm upgrade --reuse-values observe-agent observe/agent -n observe --values trace-values.yaml

6. Restart the pods.

kubectl rollout restart deployment -n observe
kubectl rollout restart daemonset -n observe

6. Once the Observe Agent is up and running on a Kubernetes cluster, you can configure your application running on the same Kubernetes cluster to send telemetry data to the Observe Agent using one of the following addresses:

• OTLP/HTTP endpoint: http://observe-agent-node-logs-metrics.observe.svc.cluster.local:4318

• OTLP/grpc endpoint: http://observe-agent-node-logs-metrics.observe.svc.cluster.local:4317

For example, if you are using the OpenTelemetry Astronomy Shop Demo app, create the app.yaml.

default:
  envOverrides:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: 'http://observe-agent-node-logs-metrics.observe.svc.cluster.local:4318'

Upgrade the helm chart for the OpenTelemetry Astronomy Shop Demo app.

helm upgrade --reuse-values -f app.yaml my-otel-demo open-telemetry/opentelemetry-demo

7. Explore your trace data using the Trace Explorer and the Service Explorer

Uninstall the Observe Agent

You can uninstall the agent from your Kubernetes cluster.

helm uninstall observe-agent -n observe
kubectl -n observe delete secret agent-credentials
kubectl delete namespace observe

• Node Affinity, Taints, and Tolerations
• Filter Logs or Metrics
• Prometheus Autodiscovery
• Tuning Service Resource Requests and Limits