Collect annotations and labels

You can collect annotations and labels by leveraging the Kubernetes Attributes Processor. For example, if the pod my-otel-demo-adservice-86f8b6d9c6-l4hvh has the following labels and annotations:

kubectl describe pod my-otel-demo-adservice-86f8b6d9c6-l4hvh

Name: my-otel-demo-adservice-86f8b6d9c6-l4hvh

Namespace: default

Priority: 0

Service Account: my-otel-demo

Node: gke-cluster-5-default-pool-d1b464b0-4v86/10.128.0.15

Start Time: Wed, 29 Jan 2025 03:08:33 +0000

Labels: app.kubernetes.io/component=adservice

app.kubernetes.io/instance=my-otel-demo

app.kubernetes.io/name=my-otel-demo-adservice

opentelemetry.io/name=my-otel-demo-adservice

pod-template-hash=86f8b6d9c6

Annotations: example.com/maintainer: teamA

...

📘
Note
To view all available Agent Chart versions, run: helm search repo observe --versions | grep observe/agent. If you’re currently using Agent Chart versions 0.38 through 0.40, upgrade to version 0.41 or later.

Create a file named k8sattributes-values.yaml with the following contents:

agent:
  config:
    # 1) Define an anchor for the repeated configuration (must appear before you reference it).
    k8sattributes_custom_base: &k8sattributes_custom_base
      passthrough: false
      extract:
        labels:
          - tag_name: app.kubernetes.io/component
            key: app.kubernetes.io/component
            from: pod
          - tag_name: app.kubernetes.io/instance
            key: app.kubernetes.io/instance
            from: pod
          - tag_name: app.kubernetes.io/name
            key: app.kubernetes.io/name
            from: pod
          - tag_name: opentelemetry.io/name
            key: opentelemetry.io/name
            from: pod
          - tag_name: pod-template-hash
            key: pod-template-hash
            from: pod
        annotations:
          - tag_name: example.com/maintainer
            key: example.com/maintainer
            from: pod
      pod_association:
        - sources:
            - from: resource_attribute
              name: k8s.pod.ip
        - sources:
            - from: resource_attribute
              name: k8s.pod.uid
        - sources:
            - from: connection

    nodeLogsMetrics:
      processors:
        # 2) Reference the anchor
        k8sattributes/custom: *k8sattributes_custom_base
        #
        # (Any other processors remain the same)
        #
      service:
        pipelines:
          logs:
            processors: [memory_limiter, k8sattributes, batch, resourcedetection/cloud, resource/observe_common, attributes/debug_source_pod_logs, k8sattributes/custom]
          metrics/hostmetrics:
            processors: [memory_limiter, k8sattributes, batch, resourcedetection/cloud, resource/observe_common, attributes/debug_source_hostmetrics, k8sattributes/custom]
          metrics/kubeletstats:
            processors: [memory_limiter, k8sattributes, batch, resourcedetection/cloud, resource/observe_common, attributes/debug_source_kubeletstats_metrics, k8sattributes/custom]

    clusterEvents:
      processors:
        k8sattributes/custom: *k8sattributes_custom_base
      service:
        pipelines:
          logs/cluster:
            processors: [memory_limiter, batch, resource/observe_common, filter/cluster, transform/cluster, k8sattributes/custom]
          logs/objects:
            processors: [memory_limiter, batch, resource/observe_common, transform/unify, observek8sattributes, transform/object, k8sattributes/custom]

    clusterMetrics:
      processors:
        k8sattributes/custom: *k8sattributes_custom_base
      service:
        pipelines:
          metrics/pod_metrics:
            processors: [memory_limiter, k8sattributes, batch, resource/observe_common, attributes/debug_source_pod_metrics, k8sattributes/custom]
          metrics:
            processors: [memory_limiter, k8sattributes, batch, resource/observe_common, attributes/debug_source_cluster_metrics, k8sattributes/custom]

    monitor:
      processors:
        k8sattributes/custom: *k8sattributes_custom_base
      service:
        pipelines:
          metrics:
            processors: [memory_limiter, k8sattributes, batch, resource/observe_common, attributes/debug_source_agent_monitor, k8sattributes/custom]

    forwarder:
      processors:
        k8sattributes/custom: *k8sattributes_custom_base
      service:
        pipelines:
          traces/observe-forward:
            processors: [memory_limiter, k8sattributes, batch, resourcedetection/cloud, resource/observe_common, attributes/debug_source_app_traces, k8sattributes/custom]
          logs/observe-forward:
            processors: [memory_limiter, k8sattributes, batch, resourcedetection/cloud, resource/observe_common, attributes/debug_source_app_logs, k8sattributes/custom]
          metrics/observe-forward:
            processors: [memory_limiter, k8sattributes, batch, resourcedetection/cloud, resource/observe_common, attributes/debug_source_app_metrics, k8sattributes/custom]

Redeploy the Observe Agent with the updated config.

helm upgrade --reuse-values observe-agent observe/agent -n observe --values k8sattributes-values.yaml

Restart the pods.

kubectl rollout restart deployment -n observe
kubectl rollout restart daemonset -n observe