Deploy the Observe Agent to a Serverless Kubernetes cluster (ex AWS EKS Fargate)

Serverless kubernetes services do not support daemonsets, so the observe-agent must be configured to avoid installing these. To accomplish this, add the following to your values.yaml file:

node:
  # Disables the node-logs-metrics daemonset.
  # This workload is currently not supported in serverless kubernetes.
  enabled: false
  forwarder:
    enabled: true

forwarder:
  # Changes the forwarder from a daemonset to a deployment
  mode: deployment
  # Sets the number of replicas for the forwarder deployment.
  # This can be adjusted based on your needs.
  replicaCount: 2

After this, you can continue sending OTLP data to the forwarder with the same service URI, eg http://observe-agent-forwarder.observe.svc.cluster.local:4318 (for OTLP/HTTP, or port 4317 for OTLP/gRPC).

EKS Fargate Pod Metrics

For EKS Fargate, we support using an OpenTelemetry Operator to install a sidecar container in your application pods that will collect your node and pod metrics data. To do so:

1. To run observe on fargate:

   # fill in your cluster name and region
   eksctl create fargateprofile \
   --cluster demo-fargate-cluster \
   --name observe-profile \
   --namespace observe \
   --region us-east-2
   

2. Install the opentelemetry-operator helm chart

   helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
   --set "manager.collectorImage.repository=ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s" \
   --set admissionWebhooks.certManager.enabled=false \
   --set admissionWebhooks.autoGenerateCert.enabled=true \
   --namespace observe
   

3. Wait for the new pods to be running and ready: (Run kubectl get pods -n observe, and you should see a pod named opentelemetry-operator-[hash string])

4. Add the following to your values.yaml file and install/upgrade the helm chart:

   nodeless:
    enabled: true
    hostingPlatform: fargate
    metrics:
      enabled: true
   
    # this is a map from namespaces to service accounts within that namespace. 
    # It will apply the cluster role for that namespace and serviceAccount that 
    # you would otherwise apply manually in step 5
    serviceAccounts:
      dev: ["devServiceAccount1", "devServiceAccount2"]
      production: ["productionServiceAccount1", "productionServiceAccount2"]
   

5. To grant permissions to the serviceAccounts manually, apply a cluster role to allow the sidecar to query the kubelet API

   # create a file: cluster-role.yaml
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-sidecar-role
   rules:
     - apiGroups: [""]
       resources:
         - nodes
         - nodes/proxy
         - namespaces
         - pods
       verbs: ["get", "list", "watch"]
   
     - apiGroups: ["apps"]
       resources:
         - replicasets
       verbs: ["get", "list", "watch"]
   
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-sidecar-role-binding
   subjects:
   - kind: ServiceAccount
     name: [your service account]
     namespace: [namespace to monitor]
   roleRef:
     kind: ClusterRole
     name: otel-sidecar-role
     apiGroup: rbac.authorization.k8s.io
   

   Then, run: kubectl apply -f cluster-role.yaml to apply the changes.

6. Add "sidecar.opentelemetry.io/inject": "observe/fargate-collector" as an annotation to all deployments whose pods you wish to monitor. To quickly do this for all deployments in a namespace, run:

   for d in $(kubectl get deployments -n $TARGET_NAMESPACE -o name); do
     kubectl patch $d -n $1 --type='merge' -p '{"spec": {"template": {"metadata": {"annotations": {"sidecar.opentelemetry.io/inject": "observe/fargate-collector"}}}}}'
   done
   

   This should force a rolling restart of pods in that namespace, which is necessary for the operator to inject a sidecar into the application pods. To do so manually, you can run kubectl -n [namespace with pods to monitor] rollout restart deploy