Collect Statsd Metrics using UDS

StatsD is a network daemon (originally written by Etsy) that listens for metrics (such as counters, timers, and gauges) sent over UDP & UDS (User Datagram Protocol). Its primary job is to aggregate these metrics periodically and then pass them along to a backend service for storage and visualization. Below steps outline how to configure the Observe agent to listen to a unix domain socket.

Agent Chart 0.43+

Note

To view all available Agent Chart versions, run: helm search repo observe --versions | grep observe/agent. If you’re currently using Agent Chart versions 0.38 through 0.42, please upgrade to version 0.43 or later.

1. After the Observe Agent is up and running, create a statsduds-values.yaml file.

agent:
  config:
    forwarder:
      receivers:
        statsd/uds:
          endpoint: ///var/run/obssocket/statsd.socket
          aggregation_interval: 60s 
          transport: "unixgram"
          is_monotonic_counter: true
          enable_metric_type: true
      processors:
        attributes/debug_source_statsd_metrics:
          actions:
          - action: insert
            key: debug_source
            value: statsduds     
      service:
        pipelines:
          metrics/custom-statsd:
            receivers: [statsd/uds]
            processors: [memory_limiter, k8sattributes, deltatocumulative/observe, batch, resourcedetection/cloud, resource/observe_common, attributes/debug_source_statsd_metrics]
            exporters: [prometheusremotewrite/observe]

forwarder:
  extraVolumes:
    - name: "observe-agent-deployment-config"
      configMap:
        name: "observe-agent"
        items:
          - key: "relay"
            path: "observe-agent.yaml"
        defaultMode: 420
    - name: obssocket
      hostPath:
        path: /tmp
  extraVolumeMounts:
    - name: observe-agent-deployment-config
      mountPath: /observe-agent-conf
    - name: obssocket
      mountPath: /var/run/obssocket
  lifecycleHooks:
    postStart:
      exec:
        command: ["/bin/sh", "-c", "sleep 20 && chmod o+rwx /var/run/obssocket/statsd.socket"]

2. Redeploy the Observe Agent

Run the following command to redeploy the Observe Agent in the observe namespace with the prometheusScrape configuration.

helm upgrade --reuse-values observe-agent observe/agent -n observe --values statsduds-values.yaml

3. Restart pods

Run the following commands to restart the pods with the updated configuration.

kubectl rollout restart deployment -n observe
kubectl rollout restart daemonset -n observe

Run the following command to make sure the Observe Agent has been redeployed successfully.

kubectl get pods -o wide -n observe

4. Once the Observe Agent is up and running on a Kubernetes cluster, you can configure your application running on the same Kubernetes cluster to send statsd metrics to the sock file below

/var/run/obssocket/statsd.sock

5. Explore your statsd metrics using the Metrics Explorer