Kubernetes Integration

Kubernetes, also called K8s, is an open source system for automating deployment, scaling, and management of containerized applications. K8s groups containers that make up an application into logical units for management and discovery.

Observe provides a manifest to collect telemetry within a Kubernetes cluster. By omission, Observe gathers all events, logs, and metrics within a cluster using open source collectors.

Installation

You can install Kubernetes using the Kubernetes app located on the Apps page. The app allows you to easily create a data stream token for Kubernetes and provides a seamless experience for the installation.

Installing Kubernetes using the Observe app

Figure 1 - The Observe Kubernetes app

Click Create connection to create a token for the data stream and configure the app.

If you prefer to install Kubernetes manually, use the following installation instructions.

Important

To proceed with this step, you need an Observe customer ID and token.

Observe provides a manifest that installs all the necessary components for collecting telemetry data from Kubernetes. This manifest can be retrieved directly from https://github.com/observeinc/manifests.

The installation process consists of two steps:

$ kubectl apply -k https://github.com/observeinc/manifests/stack && \
	kubectl -n observe create secret generic credentials \
	--from-literal=OBSERVE_CUSTOMER=${OBSERVE_CUSTOMER?} \
	--from-literal=OBSERVE_TOKEN=${OBSERVE_TOKEN?}

Observe provides this example for illustrative purposes only. In production environments, Observe recommends downloading the manifest separately and tracking changes over time using your configuration management tool of choice. You can use kubectl kustomize <URL> to generate a static version of the Observe manifest.

By omission, the Observe manifest creates an observe namespace that contains all of the collection infrastructures. Only then can Observe create a secret containing the appropriate credentials for sending data to Observe.

Once you apply your manifest, wait for all pods within the namespace to be ready:

$ kubectl wait --timeout=60s pods -n observe --for=condition=Ready --all

If you monitor multiple clusters, provide a friendly name for each one. You may attach an identifier by providing a observeinc.com/cluster-name annotation:

$ kubectl annotate namespace observe observeinc.com/cluster-name="My Cluster"

Note

If you want to link to the AWS app set observeinc.com/cluster-name equal to your clusters arn - ex - kubectl annotate namespace observe observeinc.com/cluster-name=”arn:aws:eks:us-west-2:[ACCOUNT]:cluster/[CLUSTER_NAME]

If you want to link to the GCP app set observeinc.com/cluster-name equal to your clusters asset inventory name - ex - kubectl annotate namespace observe observeinc.com/cluster-name=” //container.googleapis.com/projects/[PROJECT_ID]/locations/[REGION]/clusters/[CLUSTER_NAME]”

Integrating with Observe AWS and GCP apps

If you also use Kubernetes with Amazon Elastic Kubernetes Service (EKS) or Google Kubernetes Engine (GKE) for GCP, send data from these services to the Observe Kubernetes app by configuring them in the Configuration tab.

  1. In the Kubernetes app, click the Configuration tab to open the app settings.

  2. Under AWS App, select AWS in the dropdown.

  3. Under GCP App, select GCP.

  4. Click Update to save your changes.

Updating and Uninstalling

To completely remove the Observe integration, delete all of the resources included in the original installation manifest:

$ kubectl delete -k https://github.com/observeinc/manifests/stack

Manifest Versioning

Observe uses versioned manifests. To install or uninstall a specific version, add the version parameter to the URL:

$ kubectl apply -k 'https://github.com/observeinc/manifests/stack?ref=v0.5.0'
$ kubectl delete -k 'https://github.com/observeinc/manifests/stack?ref=v0.5.0'

You can find the list of published versions on the releases page.

Metrics discovery

If you want to expose Prometheus metrics from your pod, you have two options:

  • Set the port in the annotations. This method has the disadvantage that it can only surface one HTTP endpoint for collection.

     annotations:
       prometheus.io/port: "9999"
    
  • Expose the metrics endpoint through a port with a name ending with metrics. For example:

    ports:
    - containerPort: 9999
      name: metrics
    - containerPort: 12345
      name: sidecar-metrics
    

You can use the following annotations to further influence the discovery process:

  • prometheus.io/scrape: if set to false, pod will be ignored

  • prometheus.io/scheme: set to http or https

  • prometheus.io/path: defaults to /metrics

AWS Fargate support

Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes service that can be configured to run on AWS Fargate. AWS does not support Daemonsets when running on a serverless compute engine such as AWS Fargate. Since the Observe standard log collection method relies on daemonsets, you must add the necessary configuration to collect logs for pods scheduled on Fargate.

AWS provides a method of streaming logs from pods running on Fargate. This requires setting up a Kinesis Firehose that sends data to Observe. Observe provides a Terraform module that automates this setup here. If you do not use Terraform and want to set up EKS log forwarding for Fargate, please contact support.

Migrating from legacy manifests

Observe hosts legacy manifests at https://api.observeinc.com/v1/kubernetes/manifest. As of March 2022, these manifests receive only critical security fixes. To migrate to the latest manifest:

  • Delete all existing Observe resources:

$ kubectl delete -f https://api.observeinc.com/v1/kubernetes/manifest
  • Install the latest manifest:

$ kubectl apply -k https://github.com/observeinc/manifests/stack && \
	kubectl -n observe create secret generic credentials \
	--from-literal=OBSERVE_CUSTOMER=${OBSERVE_CUSTOMER?} \
	--from-literal=OBSERVE_TOKEN=${OBSERVE_TOKEN?}

You need to add a few structural changes between the legacy and current manifests:

  • Observe removed the proxy deployment. If you used proxying custom telemetry in your cluster, please reach out to support.

  • Observe now collects metrics through Grafana Agent rather than Telegraf. The kubernetes/Metrics dataset, which contains Telegraf data, was deprecated in favor of kubernetes/Pod Metrics and kubernetes/Cadvisor Metrics. If you have custom datasets or monitors built from the former, please reach out to support for assistance.

FAQ

What Kubernetes versions does Observe support?

Kubernetes targets a minor release every 3 months and maintains a concept of a “support window” which spans one year from the original release date. This typically results in support for four minor versions at any given time.

The Observe support policy provides the following:

To find the latest supported version, go to GitHub.

What container runtimes does Observe support?

The container runtime only affects log collection. Observe validated the current Fluent Bit configuration on both Docker and containerd. Other runtimes or older versions may require minor configuration adjustments - please reach out to support for assistance.

Can I restrict a collection to a specific namespace?

Observe does not currently support this. Collecting Kubernetes state, in particular, requires accessing resources that are not namespaced, such as node and persistent volume. Log collection may be restricted to specific namespaces, however - see the next question for details.

How can I update the credentials used?

You should first regenerate the credentials secret. The simplest way of achieving this is to delete the existing secret and create a new one:

$ kubectl delete -n observe secret credentials --ignore-not-found && \
	kubectl -n observe create secret generic credentials \
	--from-literal=OBSERVE_CUSTOMER=${OBSERVE_CUSTOMER?} \
	--from-literal=OBSERVE_TOKEN=${OBSERVE_TOKEN?}

Now that you have new credentials, you need to restart pods in the observe namespace to pick up the new values:

$ kubectl rollout restart -n observe daemonset && \
	kubectl rollout restart -n observe deployment

Can I filter the logs sent to Observe?

Observe currently supports filtering on log files pulled using Fluent Bit with the inclusion of a fluent-bit-extra.conf file alongside the fluent-bit.conf file deployed via our manifest. An example manifest to filter out logs from the “namespacePattern” namespace:

apiVersion: v1
data:
  fluent-bit-extra.conf: |-
    [FILTER]
        Name         grep
        Match        k8slogs
        Exclude      namespace /.*namespacePattern.*/

Depending on the shape of your data, you may be able to filter on additional keys for greater specificity. See the Fluent Bit grep filter documentation for more information.

How can I disable scheduling a daemonset pod on a specific node?

In order to provide full coverage of your cluster, the observe-logs daemonset is by design scheduled onto all nodes. If you want to remove it from a subset of nodes, you can add a taint:

$ kubectl taint nodes ${NODENAME?} observeinc.com/unschedulable

This taint is only verified during scheduling. If an observe-logs pod is already running on the node, you must delete it manually:

$ kubectl delete pods -n observe -l name=observe-logs --field-selector=spec.nodeName=${NODENAME?}

What does a “mem buf overlimit” warning mean?

When reviewing fluent-bit logs, you may encounter a warning similar to the following example:

[2022/03/14 16:00:18] [ warn] [input] tail.0 paused (mem buf overlimit)

This indicates that fluent-bit temporarily stopped reading logs from the files on the disk, and waiting for existing buffered data to be successfully uploaded. Typically you don’t need to take any action, and fluent-bit does not send a message once fluent-bit restarts tailing of logs.

If you see this message very frequently, you may have a very bursty log source or limited upload bandwidth. In either case, please reach out to support for assistance.