Kubernetes Data Collection and Agent Interface

The Observe Agent Helm Chart scrapes and produces some data by default in addition to being able to ingest various other user-provided data. This document will describe the schemas of the data available by default when installing the Agent Helm Chart but does not represent a comprehensive list of all data that might be present after installing the Helm Chart.

Metrics

We gather metrics through multiple components including the hostmetrics receiver, the k8scluster receiver, the kubeletstats receiver, and the prometheus receiver. For each of these receivers, we configure a subset of metrics to gather from the overall list that is available. The metrics are gathered into two datasets, one representing the metric point and another representing the metric metadata such as unit, description, etc.

Prometheus Metrics Schema

Column Name	Column Type	Description
timestamp	datetime	Timestamp of metric
metric	string	Metric name
value	double	Metric Value
labels	JSON	Labels associated with metric
meta	JSON	Ingestion related metadata of metric

Prometheus Metadata Schema

Column Name	Column Type	Description
timestamp	datetime	Timestamp of metric metadata
metric	string	Metric name
metric_type	string	Metric type
metric_unit	string	Metric unit
metric_description	string	Metric description
meta	JSON	Ingestion related metadata of metric

hostmetrics receiver

List of metrics produced:

• system_filesystem_usage_bytes

• system_disk_operation_time_seconds_total

• system_disk_io_bytes_total

• system_disk_merged_total

• system_disk_operations_total

• system_filesystem_inodes_usage

• system_cpu_time_seconds_total

• system_disk_weighted_io_time_seconds_total

• system_disk_pending_operations

• system_disk_io_time_seconds_total

• system_network_connections

• system_memory_usage_bytes

• system_network_errors_total

• system_network_packets_total

• system_network_dropped_total

• system_network_io_bytes_total

• system_cpu_load_average_1m

• system_cpu_load_average_15m

• system_cpu_load_average_5m

Labels

Name	Description	Type	Required
k8s_cluster_name	Custom name for cluster provided from config/env var	String	No
k8s_cluster_uid	Custom uid for cluster provided from config/env var	String	No

kubeletstats receiver

This receiver collects metrics by querying the kubelet’s metrics API on each node. For metrics collected through this receiver, the details including description, unit, and type for each metric can be found in the documentation of this receiver. Note that since Observe exports the metrics in Prometheus format, the unit if available is appended to the end of the metric name and the period delimiter is converted to underscore. So for example container_cpu_time_seconds_total appears in the component documentation as container.cpu.time. This document will specify the exact metric name that appears in Observe.

List of metrics produced:

• container_cpu_time_seconds_total

• container_filesystem_available_bytes

• container_filesystem_capacity_bytes

• container_filesystem_usage_bytes

• container_memory_available_bytes

• container_memory_major_page_faults_ratio

• container_memory_page_faults_ratio

• container_memory_rss_bytes

• container_memory_usage_bytes

• container_memory_working_set_bytes

• k8s_node_cpu_time_seconds_total

• k8s_node_filesystem_available_bytes

• k8s_node_filesystem_capacity_bytes

• k8s_node_filesystem_usage_bytes

• k8s_node_memory_available_bytes

• k8s_node_memory_major_page_faults_ratio

• k8s_node_memory_page_faults_ratio

• k8s_node_memory_rss_bytes

• k8s_node_memory_usage_bytes

• k8s_node_memory_working_set_bytes

• k8s_node_network_errors_total

• k8s_node_network_io_bytes_total

• k8s_pod_cpu_time_seconds_total

• k8s_pod_filesystem_available_bytes

• k8s_pod_filesystem_capacity_bytes

• k8s_pod_filesystem_usage_bytes

• k8s_pod_memory_available_bytes

• k8s_pod_memory_major_page_faults_ratio

• k8s_pod_memory_page_faults_ratio

• k8s_pod_memory_rss_bytes

• k8s_pod_memory_usage_bytes

• k8s_pod_memory_working_set_bytes

• k8s_pod_network_errors_total

• k8s_pod_network_io_bytes_total

• container_cpu_usage

• container_uptime_seconds_total

• k8s_container_cpu_node_utilization_ratio

• k8s_container_cpu_limit_utilization_ratio

• k8s_container_cpu_request_utilization_ratio

• k8s_container_memory_node_utilization_ratio

• k8s_container_memory_limit_utilization_ratio

• k8s_container_memory_request_utilization_ratio

• k8s_node_cpu_usage

• k8s_node_uptime_seconds_total

• k8s_pod_cpu_node_utilization_ratio

• k8s_pod_cpu_usage

• k8s_pod_cpu_limit_utilization_ratio

• k8s_pod_cpu_request_utilization_ratio

• k8s_pod_memory_node_utilization_ratio

• k8s_pod_memory_limit_utilization_ratio

• k8s_pod_memory_request_utilization_ratio

• k8s_pod_uptime_seconds_total

Labels

Name	Description	Type	Required
k8s_cluster_name	Custom name for cluster provided from config/env var	String	No
k8s_cluster_uid	Custom uid for cluster provided from config/env var	String	No
k8s_namespace_name	Name of parent namespace	String	No
k8s_deployment_name	Name of related deployment	String	No
k8s_replicaset_name	Name of related replicaset	String	No
k8s_statefulset_name	Name of related statefulset	String	No
k8s_daemonset_name	Name of related daemonset	String	No
k8s_cronjob_name	Name of related cronjob	String	No
k8s_job_name	Name of related job	String	No
k8s_node_name	Name of node	String	No
k8s_node_uid	Uid of node	String	No
k8s_pod_name	Name of pod	String	No
k8s_pod_uid	Uid of pod	String	No

k8scluster receiver

This receiver collects cluster level metrics from the Kubernetes API server. These are scraped on an interval from the cluster API server. For metrics collected through this receiver, the details including description, unit, and type for each metric can be found in the documentation of this receiver. Note that since Observe exports the metrics in Prometheus format, the unit if available is appended to the end of the metric name and the period delimiter is converted to underscore. So for example k8s_container_cpu_limit appears in the component documentation as k8s.container.cpu_limit. This document will specify the exact metric name that appears in Observe.

List of metrics produced:

• k8s_container_cpu_limit

• k8s_container_cpu_request

• k8s_container_memory_limit_bytes

• k8s_container_memory_request_bytes

• k8s_container_storage_limit_bytes

• k8s_container_storage_request_bytes

• k8s_container_ephemeralstorage_limit_bytes

• k8s_container_ephemeralstorage_request_bytes

• k8s_container_ready

• k8s_container_restarts

• k8s_cronjob_active_jobs

• k8s_daemonset_current_scheduled_nodes

• k8s_daemonset_desired_scheduled_nodes

• k8s_daemonset_misscheduled_nodes

• k8s_daemonset_ready_nodes

• k8s_deployment_available

• k8s_deployment_desired

• k8s_hpa_current_replicas

• k8s_hpa_desired_replicas

• k8s_hpa_max_replicas

• k8s_hpa_min_replicas

• k8s_job_active_pods

• k8s_job_desired_successful_pods

• k8s_job_failed_pods

• k8s_job_max_parallel_pods

• k8s_job_successful_pods

• k8s_namespace_phase_cores

• k8s_pod_phase

• k8s_replicaset_available

• k8s_replicaset_desired

• k8s_replication_controller_available

• k8s_replication_controller_desired

• k8s_statefulset_current_pods

• k8s_statefulset_desired_pods

• k8s_statefulset_ready_pods

• k8s_statefulset_updated_pods

• k8s_node_condition

• k8s_node_condition_disk_pressure

• k8s_node_condition_memory_pressure

• k8s_node_condition_ready

Labels

Name	Description	Type	Required
k8s_cluster_name	Custom name for cluster provided from config/env var	String	No
k8s_cluster_uid	Custom uid for cluster provided from config/env var	String	No
k8s_namespace_name	Name of parent namespace	String	No
k8s_deployment_name	Name of related deployment	String	No
k8s_deployment_uid	Uid of related deployment	String	No
k8s_daemonset_name	Name of related deployment	String	No
k8s_daemonset_uid	Uid of related deployment	String	No
k8s_node_name	Name of node	String	No
k8s_node_uid	Uid of node	String	No
k8s_container_name	Name of related container	String	No
k8s_pod_name	Name of related pod	String	No
k8s_pod_uid	Uid of related pod	String	No
k8s_hpa_name	Name of related hpa	String	No
k8s_hpa_uid	Uid of related hpa	String	No
k8s_cronjob_name	Name of related hpa	String	No
k8s_cronjob_uid	Uid of related hpa	String	No
k8s_job_name	Name of related hpa	String	No
k8s_job_uid	Uid of related hpa	String	No
k8s_replicaset_name	Name of related replicaset	String	No
k8s_replicaset_uid	Uid of related replicaset	String	No
k8s_statefulset_name	Name of related statefulset	String	No
k8s_statefulset_uid	Uid of related replicaset	String	No
k8s_replicationcontroller_name	Name of related replication controller	String	No
k8s_replicationcontroller_uid	Uid of related replicaset	String	No
condition	Condition of node	String	No

prometheus receiver

This receiver scrapes various prometheus endpoints and gathers metrics from them. By default it scrapes the Observe Agent’s own metrics. The full list of metrics depends on the detail level configured for the agent self-telemetry and is available in the documentation.

Labels

Name	Description	Type	Required
k8s_namespace_name	Name of parent namespace	String	No
k8s_pod_name	Name of related pod	String	No
k8s_pod_uid	Uid of related pod	String	No
k8s_deployment_name	Name of related deployment	String	No
k8s_replicaset_name	Name of related replicaset	String	No
k8s_statefulset_name	Name of related statefulset	String	No
k8s_daemonset_name	Name of related daemonset	String	No
k8s_cronjob_name	Name of related cronjob	String	No
k8s_job_name	Name of related job	String	No
k8s_cluster_uid	Uid of the cluster	String	No
k8s_node_name	Name of node	String	No
k8s_node_uid	Uid of node	String	No
k8s_container_name	Name of related container	String	No
container_id	Uid of related container	String	No

Logs

The Observe Kubernetes Agent Helm Chart can scrape container logs from the nodes in the cluster. These logs are ingested to the OpenTelemetry Logs dataset.

OpenTelemetry Logs Schema

Column Name	Column Type	Description
timestamp	datetime	Timestamp of log
body	string	Log body
attributes	JSON	Attributes associated with metric
resource_attributes	JSON	Resource attributes associated with metric. Kubernetes entity metadata will be located as attributes here.
instrumentation_scope	JSON
fields	JSON
meta	JSON	Ingestion related metadata of metric

Attributes

Name	Description	Type	Required
log.file.path	Path of log file	string

Resource Attributes

Name	Description	Type	Required
k8s_namespace_name	Name of parent namespace	String	No
k8s_pod_name	Name of related pod	String	No
k8s_pod_uid	Uid of related pod	String	No
k8s_deployment_name	Name of related deployment	String	No
k8s_replicaset_name	Name of related replicaset	String	No
k8s_statefulset_name	Name of related statefulset	String	No
k8s_daemonset_name	Name of related daemonset	String	No
k8s_cronjob_name	Name of related cronjob	String	No
k8s_job_name	Name of related job	String	No
k8s_cluster_uid	Uid of the cluster	String	No
k8s_node_name	Name of node	String	No
k8s_node_uid	Uid of node	String	No
k8s_container_name	Name of related container	String	No
container_id	Uid of related container	String	No