Kubernetes data collection and agent interface
The Observe Agent Helm Chart scrapes and produces some data by default in addition to being able to ingest various other user-provided data. This document will describe the schemas of the data available by default when installing the Agent Helm Chart but does not represent a comprehensive list of all data that might be present after installing the Helm Chart.
Metrics
We gather metrics through multiple components including the hostmetrics receiver, the k8scluster receiver, the kubeletstats receiver, and the prometheus receiver. For each of these receivers, we configure a subset of metrics to gather from the overall list that is available. The metrics are gathered into two datasets, one representing the metric point and another representing the metric metadata such as unit, description, etc.
Prometheus Metrics Schema
Prometheus Metrics Schema| Column Name | Column Type | Description |
|---|---|---|
| timestamp | datetime | Timestamp of metric |
| metric | string | Metric name |
| value | double | Metric Value |
| labels | JSON | Labels associated with metric |
| meta | JSON | Ingestion related metadata of metric |
Prometheus Metadata Schema
Prometheus Metadata Schema| Column Name | Column Type | Description |
|---|---|---|
| timestamp | datetime | Timestamp of metric metadata |
| metric | string | Metric name |
| metric_type | string | Metric type |
| metric_unit | string | Metric unit |
| metric_description | string | Metric description |
| meta | JSON | Ingestion related metadata of metric |
hostmetrics receiver
hostmetrics receiverList of metrics produced:
- system_filesystem_usage_bytes
- system_disk_operation_time_seconds_total
- system_disk_io_bytes_total
- system_disk_merged_total
- system_disk_operations_total
- system_filesystem_inodes_usage
- system_cpu_time_seconds_total
- system_disk_weighted_io_time_seconds_total
- system_disk_pending_operations
- system_disk_io_time_seconds_total
- system_network_connections
- system_memory_usage_bytes
- system_network_errors_total
- system_network_packets_total
- system_network_dropped_total
- system_network_io_bytes_total
- system_cpu_load_average_1m
- system_cpu_load_average_15m
- system_cpu_load_average_5m
Labels
| Name | Description | Type | Required |
|---|---|---|---|
| k8s_cluster_name | Custom name for cluster provided from config/env var | String | No |
| k8s_cluster_uid | Custom uid for cluster provided from config/env var | String | No |
kubeletstats receiver
kubeletstats receiverThis receiver collects metrics by querying the kubelet’s metrics API on each node. For metrics collected through this receiver, the details including description, unit, and type for each metric can be found in the documentation of this receiver. Note that since Observe exports the metrics in Prometheus format, the unit if available is appended to the end of the metric name and the period delimiter is converted to underscore. So for example container_cpu_time_seconds_total appears in the component documentation as container.cpu.time. This document will specify the exact metric name that appears in Observe.
List of metrics produced:
- container_cpu_time_seconds_total
- container_filesystem_available_bytes
- container_filesystem_capacity_bytes
- container_filesystem_usage_bytes
- container_memory_available_bytes
- container_memory_major_page_faults_ratio
- container_memory_page_faults_ratio
- container_memory_rss_bytes
- container_memory_usage_bytes
- container_memory_working_set_bytes
- k8s_node_cpu_time_seconds_total
- k8s_node_filesystem_available_bytes
- k8s_node_filesystem_capacity_bytes
- k8s_node_filesystem_usage_bytes
- k8s_node_memory_available_bytes
- k8s_node_memory_major_page_faults_ratio
- k8s_node_memory_page_faults_ratio
- k8s_node_memory_rss_bytes
- k8s_node_memory_usage_bytes
- k8s_node_memory_working_set_bytes
- k8s_node_network_errors_total
- k8s_node_network_io_bytes_total
- k8s_pod_cpu_time_seconds_total
- k8s_pod_filesystem_available_bytes
- k8s_pod_filesystem_capacity_bytes
- k8s_pod_filesystem_usage_bytes
- k8s_pod_memory_available_bytes
- k8s_pod_memory_major_page_faults_ratio
- k8s_pod_memory_page_faults_ratio
- k8s_pod_memory_rss_bytes
- k8s_pod_memory_usage_bytes
- k8s_pod_memory_working_set_bytes
- k8s_pod_network_errors_total
- k8s_pod_network_io_bytes_total
- container_cpu_usage
- container_uptime_seconds_total
- k8s_container_cpu_node_utilization_ratio
- k8s_container_cpu_limit_utilization_ratio
- k8s_container_cpu_request_utilization_ratio
- k8s_container_memory_node_utilization_ratio
- k8s_container_memory_limit_utilization_ratio
- k8s_container_memory_request_utilization_ratio
- k8s_node_cpu_usage
- k8s_node_uptime_seconds_total
- k8s_pod_cpu_node_utilization_ratio
- k8s_pod_cpu_usage
- k8s_pod_cpu_limit_utilization_ratio
- k8s_pod_cpu_request_utilization_ratio
- k8s_pod_memory_node_utilization_ratio
- k8s_pod_memory_limit_utilization_ratio
- k8s_pod_memory_request_utilization_ratio
- k8s_pod_uptime_seconds_total
Labels
| Name | Description | Type | Required |
|---|---|---|---|
| k8s_cluster_name | Custom name for cluster provided from config/env var | String | No |
| k8s_cluster_uid | Custom uid for cluster provided from config/env var | String | No |
| k8s_namespace_name | Name of parent namespace | String | No |
| k8s_deployment_name | Name of related deployment | String | No |
| k8s_replicaset_name | Name of related replicaset | String | No |
| k8s_statefulset_name | Name of related statefulset | String | No |
| k8s_daemonset_name | Name of related daemonset | String | No |
| k8s_cronjob_name | Name of related cronjob | String | No |
| k8s_job_name | Name of related job | String | No |
| k8s_node_name | Name of node | String | No |
| k8s_node_uid | Uid of node | String | No |
| k8s_pod_name | Name of pod | String | No |
| k8s_pod_uid | Uid of pod | String | No |
k8scluster receiver
k8scluster receiverThis receiver collects cluster level metrics from the Kubernetes API server. These are scraped on an interval from the cluster API server. For metrics collected through this receiver, the details including description, unit, and type for each metric can be found in the documentation of this receiver. Note that since Observe exports the metrics in Prometheus format, the unit if available is appended to the end of the metric name and the period delimiter is converted to underscore. So for example k8s_container_cpu_limit appears in the component documentation as k8s.container.cpu_limit. This document will specify the exact metric name that appears in Observe.
List of metrics produced:
- k8s_container_cpu_limit
- k8s_container_cpu_request
- k8s_container_memory_limit_bytes
- k8s_container_memory_request_bytes
- k8s_container_storage_limit_bytes
- k8s_container_storage_request_bytes
- k8s_container_ephemeralstorage_limit_bytes
- k8s_container_ephemeralstorage_request_bytes
- k8s_container_ready
- k8s_container_restarts
- k8s_cronjob_active_jobs
- k8s_daemonset_current_scheduled_nodes
- k8s_daemonset_desired_scheduled_nodes
- k8s_daemonset_misscheduled_nodes
- k8s_daemonset_ready_nodes
- k8s_deployment_available
- k8s_deployment_desired
- k8s_hpa_current_replicas
- k8s_hpa_desired_replicas
- k8s_hpa_max_replicas
- k8s_hpa_min_replicas
- k8s_job_active_pods
- k8s_job_desired_successful_pods
- k8s_job_failed_pods
- k8s_job_max_parallel_pods
- k8s_job_successful_pods
- k8s_namespace_phase_cores
- k8s_pod_phase
- k8s_replicaset_available
- k8s_replicaset_desired
- k8s_replication_controller_available
- k8s_replication_controller_desired
- k8s_statefulset_current_pods
- k8s_statefulset_desired_pods
- k8s_statefulset_ready_pods
- k8s_statefulset_updated_pods
- k8s_node_condition
- k8s_node_condition_disk_pressure
- k8s_node_condition_memory_pressure
- k8s_node_condition_ready
Labels
| Name | Description | Type | Required |
|---|---|---|---|
| k8s_cluster_name | Custom name for cluster provided from config/env var | String | No |
| k8s_cluster_uid | Custom uid for cluster provided from config/env var | String | No |
| k8s_namespace_name | Name of parent namespace | String | No |
| k8s_deployment_name | Name of related deployment | String | No |
| k8s_deployment_uid | Uid of related deployment | String | No |
| k8s_daemonset_name | Name of related deployment | String | No |
| k8s_daemonset_uid | Uid of related deployment | String | No |
| k8s_node_name | Name of node | String | No |
| k8s_node_uid | Uid of node | String | No |
| k8s_container_name | Name of related container | String | No |
| k8s_pod_name | Name of related pod | String | No |
| k8s_pod_uid | Uid of related pod | String | No |
| k8s_hpa_name | Name of related hpa | String | No |
| k8s_hpa_uid | Uid of related hpa | String | No |
| k8s_cronjob_name | Name of related hpa | String | No |
| k8s_cronjob_uid | Uid of related hpa | String | No |
| k8s_job_name | Name of related hpa | String | No |
| k8s_job_uid | Uid of related hpa | String | No |
| k8s_replicaset_name | Name of related replicaset | String | No |
| k8s_replicaset_uid | Uid of related replicaset | String | No |
| k8s_statefulset_name | Name of related statefulset | String | No |
| k8s_statefulset_uid | Uid of related replicaset | String | No |
| k8s_replicationcontroller_name | Name of related replication controller | String | No |
| k8s_replicationcontroller_uid | Uid of related replicaset | String | No |
| condition | Condition of node | String | No |
| service_name | Name of the service | String | No |
| app_kubernetes_io_name | Name of the application | String | No |
| app_kubernetes_io_instance | A unique name identifying the instance of an application | String | No |
| app_kubernetes_io_version | Current version of the application | String | No |
| app_kubernetes_io_component | The component within the application architecture | String | No |
| app_kubernetes_io_part-of | Name of a higher-level application this object is part of | String | No |
| app_kubernetes_io_managed-by | The tool being used to manage the operation of an application | String | No |
prometheus receiver
prometheus receiverThis receiver scrapes various prometheus endpoints and gathers metrics from them. By default it scrapes the Observe Agent’s own metrics. The full list of metrics depends on the detail level configured for the agent self-telemetry and is available in the documentation.
Labels
| Name | Description | Type | Required |
|---|---|---|---|
| k8s_namespace_name | Name of parent namespace | String | No |
| k8s_pod_name | Name of related pod | String | No |
| k8s_pod_uid | Uid of related pod | String | No |
| k8s_deployment_name | Name of related deployment | String | No |
| k8s_replicaset_name | Name of related replicaset | String | No |
| k8s_statefulset_name | Name of related statefulset | String | No |
| k8s_daemonset_name | Name of related daemonset | String | No |
| k8s_cronjob_name | Name of related cronjob | String | No |
| k8s_job_name | Name of related job | String | No |
| k8s_cluster_uid | Uid of the cluster | String | No |
| k8s_node_name | Name of node | String | No |
| k8s_node_uid | Uid of node | String | No |
| k8s_container_name | Name of related container | String | No |
| container_id | Uid of related container | String | No |
Logs
The Observe Kubernetes Agent Helm Chart can scrape container logs from the nodes in the cluster. These logs are ingested to the OpenTelemetry Logs dataset.
OpenTelemetry Logs Schema
OpenTelemetry Logs Schema| Column Name | Column Type | Description |
|---|---|---|
| timestamp | datetime | Timestamp of log |
| body | string | Log body |
| attributes | JSON | Attributes associated with metric |
| resource_attributes | JSON | Resource attributes associated with metric. Kubernetes entity metadata will be located as attributes here. |
| instrumentation_scope | JSON | |
| fields | JSON | |
| meta | JSON | Ingestion related metadata of metric |
Attributes
| Name | Description | Type | Required |
|---|---|---|---|
| log.file.path | Path of log file | string |
Resource Attributes
| Name | Description | Type | Required |
|---|---|---|---|
| k8s_namespace_name | Name of parent namespace | String | No |
| k8s_pod_name | Name of related pod | String | No |
| k8s_pod_uid | Uid of related pod | String | No |
| k8s_deployment_name | Name of related deployment | String | No |
| k8s_replicaset_name | Name of related replicaset | String | No |
| k8s_statefulset_name | Name of related statefulset | String | No |
| k8s_daemonset_name | Name of related daemonset | String | No |
| k8s_cronjob_name | Name of related cronjob | String | No |
| k8s_job_name | Name of related job | String | No |
| k8s_cluster_uid | Uid of the cluster | String | No |
| k8s_node_name | Name of node | String | No |
| k8s_node_uid | Uid of node | String | No |
| k8s_container_name | Name of related container | String | No |
| container_id | Uid of related container | String | No |
Updated 24 days ago