Linux Host Monitoring Integration¶
The Linux Integration ingests data using several common open source tools. See details of each host and its users, volumes, network interfaces, and more.
With this data coming into Observe, we can assist you with shaping it into datasets and boards so you can see what is happening with your Linux infrastructure. You can see this in practice in this Linux Host Monitoring blog post. Please talk to your Observe Data Engineer for more details.
What data does it ingest?¶
The Linux Integration collects data from several sources:
Fluentbit -
systemd
logsOsquery - system properties, user information, and shell history
Server - Resource datasets for hosts, volumes, interfaces, and users
Telegraf - Metrics for various subcomponents, such as CPUs, processes, and disk I/O
This data gives you status, metrics, and other details about your Linux hosts.
Using the Linux Integration data¶
Many host-related investigations start with the Host dataset, showing the major performance counters useful for troubleshooting host issues. Here are some Resources based on this data:
Server:
Host - CPU utilization, memory and disk usage, hosts per datacenter
Interface - packets and bytes sent/received, most active MAC addresses
Volume - free and used space, volumes by host
Osquery:
Logged In Users - active users, users recently added or removed
Setup¶
The Linux Integration uses osquery, Fluent Bit, and Telegraf to send logs and metrics to Observe. Once the forwarders are installed and sending data, ask us about modeling the data and creating datasets.
You will need:
Your Observe Customer ID
An ingest token - for details on creating an ingest token for a datastream, see Data streams
One or more Linux hosts to monitor
The instructions below work with the following:
Amazon Linux 2
Ubuntu 20.04 LTS
Centos 7+
Installation¶
To ingest data, install and configure the needed agents on each host.
Install Osquery, Fluent Bit, and Telegraf¶
Osquery
Install the latest version of osquery on Amazon Linux 2 with the following command:
curl -L https://pkg.osquery.io/rpm/GPG | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-osquery
sudo yum-config-manager --add-repo https://pkg.osquery.io/rpm/osquery-s3-rpm.repo
sudo yum-config-manager --enable osquery-s3-rpm-repo
sudo yum install osquery -y
sudo service osqueryd start 2>/dev/null || true
Install the latest version of osquery on Ubuntu 20.04 with the following commands:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
if ! grep -Fq https://pkg.osquery.io/deb /etc/apt/sources.list.d/osquery.list
then
sudo tee -a /etc/apt/sources.list.d/osquery.list > /dev/null <<EOT
deb [arch=amd64] https://pkg.osquery.io/deb deb main
EOT
fi
sudo apt-get update
sudo apt-get install -y osquery
sudo service osqueryd start 2>/dev/null || true
Ubuntu on EC2 requires some additional configuration. For hosts running on AWS, add the following flags in /etc/osquery/osquery.flags
:
--enable_syslog=true
--audit_allow_config=true
--audit_allow_sockets
--audit_persist=true
--disable_audit=false
--events_expiry=1
--events_Max=500000
--logger_min_status=1
--logger_plugin=filesystem
--watchdog_memory_limit=350
--watchdog_utilization_limit=130
To install the latest version of osquery on CentOS:
Install the
yum-utils
package.yum install yum-utils
Install the osquery repository:
Fetch the signing key:
curl -L https://pkg.osquery.io/rpm/GPG | tee /etc/pki/rpm-gpg/RPM-GPG-KEY-osquery
Add the package repository:
yum-config-manager --add-repo https://pkg.osquery.io/rpm/osquery-s3-rpm.repo
Enable the package repository:
yum-config-manager --enable osquery-s3-rpm-repo
Install osquery:
yum install osquery
Respond
yes
when prompted to approve the package install and accept the signing key.
Osquery configuration
Configure osquery using the following in /etc/osquery/osquery.conf
{
"options": {
"config_plugin": "filesystem",
"logger_plugin": "filesystem",
"database_path": "/var/osquery/osquery.db",
"utc": "true"
},
"schedule": {
"system_info": {
"query": "SELECT hostname, cpu_brand, physical_memory FROM system_info;",
"snapshot": true,
"interval": 60
},
"mounts_snapshot": {
"query": "SELECT device, device_alias, path, type, blocks, blocks_size, flags FROM mounts where path not like '/var/lib/%' and path not like '/run/docker/%' and path not like '/snap/%';",
"snapshot": true,
"interval": 60
},
"interfaces_snapshot": {
"query": "SELECT interface, mac, type, mtu, metric, flags, link_speed FROM interface_details;",
"snapshot": true,
"interval": 60
},
"system_uptime": {
"query": "SELECT * FROM uptime;",
"snapshot": true,
"interval": 300
},
"logged_in_users_snapshot": {
"query": "SELECT type, user, tty, host, time, pid FROM logged_in_users;",
"snapshot": true,
"interval": 60
},
"shell_history": {
"query": "SELECT * FROM users join shell_history using (uid);",
"interval": 10
},
"logged_in_users": {
"query": "SELECT type, user, tty, host, time, pid FROM logged_in_users;",
"interval": 10
},
"logged_in_users_snapshot": {
"query": "SELECT type, user, tty, host, time, pid FROM logged_in_users;",
"snapshot": true,
"interval": 300
},
"users_snapshot": {
"query": "SELECT uid, gid, uid_signed, gid_signed, username, description, directory, shell, uuid FROM users;",
"snapshot": true,
"interval": 60
}
},
"decorators": {
"load": [
"SELECT uuid AS host_uuid FROM system_info;",
"SELECT user AS username FROM logged_in_users ORDER BY time DESC LIMIT 1;"
]
},
"packs": {
}
}
To enable log rotation for Osquery’s own logs, add the following configuration in /etc/osquery/osquery.flags
. The example below creates up to 3 250MB log files. Adjust as necessary for your environment.
--logger_rotate=true
--logger_rotate_size=262144000
--logger_rotate_max_files=3
Restart the service to apply your configuration changes:
sudo service osqueryd restart
Fluent Bit
Install the latest version of Fluent Bit on Amazon Linux 2 with the following commands:
sudo tee /etc/yum.repos.d/td-agent-bit.repo > /dev/null <<EOT
[td-agent-bit]
name = TD Agent Bit
baseurl = https://packages.fluentbit.io/amazonlinux/2/\$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1
EOT
sudo yum install td-agent-bit -y
Install the latest version of Fluent Bit on Ubuntu 20.04 with the following commands:
wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -
Next, update your sources list: add the following line at the bottom of your /etc/apt/sources.list
file:
deb https://packages.fluentbit.io/ubuntu/focal focal main
finally refresh your packages and install fluentbit
sudo apt-get update
sudo apt-get install -y td-agent-bit
Note
For CentOS 7, please see the FAQ section for more about TLS certificates.
To install the latest version of Fluent Bit on CentOS:
Add the
td-agent-bit
repository reference:Create a new file,
td-agent-bit.repo
in/etc/yum.repos.d
containing the following:[td-agent-bit] name = TD Agent Bit baseurl = https://packages.fluentbit.io/centos/7/$basearch/ gpgcheck=1 gpgkey=https://packages.fluentbit.io/fluentbit.key enabled=1
Install
td-agent-bit
:yum install td-agent-bit
Start the
td-agent-bit
service:sudo service td-agent-bit start
Fluent Bit configuration
Configure Fluent Bit using the following in /etc/td-agent-bit/td-agent-bit.conf
, replacing MY_CUSTOMER_ID
and MY_INGEST_TOKEN
with your ID and token.
Important
If you are running on AWS EC2, uncomment the AWS metatags [FILTER]
block. This enables links back to your EC2 instance.
[SERVICE]
flush 10
daemon Off
log_level info
parsers_file parsers.conf
parsers_file input-parsers.conf
plugins_file plugins.conf
http_server Off
http_listen 0.0.0.0
http_port 2020
storage.metrics on
# Uncomment the below section if using AWS EC2
#[FILTER]
# Name aws
# Match *
# imds_version v1
# az true
# ec2_instance_id true
# ec2_instance_type true
# account_id true
# hostname true
# vpc_id true
[FILTER]
Name record_modifier
Match *
# if you want to group your servers into an application group
# [e.g. Proxy nodes] so you have have custom alert levels for them
# uncomment this next line
# Record appgroup ha-proxy
Record host ${HOSTNAME}
Record datacenter aws
Remove_key _MACHINE_ID
[INPUT]
name systemd
tag systemd
Read_From_Tail on
[OUTPUT]
name http
match systemd*
host collect.observeinc.com
port 443
URI /v1/http/fluentbit/systemd
Format msgpack
Header X-Observe-Decoder fluent
Compress gzip
http_User MY_CUSTOMER_ID
http_Passwd MY_INGEST_TOKEN
tls on
[INPUT]
name tail
tag tail_osquery_results
Path_Key path
path /var/log/osquery/osqueryd.results.log
Read_from_Head False
db osquery-results.db
[INPUT]
name tail
tag tail_osquery_snapshots
Path_Key path
path /var/log/osquery/osqueryd.snapshots.log
Read_from_Head False
db osquery-snapshots.db
[OUTPUT]
name http
match tail*
host collect.observeinc.com
port 443
URI /v1/http/fluentbit/tail
Format msgpack
Header X-Observe-Decoder fluent
Compress gzip
http_User MY_CUSTOMER_ID
http_Passwd MY_INGEST_TOKEN
tls on
Restart the service to apply the new configuration:
sudo service td-agent-bit restart
Telegraf
Install the latest version of Telegraf on Amazon Linux 2 with the following commands:
sudo tee /etc/yum.repos.d/influxdb.repo > /dev/null <<EOT
[influxdb]
name = InfluxDB Repository - RHEL
baseurl = https://repos.influxdata.com/rhel/7/x86_64/stable/
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOT
sudo yum install telegraf -y
Install the latest version of Telegraf on Ubuntu 20.04 with the following commands:
wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/lsb-release
if ! grep -Fq https://repos.influxdata.com/${DISTRIB_ID,,} /etc/apt/sources.list.d/influxdb.list
then
sudo tee -a /etc/apt/sources.list.d/influxdb.list > /dev/null <<EOT
deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable
EOT
fi
sudo apt-get update
sudo apt-get install -y telegraf
Install the latest version of Telegraf on CentOS with the following commands:
cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo
[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOF
sudo yum install telegraf
Then start the service:
sudo service telegraf start
Telegraf Configuration
Configure Telegraf using the following in /etc/telegraf/telegraf.conf
, replacing MY_CUSTOMER_ID
and MY_INGEST_TOKEN
with your ID and token. Also change datacenter
and host
to your desired values in [global_tags]
.
Important
If you are running on AWS EC2, uncomment the AWS metatags in the [[processors.aws_ec2]]
block. This enables links back to your EC2 instance.
[global_tags]
# update datacenter names to match Fluent Bit config
datacenter = "aws"
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
omit_hostname = false
[[outputs.http]]
url = "https://collect.observeinc.com:443/v1/http/telegraf"
timeout = "5s"
method = "POST"
username = "MY_CUSTOMER_ID"
password = "MY_INGEST_TOKEN"
insecure_skip_verify = true
data_format = "json"
content_encoding = "gzip"
[outputs.http.headers]
Content-Type = "application/json"
X-Observe-Decoder = "nested"
[[inputs.cpu]]
percpu = true
totalcpu = false
collect_cpu_time = false
report_active = false
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs","tracefs"]
[[inputs.diskio]]
# no configuration
[[inputs.net]]
# no configuration
[[inputs.kernel]]
# no configuration
[[inputs.mem]]
# no configuration
[[inputs.processes]]
# no configuration
[[inputs.swap]]
# no configuration
[[inputs.system]]
# no configuration
[[inputs.system]]
# no configuration
[[inputs.linux_sysctl_fs]]
# no configuration
#[[inputs.ntpq]]
# dns_lookup = true
[[inputs.procstat]]
exe = "."
prefix = "pgrep_serviceprocess"
interval = "60s"
period = "60s"
# Uncomment below metatags if using AWS EC2
#[[processors.aws_ec2]]
# imds_tags = [ "accountId", "instanceId"]
# timeout = "10s"
# max_parallel_calls = 10
Restart the service with
sudo service telegraf restart
To use Network Time Protocol (ntpq
) to get the hostname via DNS lookup:
Install
ntp
Ubuntu
sudo apt-get install ntp
CentOS
yum install ntp
Uncomment the following at the bottom of
telegraf.conf
:[[inputs.ntpq]] dns_lookup = true
Note
A host must use the same hostname for both Fluent Bit and Telegraf. To manually set the hostname in telegraf.conf
, specify it in [global_tags]
instead of using NTP dns_lookup
:
[global_tags]
datacenter = "test-datacenter"
# Set the hostname manually
host = "test-host"
Confirm data is being sent to Observe¶
To test that the forwarders are sending data, look for /telegraf
and /fluentbit
in the EXTRA
field of the associated data stream:

You can also check the status of the services:
sudo service telegraf status
sudo service osqueryd status
sudo service td-agent-bit status
Next Steps¶
The Linux Host Integration works with the datasets in your workspace. Contact us for assistance creating datasets and modeling the relationships between them. We can automate many common data modeling tasks for you, ensuring an accurate picture of your infrastructure. We can also update your workspace with improved and new datasets, troubleshooting boards and out-of-the-box monitors as we release new functionality for this integration.
FAQ¶
CentOS 7 certificates¶
If you are using CentOS 7 and are not able to update the default (expired) Root CA X3 certificate, you may see an error from Fluent Bit.
While we don’t recommend disabling TLS, if you need to temporarily do so you can add the following at the bottom of both [OUTPUT]
stanzas in your td-agent-bit.conf
:
# Turn transport layer security off - use common sense and make sure you don't send sensitive log content
tls.verify off
Adding a custom log file or entire directory¶
If you want to monitor a custom log file and forward log entries to observe as they are added to the file add the following section to your /etc/td-agent-bit/td-agent-bit.conf
:
[INPUT]
name tail
# specify a logfile tag
tag tail_myfile
Path_Key path
# specify the correct path or directory structure
#path /var/log/containers/*.log
path /var/log/myfile.log
Read_from_Head False
#provide a unique fluentbit checkpoint name
db myfilelog.db
[OUTPUT]
name http
match tail*
host collect.observeinc.com
port 443
URI /v1/http/fluentbit/tailmylog
Format msgpack
Header X-Observe-Decoder fluent
Compress gzip
http_User MY_CUSTOMER_ID
http_Passwd MY_INGEST_TOKEN
tls on
Sending data through a proxy¶
If your Linux hosts are not able to directly communicate with Observe endpoints, add a proxy configuration for systemd. The following example is for AWS Linux hosts:
For Fluent Bit, modify /usr/lib/systemd/system/td-agent-bit.service
to include the proxy settings similar to the example below:
[Unit]
Description=TD Agent Bit
Requires=network.target
After=network.target
[Service]
Type=simple
Environment="HTTP_PROXY=http://172.31.33.33:3128/"
Environment="HTTPS_PROXY=http://172.31.33.33:3128/"
ExecStart=/opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf
Restart=always
[Install]
WantedBy=multi-user.target
For Telegraf, modify /etc/systemd/system/multi-user.target.wants/telegraf.service
:
[Unit]
Description=The plugin-driven server agent for reporting metrics into InfluxDB
Documentation=https://github.com/influxdata/telegraf
After=network.target
[Service]
EnvironmentFile=-/etc/default/telegraf
User=telegraf
Environment="HTTP_PROXY=http://172.31.33.33:3128/"
Environment="HTTPS_PROXY=http://172.31.33.33:3128/"
ExecStart=/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d $TELEGRAF_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartForceExitStatus=SIGPIPE
KillMode=control-group
[Install]
WantedBy=multi-user.target