Host Monitoring installation guide¶
The Host Monitoring Integration uses osquery, Fluent Bit, and Telegraf to send logs and metrics to Observe. Once the forwarders are installed and sending data, contact Observe about modeling the data and creating datasets. For more about exploring this data, see Host Monitoring Integration.
Setup¶
You need:
Your Observe Customer ID
An ingest token - for details on creating an ingest token for a datastream, see Data streams
One or more Linux hosts to monitor
The instructions below work with the following:
Amazon Linux 2
Ubuntu 20.04 LTS
Centos 7+
Windows Server 2012 R2
Windows Server 2016+
Using the Host Monitoring App¶
You can use the Host Monitoring app located on the Apps page to install host monitoring on your Linux or Windows hosts.

Figure 1 - The Host Monitoring App
When you click Create connection, you then create a token to use for the data stream. After you create the token, follow the instructions to install the agents on your hosts.

Figure 2 - Create a token for the data stream.
Manually installing host monitoring agents¶
To ingest data, install and configure the needed agents on each host.
Install Osquery, Fluent Bit, and Telegraf¶
Osquery
Install the latest version of osquery on Amazon Linux 2 with the following command:
curl -L https://pkg.osquery.io/rpm/GPG | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-osquery
sudo yum-config-manager --add-repo https://pkg.osquery.io/rpm/osquery-s3-rpm.repo
sudo yum-config-manager --enable osquery-s3-rpm-repo
sudo yum install osquery -y
sudo service osqueryd start 2>/dev/null || true
Install the latest version of osquery on Ubuntu 20.04 with the following commands:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
if ! grep -Fq https://pkg.osquery.io/deb /etc/apt/sources.list.d/osquery.list
then
sudo tee -a /etc/apt/sources.list.d/osquery.list > /dev/null <<EOT
deb [arch=amd64] https://pkg.osquery.io/deb deb main
EOT
fi
sudo apt-get update
sudo apt-get install -y osquery
sudo service osqueryd start 2>/dev/null || true
Ubuntu on EC2 requires some additional configuration. For hosts running on AWS, add the following flags in /etc/osquery/osquery.flags
:
--enable_syslog=true
--audit_allow_config=true
--audit_allow_sockets
--audit_persist=true
--disable_audit=false
--events_expiry=1
--events_Max=500000
--logger_min_status=1
--logger_plugin=filesystem
--watchdog_memory_limit=350
--watchdog_utilization_limit=130
To install the latest version of osquery on CentOS:
Install the
yum-utils
package.yum install yum-utils
Install the osquery repository:
Fetch the signing key:
curl -L https://pkg.osquery.io/rpm/GPG | tee /etc/pki/rpm-gpg/RPM-GPG-KEY-osquery
Add the package repository:
yum-config-manager --add-repo https://pkg.osquery.io/rpm/osquery-s3-rpm.repo
Enable the package repository:
yum-config-manager --enable osquery-s3-rpm-repo
Install osquery:
yum install osquery
Respond
yes
when prompted to approve the package install and accept the signing key.
To install the latest version of osquery on Microsoft Windows:
Download the latest MSI for Windows from the Osquery Downloads page
Launch the MSI and follow the instructions on the screen.
Osquery configuration
Configure osquery using the following in /etc/osquery/osquery.conf
{
"options": {
"config_plugin": "filesystem",
"logger_plugin": "filesystem",
"database_path": "/var/osquery/osquery.db",
"utc": "true"
},
"schedule": {
"system_info": {
"query": "SELECT hostname, cpu_brand, physical_memory FROM system_info;",
"snapshot": true,
"interval": 60
},
"mounts_snapshot": {
"query": "SELECT device, device_alias, path, type, blocks, blocks_size, flags FROM mounts where path not like '/var/lib/%' and path not like '/run/docker/%' and path not like '/snap/%';",
"snapshot": true,
"interval": 60
},
"interfaces_snapshot": {
"query": "SELECT interface, mac, type, mtu, metric, flags, link_speed FROM interface_details;",
"snapshot": true,
"interval": 60
},
"system_uptime": {
"query": "SELECT * FROM uptime;",
"snapshot": true,
"interval": 300
},
"logged_in_users_snapshot": {
"query": "SELECT type, user, tty, host, time, pid FROM logged_in_users;",
"snapshot": true,
"interval": 60
},
"shell_history": {
"query": "SELECT * FROM users join shell_history using (uid);",
"interval": 10
},
"logged_in_users": {
"query": "SELECT type, user, tty, host, time, pid FROM logged_in_users;",
"interval": 10
},
"logged_in_users_snapshot": {
"query": "SELECT type, user, tty, host, time, pid FROM logged_in_users;",
"snapshot": true,
"interval": 300
},
"users_snapshot": {
"query": "SELECT uid, gid, uid_signed, gid_signed, username, description, directory, shell, uuid FROM users;",
"snapshot": true,
"interval": 60
}
},
"decorators": {
"load": [
"SELECT uuid AS host_uuid FROM system_info;",
"SELECT user AS username FROM logged_in_users ORDER BY time DESC LIMIT 1;"
]
},
"packs": {
}
}
Configure osquery by creating the following configuration in C:\Program Files\osquery\osquery.conf
. (If you have installed osquery in a non-default location, create this file in your installation directory.)
{
"options": {
"utc": "true"
},
"schedule": {
"system_info": {
"query": "SELECT hostname, cpu_brand, physical_memory FROM system_info;",
"snapshot": true,
"interval": 60
},
"logcial_drives_snapshot": {
"query": "SELECT boot_partition, description, device_id, file_system, size, type FROM logical_drives;",
"snapshot": true,
"interval": 60
},
"interfaces_snapshot": {
"query": "SELECT interface, mac, type, mtu, metric, flags, speed, connection_id FROM interface_details;",
"snapshot": true,
"interval": 60
},
"system_uptime": {
"query": "SELECT * FROM uptime;",
"snapshot": true,
"interval": 300
},
"logged_in_users_snapshot": {
"query": "SELECT type, user, tty, host, time, pid FROM logged_in_users;",
"snapshot": true,
"interval": 60
},
"users_snapshot": {
"query": "SELECT uid, gid, uid_signed, gid_signed, username, description, directory, shell, uuid FROM users;",
"snapshot": true,
"interval": 60
},
"os_snapshot" : {
"query": "SELECT * FROM os_version;",
"snapshot": true,
"interval": 60
}
},
"decorators": {
"load": [
"SELECT uuid AS host_uuid FROM system_info;",
"SELECT user AS username FROM logged_in_users ORDER BY time DESC LIMIT 1;"
]
},
"packs": {
}
}
Once the config is written, open a PowerShell window and run the following command to restart the osquery service
Restart-Service osqueryd
To enable log rotation for Osquery’s own logs, add the following configuration in C:\Program Files\osquery\osquery.flags
. The example below creates up to 3 250MB log files. Adjust as necessary for your environment.
--logger_rotate=true
--logger_rotate_size=262144000
--logger_rotate_max_files=3
Restart the service to apply your configuration changes in PowerShell:
Restart-Service osqueryd
Fluent Bit
Install the latest version of Fluent Bit on Amazon Linux 2 with the following commands:
sudo tee /etc/yum.repos.d/td-agent-bit.repo > /dev/null <<EOT
[td-agent-bit]
name = TD Agent Bit
baseurl = https://packages.fluentbit.io/amazonlinux/2/\$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1
EOT
sudo yum install td-agent-bit -y
Install the latest version of Fluent Bit on Ubuntu 20.04 with the following commands:
wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -
Next, update your sources list: add the following line at the bottom of your /etc/apt/sources.list
file:
deb https://packages.fluentbit.io/ubuntu/focal focal main
finally refresh your packages and install fluentbit
sudo apt-get update
sudo apt-get install -y td-agent-bit
Note
For CentOS 7, please see the FAQ section for more about TLS certificates.
To install the latest version of Fluent Bit on CentOS:
Add the
td-agent-bit
repository reference:Create a new file,
td-agent-bit.repo
in/etc/yum.repos.d
containing the following:[td-agent-bit] name = TD Agent Bit baseurl = https://packages.fluentbit.io/centos/7/$basearch/ gpgcheck=1 gpgkey=https://packages.fluentbit.io/fluentbit.key enabled=1
Install
td-agent-bit
:yum install td-agent-bit
Start the
td-agent-bit
service:sudo service td-agent-bit start
Install the latest version of Fluent Bit on Windows by:
Download the latest intaller EXE from the Fluent Bit Downloads Page.
Double click the installer and follow the instructions on the installation wizard.
Fluent Bit configuration
Configure Fluent Bit using the following in /etc/td-agent-bit/td-agent-bit.conf
, replacing MY_CUSTOMER_ID
and MY_INGEST_TOKEN
with your ID and token.
Important
If you are running on AWS EC2, uncomment the AWS metatags [FILTER]
block. This enables links back to your EC2 instance.
[SERVICE]
flush 10
daemon Off
log_level info
parsers_file parsers.conf
parsers_file input-parsers.conf
plugins_file plugins.conf
http_server Off
http_listen 0.0.0.0
http_port 2020
storage.metrics on
# Uncomment the below section if using AWS EC2
#[FILTER]
# Name aws
# Match *
# imds_version v1
# az true
# ec2_instance_id true
# ec2_instance_type true
# account_id true
# hostname true
# vpc_id true
[FILTER]
Name record_modifier
Match *
# if you want to group your servers into an application group
# [e.g. Proxy nodes] so you have have custom alert levels for them
# uncomment this next line
# Record appgroup ha-proxy
Record host ${HOSTNAME}
Record datacenter aws
Remove_key _MACHINE_ID
[INPUT]
name systemd
tag systemd
Read_From_Tail on
[OUTPUT]
name http
match systemd*
host collect.observeinc.com
port 443
URI /v1/http/fluentbit/systemd
Format msgpack
Header X-Observe-Decoder fluent
Compress gzip
http_User MY_CUSTOMER_ID
http_Passwd MY_INGEST_TOKEN
tls on
[INPUT]
name tail
tag tail_osquery_results
Path_Key path
path /var/log/osquery/osqueryd.results.log
Read_from_Head False
db osquery-results.db
[INPUT]
name tail
tag tail_osquery_snapshots
Path_Key path
path /var/log/osquery/osqueryd.snapshots.log
Read_from_Head False
db osquery-snapshots.db
[OUTPUT]
name http
match tail*
host collect.observeinc.com
port 443
URI /v1/http/fluentbit/tail
Format msgpack
Header X-Observe-Decoder fluent
Compress gzip
http_User MY_CUSTOMER_ID
http_Passwd MY_INGEST_TOKEN
tls on
Restart the service to apply the new configuration:
sudo service td-agent-bit restart
The following assumes that Fluent Bit is installed in the default location of C:\Program Files\fluent-bit
.
Configure Fluent Bit using the following in C:\Program Files\fluent-bit\fluent-bit.conf
, replacing MY_CUSTOMER_ID
and MY_INGEST_TOKEN
with your ID and token.
[SERVICE]
flush 5
daemon Off
log_level debug
log_file C:\Program Files\fluent-bit\fluent.log
[INPUT]
Name winlog
Channels Security,Application,System
Interval_Sec 5
DB C:\Program Files\fluent-bit\fluent.pos
# Uncomment the below section if using AWS EC2
#[FILTER]
# Name aws
# Match *
# imds_version v1
# az true
# ec2_instance_id true
# ec2_instance_type true
# account_id true
# hostname true
# vpc_id true
[FILTER]
Name record_modifier
Match *
# if you want to group your servers into an application group
# [e.g. Proxy nodes] so you have have custom alert levels for them
# uncomment this next line
# Record appgroup ha-proxy
Record host ${HOSTNAME}
Record datacenter aws
Remove_key _MACHINE_ID
[INPUT]
name tail
tag tail_osquery_results
Path_Key path
path C:\Program Files\osquery\log\osqueryd.results.log
Read_from_Head False
db osquery-results.db
[INPUT]
name tail
tag tail_osquery_snapshots
Path_Key path
path C:\Program Files\osquery\log\osqueryd.snapshots.log
Read_from_Head False
db osquery-snapshots.db
[OUTPUT]
name http
match *
host collect.observeinc.com
port 443
URI /v1/http/fluentbit/tail
Format msgpack
Header X-Observe-Decoder fluent
Compress gzip
http_User MY_CUSTOMER_ID
http_Passwd MY_INGEST_TOKEN
tls on
Next, create a service with the following command in a PowerShell window:
New-Service fluent-bit -BinaryPathName "C:\Program Files\fluent-bit\bin\fluent-bit.exe" -c "C:\Program Files\fluent-bit\conf\fluent-bit.conf" -StartupType Automatic
Start-Service fluent-bit
Telegraf
Install the latest version of Telegraf on Amazon Linux 2 with the following commands:
sudo tee /etc/yum.repos.d/influxdb.repo > /dev/null <<EOT
[influxdb]
name = InfluxDB Repository - RHEL
baseurl = https://repos.influxdata.com/rhel/7/x86_64/stable/
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOT
sudo yum install telegraf -y
Install the latest version of Telegraf on Ubuntu 20.04 with the following commands:
wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/lsb-release
if ! grep -Fq https://repos.influxdata.com/${DISTRIB_ID,,} /etc/apt/sources.list.d/influxdb.list
then
sudo tee -a /etc/apt/sources.list.d/influxdb.list > /dev/null <<EOT
deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable
EOT
fi
sudo apt-get update
sudo apt-get install -y telegraf
Install the latest version of Telegraf on CentOS with the following commands:
cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo
[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOF
sudo yum install telegraf
Then start the service:
sudo service telegraf start
Install the latest version of Telegraf by:
Download the latest version of Telegraf from the InfluxData Downloads page.
Extract the contents of the ZIP archive to `C:\Program Files\InfluxData\Telegraf
Telegraf Configuration
Configure Telegraf using the following in /etc/telegraf/telegraf.conf
, replacing MY_CUSTOMER_ID
and MY_INGEST_TOKEN
with your ID and token. Also change datacenter
and host
to your desired values in [global_tags]
.
Important
If you are running on AWS EC2, uncomment the AWS metatags in the [[processors.aws_ec2]]
block. This enables links back to your EC2 instance.
[global_tags]
# update datacenter names to match Fluent Bit config
datacenter = "aws"
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
omit_hostname = false
[[outputs.http]]
url = "https://collect.observeinc.com:443/v1/http/telegraf"
timeout = "5s"
method = "POST"
username = "MY_CUSTOMER_ID"
password = "MY_INGEST_TOKEN"
insecure_skip_verify = true
data_format = "json"
content_encoding = "gzip"
[outputs.http.headers]
Content-Type = "application/json"
X-Observe-Decoder = "nested"
[[inputs.cpu]]
percpu = true
totalcpu = false
collect_cpu_time = false
report_active = false
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs","tracefs"]
[[inputs.diskio]]
# no configuration
[[inputs.net]]
# no configuration
[[inputs.kernel]]
# no configuration
[[inputs.mem]]
# no configuration
[[inputs.processes]]
# no configuration
[[inputs.swap]]
# no configuration
[[inputs.system]]
# no configuration
[[inputs.system]]
# no configuration
[[inputs.linux_sysctl_fs]]
# no configuration
#[[inputs.ntpq]]
# dns_lookup = true
[[inputs.procstat]]
exe = "."
prefix = "pgrep_serviceprocess"
interval = "60s"
period = "60s"
# Uncomment below metatags if using AWS EC2
#[[processors.aws_ec2]]
# imds_tags = [ "accountId", "instanceId"]
# timeout = "10s"
# max_parallel_calls = 10
Restart the service with
sudo service telegraf restart
To use Network Time Protocol (ntpq
) to get the hostname via DNS lookup:
Install
ntp
Ubuntu
sudo apt-get install ntp
CentOS
yum install ntp
Uncomment the following at the bottom of
telegraf.conf
:[[inputs.ntpq]] dns_lookup = true
Configure Telegraf using the following in C:\Program Files\InfluxData\telegraf\telegraf.conf
, replacing MY_CUSTOMER_ID
and MY_INGEST_TOKEN
with your ID and token. Also change datacenter
and host
to your desired values in [global_tags]
.
Important
If you are running on AWS EC2, uncomment the AWS metatags in the [[processors.aws_ec2]]
block. This enables links back to your EC2 instance.
[global_tags]
# update datacenter names to match Fluent Bit config
datacenter = "aws"
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
omit_hostname = false
# OUTPUT PLUGINS
[[outputs.http]]
url = "https://<<observe_host_name>>:443/v1/observations/telegraf"
timeout = "5s"
method = "POST"
username = "<<customer_id>>"
password = "<<ingest_token>>"
insecure_skip_verify = true
data_format = "json"
content_encoding = "gzip"
[outputs.http.headers]
Content-Type = "application/json"
X-Observe-Decoder = "nested"
# INPUT PLUGINS
[[inputs.cpu]]
percpu = true
totalcpu = false
collect_cpu_time = false
report_active = false
[[inputs.disk]]
# no configuration
[[inputs.diskio]]
# no configuration
[[inputs.net]]
# no configuration
[[inputs.mem]]
# no configuration
[[inputs.processes]]
# no configuration
[[inputs.swap]]
# no configuration
[[inputs.system]]
# no configuration
[[inputs.system]]
# no configuration
#[[inputs.ntpq]]
# dns_lookup = true
[[inputs.win_perf_counters]]
UseWildcardsExpansion = true
[[inputs.win_perf_counters.object]]
ObjectName = "Process"
Counters = ["% Processor Time","Handle Count","Private Bytes","Thread Count","Virtual Bytes","Working Set","ID Process"]
#Counters = ["*"]
Instances = ["*"]
Measurement = "win_proc"
#IncludeTotal=false #Set to true to include _Total instance when querying for all (*).
[[inputs.win_services]]
# no configurations
# Uncomment below metatags if using AWS EC2
#[[processors.aws_ec2]]
# imds_tags = [ "accountId", "instanceId"]
# timeout = "10s"
# max_parallel_calls = 10
Create the service by opening up a command prompt and entering the following:
cd "C:\Program Files\InfluxData\telegraf\"
.\telegraf.exe --service install --config "C:\Program Files\InfluxData\telegraf\telegraf.conf"
.\telegraf.exe --service start
Note
A host must use the same hostname for both Fluent Bit and Telegraf. To manually set the hostname in telegraf.conf
, specify it in [global_tags]
instead of using NTP dns_lookup
:
[global_tags]
datacenter = "test-datacenter"
# Set the hostname manually
host = "test-host"
Confirm data is being sent to Observe¶
To test that the forwarders are sending data, look for /telegraf
and /fluentbit
in the EXTRA
field of the associated data stream:

You can also check the status of the services:
sudo service telegraf status
sudo service osqueryd status
sudo service td-agent-bit status
In a PowerShell window, run the following commands:
Get-Service telegraf
Get-Service osquerd
Get-Service fluent-bit
Next Steps¶
The Host Integration works with the datasets in your workspace. Contact us for assistance creating datasets and modeling the relationships between them. We can automate many common data modeling tasks for you, ensuring an accurate picture of your infrastructure. We can also update your workspace with improved and new datasets, troubleshooting boards and out-of-the-box monitors as we release new functionality for this integration.
FAQ¶
CentOS 7 certificates¶
If you are using CentOS 7 and are not able to update the default (expired) Root CA X3 certificate, you may see an error from Fluent Bit.
While we don’t recommend disabling TLS, if you need to temporarily do so you can add the following at the bottom of both [OUTPUT]
stanzas in your td-agent-bit.conf
:
# Turn transport layer security off - use common sense and make sure you don't send sensitive log content
tls.verify off
Adding a custom log file or entire directory¶
If you want to monitor a custom log file and forward log entries to observe as they are added to the file add the following section to your /etc/td-agent-bit/td-agent-bit.conf
:
[INPUT]
name tail
# specify a logfile tag
tag tail_myfile
Path_Key path
# specify the correct path or directory structure
#path /var/log/containers/*.log
path /var/log/myfile.log
Read_from_Head False
#provide a unique fluentbit checkpoint name
db myfilelog.db
[OUTPUT]
name http
match tail*
host collect.observeinc.com
port 443
URI /v1/http/fluentbit/tailmylog
Format msgpack
Header X-Observe-Decoder fluent
Compress gzip
http_User MY_CUSTOMER_ID
http_Passwd MY_INGEST_TOKEN
tls on
Sending data through a proxy¶
If your Linux hosts are not able to directly communicate with Observe endpoints, add a proxy configuration for systemd. The following example is for AWS Linux hosts:
For Fluent Bit, modify /usr/lib/systemd/system/td-agent-bit.service
to include the proxy settings similar to the example below:
[Unit]
Description=TD Agent Bit
Requires=network.target
After=network.target
[Service]
Type=simple
Environment="HTTP_PROXY=http://172.31.33.33:3128/"
Environment="HTTPS_PROXY=http://172.31.33.33:3128/"
ExecStart=/opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf
Restart=always
[Install]
WantedBy=multi-user.target
For Telegraf, modify /etc/systemd/system/multi-user.target.wants/telegraf.service
:
[Unit]
Description=The plugin-driven server agent for reporting metrics into InfluxDB
Documentation=https://github.com/influxdata/telegraf
After=network.target
[Service]
EnvironmentFile=-/etc/default/telegraf
User=telegraf
Environment="HTTP_PROXY=http://172.31.33.33:3128/"
Environment="HTTPS_PROXY=http://172.31.33.33:3128/"
ExecStart=/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d $TELEGRAF_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartForceExitStatus=SIGPIPE
KillMode=control-group
[Install]
WantedBy=multi-user.target