Host Monitoring installation guide¶

The Host Monitoring Integration uses osquery, Fluent Bit, and Telegraf to send logs and metrics to Observe. For more about exploring this data, see Host Monitoring Integration.

Setup¶

You need the following items:

Your Observe Customer ID
An ingest token - for details on creating an ingest token for a datastream, see Data streams
One or more Linux or Windows hosts to monitor

The instructions below work with the following platforms:

Amazon Linux 2
Ubuntu 22.04 LTS, Ubuntu 20.04 LTS, Ubuntu 18.04 LTS
Centos 7+
Windows Server 2016+

NOTE: For Windows Server 2012 R2, please use the instructions to install manually in the Install the host monitoring agents section

Install the Observe Host Monitoring App¶

Install the Host Monitoring app located on the Apps page to set up host monitoring for your Linux or Windows hosts. The installation process configures the app, creates datasets, worksheets, and sample monitors, as well as prepares the app to accept data from your hosts.

Installing host monitoring using the Observe app

Figure 1 - The Host Monitoring App

When you click Create connection, you then create a token to use with the data stream. After you create the token, follow the instructions to install the agents on your hosts.

Figure 2 - Create a token for the data stream.

Install the host monitoring agents¶

To start sending data to Observe, install the agents on each host. You need your Observe customer ID and the data stream token you created in the previous step.

Linux¶

The following sample command installs osquery, Fluent Bit, and Telegraf for Linux. Replace YOUR_CUSTOMERID, YOUR_DATA_STREAM_TOKEN, MY_DATA_CENTER, and MY_APP_GROUP with appropriate values and run on each host. For full details, see Configuraton script for Linux.

curl "https://raw.githubusercontent.com/observeinc/linux-host-configuration-scripts/main/observe_configure_script.sh" | bash -s -- --customer_id YOUR_CUSTOMERID --ingest_token YOUR_DATA_STREAM_TOKEN --observe_host_name https://YOUR_CUSTOMERID.collect.observeinc.com/ --config_files_clean TRUE --ec2metadata TRUE --datacenter MY_DATA_CENTER --appgroup MY_APP_GROUP

Windows¶

The following sample command installs osquery, Fluent Bit, and Telegraf for Windows. Replace YOUR_CUSTOMERID and YOUR_DATA_STREAM_TOKEN with the appropriate values and run on each host. Optionally set the name of the Observe collection host by appending -observe_host_name OBSERVE_HOST_NAME. You may also set additional optional parameters, for full details, see Configuraton script for Windows.

[Net.ServicePointManager]::SecurityProtocol = "Tls, Tls11, Tls12, Ssl3"; Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/observeinc/windows-host-configuration-scripts/main/agents.ps1" -outfile .\agents.ps1; .\agents.ps1  -ingest_token YOUR_DATA_STREAM_TOKEN -customer_id YOUR_CUSTOMERID

For details on installing and configuring osquery, Fluent Bit, and Telegraf individually, expand the section below, choose the desired package, and select the tab for your platform.

Confirm sending data to Observe¶

To test that the forwarders send data to Observe, look for /telegraf and /fluentbit in the EXTRA field of the associated data stream:

Filter dialog open for the EXTRA column, with "/telegraf", "/fluentbit/systemd", and "/fluentbit/tail" selected.

You can also check the status of the services on your hosts:

Linux

sudo service telegraf status
sudo service osqueryd status
sudo service td-agent-bit status

Windows

In a PowerShell window, run the following commands:

Get-Service telegraf
Get-Service osquerd
Get-Service fluent-bit

Next Steps¶

The Host Integration works with the datasets in your workspace. Contact us for assistance creating datasets and modeling additional relationships between them. We can automate many common data modeling tasks for you, ensuring an accurate picture of your infrastructure. We can also update your workspace with improved and new datasets, troubleshooting boards, and out-of-the-box monitors as we release new functionality for this integration.

FAQ¶

CentOS 7 certificates¶

If you use CentOS 7 and cannot update the default (expired) Root CA X3 certificate, you may see an error from Fluent Bit.

While we don’t recommend disabling TLS, if you need to temporarily do so you can add the following at the bottom of both [OUTPUT] stanzas in your td-agent-bit.conf:

# Turn transport layer security off - use common sense and make sure you don't send sensitive log content
tls.verify  off

Adding a custom log file or entire directory¶

If you want to monitor a custom log file and forward log entries to Observe as they are added to the file, add the following section to your /etc/td-agent-bit/td-agent-bit.conf:

[INPUT]
    name tail
    # specify a logfile tag
    tag  tail_myfile
    Path_Key path
    # specify the correct path or directory structure
    #path /var/log/containers/*.log
    path /var/log/myfile.log
    Read_from_Head False
    #provide a unique fluentbit checkpoint name
    db      myfilelog.db
[OUTPUT]
    name        http
    match       tail*
    host        MY_CUSTOMER_ID.collect.observeinc.com
    port        443
    URI         /v1/http/fluentbit/tailmylog
    Format      msgpack
    Header      X-Observe-Decoder fluent
    Compress    gzip
    http_Passwd MY_INGEST_TOKEN
    tls         on

Restart the service to apply the new configuration:

sudo service td-agent-bit restart

Sending data through a proxy¶

If your Linux hosts cannot directly communicate with Observe endpoints, add a proxy configuration for systemd. The following example is for AWS Linux hosts:

For Fluent Bit, modify /usr/lib/systemd/system/td-agent-bit.service to include the proxy settings similar to the example below:

[Unit]
Description=TD Agent Bit
Requires=network.target
After=network.target

[Service]
Type=simple
Environment="HTTP_PROXY=http://172.31.33.33:3128/"
Environment="HTTPS_PROXY=http://172.31.33.33:3128/" 
ExecStart=/opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf
Restart=always

[Install]
WantedBy=multi-user.target

For Telegraf, modify /etc/systemd/system/multi-user.target.wants/telegraf.service:

[Unit]
Description=The plugin-driven server agent for reporting metrics into InfluxDB
Documentation=https://github.com/influxdata/telegraf
After=network.target

[Service]
EnvironmentFile=-/etc/default/telegraf
User=telegraf
Environment="HTTP_PROXY=http://172.31.33.33:3128/"
Environment="HTTPS_PROXY=http://172.31.33.33:3128/" 
ExecStart=/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d $TELEGRAF_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartForceExitStatus=SIGPIPE
KillMode=control-group

[Install]
WantedBy=multi-user.target