Install the Host Monitoring app¶

The Host Monitoring Integration app uses osquery, Fluent Bit, and Telegraf to send logs and metrics to Observe. For more about exploring this data, see Host Monitoring Integration.

Setup¶

You need the following items:

Your Observe Customer ID
An ingest token - for details on creating an ingest token for a datastream, see Data streams
One or more Linux, Windows, or MacOS hosts to monitor

Note

Some Observe instances may optionally use a name instead of Customer ID; if this is the case for your instance, contact your Observe Data Engineer to discuss implementation. A stem name will work as is, but a DNS redirect name may require client configuration.

The instructions work with the following platforms:

Note

For Windows Server 2012 R2, please use the instructions to install manually in the Install the host monitoring agents section.

For MacOS, homebrew is required by the installation script.

Note

The command line options are different for MacOS than Linux or Windows, and should be reviewed.

Install the Observe Host Monitoring App¶

Install the Host Monitoring app located on the Apps page to set up host monitoring for your Linux or Windows hosts. The installation process configures the app, creates datasets, worksheets, and sample monitors, as well as prepares the app to accept data from your hosts.

Installing host monitoring using the Observe app

Figure 1 - The Host Monitoring App

When you click Create connection, you create a token to use with the data stream. After you create the token, follow the instructions to install the agents on your hosts.

Figure 2 - Create a token for the data stream.

Next Steps¶

The Host Monitoring app works with the datasets in your workspace. Contact your Observe Data Engineer for assistance creating datasets and modeling additional relationships between them. You can automate many common data modeling tasks, ensuring an accurate picture of your infrastructure.

FAQ¶

How do I filter the events received from Fluent Bit?¶

By default, Observe ingests all events received from Fluent Bit. If you don’t want to receive all of the events, you can filter the events using instructions found in Fluent Bit documentation.

Expected logs do not appear in Observe or the logs stopped appearing in Observe?¶

This may occur when Fluent Bit becomes stuck when dealing with long log lines without an appropriately sized buffer.

To learn more about Fluent Bit configurations, and how to adjust buffer sizes, please review this information for details.

For Linux, you view the size of the longest log line with the following command:

  
    awk '{print length}' <file> |sort -nr|head -1

For example:

    % awk '{print length}' osqueryd.snapshots.log |sort -nr|head -1
    220541

The example above means the longest line is 220541 bytes. You might set Buffer_Max_Size to 256k or higher.

If you believe long log lines are not critical and you want to skip them gracefully and continue monitoring your log file, you can set Skip_Long_Lines to On.

CentOS 7 certificates¶

If you use CentOS 7 and cannot update the default (expired) Root CA X3 certificate, you may see an error from Fluent Bit.

While Observe does not recommend disabling TLS, if you need to temporarily do so you can add the following at the bottom of both [OUTPUT] stanzas in your td-agent-bit.conf:

# Turn transport layer security off - use common sense and make sure you don't send sensitive log content
tls.verify  off

Adding a custom log file or directory¶

If you want to monitor a custom log file and forward log entries to Observe as they become available in the file, add the following section to your td-agent-bit.conf or fluent-bit.conf:

[INPUT]
    name tail
    # specify a logfile tag
    tag  tail_myfile
    Path_Key path
    # specify the correct path or directory structure
    #path /var/log/containers/*.log
    path /var/log/myfile.log
    Read_from_Head False
    #provide a unique Fluent Bit checkpoint name
    db      myfilelog.db
[OUTPUT]
    name        http
    match       tail*
    host        OBSERVE_CUSTOMER.collect.observeinc.com
    port        443
    URI         /v1/http/Fluent Bit/tailmylog
    Format      msgpack
    Header      X-Observe-Decoder fluent
    Compress    gzip
    http_Passwd MY_INGEST_TOKEN
    tls         on

Restart the service to apply the new configuration:

sudo service td-agent-bit restart

Sending data through a proxy¶

If your Linux hosts cannot directly communicate with Observe endpoints, add a proxy configuration for systemd. The following example is for AWS Linux hosts:

For Fluent Bit, modify /usr/lib/systemd/system/td-agent-bit.service to include the proxy settings similar to the example below:

[Unit]
Description=TD Agent Bit
Requires=network.target
After=network.target

[Service]
Type=simple
Environment="HTTP_PROXY=http://172.31.33.33:3128/"
Environment="HTTPS_PROXY=http://172.31.33.33:3128/" 
ExecStart=/opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf
Restart=always

[Install]
WantedBy=multi-user.target

For Telegraf, modify /etc/systemd/system/multi-user.target.wants/telegraf.service:

[Unit]
Description=The plugin-driven server agent for reporting metrics into InfluxDB
Documentation=https://github.com/influxdata/telegraf
After=network.target

[Service]
EnvironmentFile=-/etc/default/telegraf
User=telegraf
Environment="HTTP_PROXY=http://172.31.33.33:3128/"
Environment="HTTPS_PROXY=http://172.31.33.33:3128/" 
ExecStart=/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d $TELEGRAF_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartForceExitStatus=SIGPIPE
KillMode=control-group

[Install]
WantedBy=multi-user.target