Fluent Bit

Fluent Bit provides lightweight log processor and forwarder services.

Installation

Fluent Bit provides detailed installation instructions on the FluentBit website. For convenience, Observe provides pointers for the most frequently requested platforms:

Fluent Bit distributes td-agent-bit for officially supported distributions:

Alternatively, you can build from source.

Ensure you download Fluent Bit for Windows 1.9.1 or higher, as older versions require a manual root certificate installation process.

  1. Install the appropriate Fluent Bit td-agent-bit package, available at https://docs.fluentbit.io/manual/installation/windows.

Fluent Bit maintains and regularly releases container images:

$ docker run fluent/fluent-bit
Fluent Bit v2.0.8
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[ info] [fluent bit] version=2.0.8, commit=9444fdc5ee, pid=1
[ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[ info] [cmetrics] version=0.5.8
[ info] [ctraces ] version=0.2.7
[ info] [input:cpu:cpu.0] initializing
[ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[ info] [sp] stream processor started
[ info] [output:stdout:stdout.0] worker #0 started

To read files from the host, mount the directory using volumes:

$ docker run --volume /path/on/host:/path/in/container:ro fluent/fluent-bit

You can obtain Fluent Bit through homebrew:

$ brew install fluent-bit

Configuration

The following snippet contains a minimal configuration to send data to Observe. It uses the dummy input plugin that generates sample events. (See Inputs in the Fluent Bit docs for a list of available plugins.)

Note

Observe strongly recommends configuring a file location for the Fluent bit service logs to assist with debugging potential ingest issues. You can do so by specifying a host file location of your choice in the [SERVICE] section of the Fluent Bit configuration. Example:

[SERVICE]
 ...
 log_file    some/host/path.log

[SERVICE]
    flush        5
    daemon       Off
    log_level    info

# Uncomment the below section if using AWS EC2
#[FILTER]
#    Name aws
#    Match *
#    imds_version v1
#    az true
#    ec2_instance_id true
#    ec2_instance_type true
#    account_id true
#    hostname true
#    vpc_id true

[INPUT]
    name  dummy
    dummy {"message":"Hello World!"}
    rate  1

[OUTPUT]
    name         http
    match        *
    host         ${OBSERVE_CUSTOMER}.collect.observeinc.com
    port         443
    tls          on

    uri          /v1/http/fluentbit

    format       msgpack
    header       Authorization     Bearer ${OBSERVE_TOKEN}
    header       X-Observe-Decoder fluent
    compress     gzip

This [INPUT] section reads from a log file using the tail plugin:

[INPUT]
    name        tail
    path        /var/log/my/service/*.log
    path_key    filename

    # Additional options for logs with very long lines
    # buffer_max_size  256k
    # skip_long_lines  on

For additional input plugins, see the Fluent Bit Inputs documentation.

Observe relies on Fluent Bit’s http output to forward data to the HTTP endpoint. Observe can export data in Fluent Bit’s native msgpack format directly.

FAQ

Retry on failure

Fluent Bit retries on 5XX and 429 Too Many Requests errors. Fluent Bit does not retry on other 4XX errors. It stops reading new log data when the buffer fills and resumes when possible. See the Fluent Bit Scheduling and Retries documentation for more about retry configuration.