Fluent Bit

Fluent Bit is a lightweight log processor and forwarder.

Installation

Fluent Bit provides detailed installation instructions on their website. For convenience, we provide pointers for the most frequently requested platforms:

Fluent Bit distributes td-agent-bit for officially supported distributions:

Alternatively, you can build from source.

Please ensure you download Fluent Bit for Windows 1.9.1 or higher - older versions require a manual root certificate installation process.

  1. Install the appropriate Fluent Bit td-agent-bit package, available at https://docs.fluentbit.io/manual/installation/windows.

Fluent Bit maintains and regularly releases container images:

$ docker run -ti fluent/fluent-bit:1.7
Fluent Bit v1.7.2
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[ info] [engine] started (pid=1)
[ info] [storage] version=1.1.1, initializing...
[ info] [storage] in-memory
[ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[ info] [sp] stream processor started

Fluent Bit is available through homebrew:

$ brew install fluent-bit

Configuration

The following snippet contains a minimal configuration to send data to Observe. It uses the dummy input plugin that generates sample events. (See Inputs in the Fluent Bit docs for a list of available plugins.)

[SERVICE]
    flush        5
    daemon       Off
    log_level    info

# Uncomment the below section if using AWS EC2
#[FILTER]
#    Name aws
#    Match *
#    imds_version v1
#    az true
#    ec2_instance_id true
#    ec2_instance_type true
#    account_id true
#    hostname true
#    vpc_id true

[INPUT]
    name  dummy
    dummy {"message":"Hello World!"}
    rate  1

[OUTPUT]
    name         http
    match        *
    host         collect.observeinc.com
    port         443
    tls          on

    http_user    ${OBSERVE_CUSTOMER}
    http_passwd  ${OBSERVE_TOKEN}
    uri          /v1/http/fluentbit

    format       msgpack
    header       X-Observe-Decoder fluent
    compress     gzip

This [INPUT] section reads from a log file using the tail plugin:

[INPUT]
    name        tail
    path        /var/log/my/service/*.log
    path_key    filename

    # Additional options for logs with very long lines
    # buffer_max_size  256k
    # skip_long_lines  on

For additional input plugins, see the Fluent Bit Inputs documentation.

We rely on Fluent Bit’s http output to forward data to Observe’s HTTP endpoint. We can export data in Fluent Bit’s native msgpack format directly.

FAQ

Retry on failure

Fluent Bit retries on 5XX and 429 Too Many Requests errors. It will stop reading new log data when its buffer fills and resume when possible. Fluent Bit does not retry on other 4XX errors. See the Fluent Bit Scheduling and Retries documentation for more about retry configuration.