Fluent Bit¶

Fluent Bit is a lightweight log processor and forwarder.

Installation¶

Fluent Bit provides detailed installation instructions on their website. For convenience, we provide pointers for the most frequently requested platforms:

Linux

Fluent Bit distributes td-agent-bit for officially supported distributions:

Alternatively, you can build from source.

Windows

Fluent Bit for Windows requires a manually provided root certificate to be able to send data to Observe.

Install the appropriate Fluent Bit td-agent-bit package, available at https://docs.fluentbit.io/manual/installation/windows.
Download the self-signed ISRG Root X1 pem certificate from the Let’s Encrypt Certificates page and copy it to the directory of your choice on the Windows host.
Add the path to isrgrootx1.pem to the [OUTPUT] section of your fluent-bit.conf file:
```
tls.ca_file  C:\td-agent-bit\isrgrootx1.pem
```

Docker

Fluent Bit maintains and regularly releases container images:

$ docker run -ti fluent/fluent-bit:1.7
Fluent Bit v1.7.2
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[ info] [engine] started (pid=1)
[ info] [storage] version=1.1.1, initializing...
[ info] [storage] in-memory
[ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[ info] [sp] stream processor started

MacOS

Fluent Bit is available through homebrew:

$ brew install fluent-bit

Configuration¶

The following snippet contains a minimal configuration to send CPU metrics to Observe:

[SERVICE]
    flush        5
    daemon       Off
    log_level    info

[INPUT]
    name cpu
    interval_sec 1

[OUTPUT]
    name         http
    match        *
    host         collect.observeinc.com
    port         443
    tls          on

    # For Windows: provide path to root cert
    #tls.ca_file  C:\td-agent-bit\isrgrootx1.pem

    http_user    ${OBSERVE_CUSTOMER}
    http_passwd  ${OBSERVE_TOKEN}
    uri          /v1/http/fluentbit

    format       msgpack
    header       X-Observe-Decoder fluent
    compress     gzip

This [INPUT] section reads from a log file using the tail plugin:

[INPUT]
    name        tail
    path        /var/log/my/service/*.log
    path_key    filename

    # Additional options for logs with very long lines
    # buffer_max_size  256k
    # skip_long_lines  on

We rely on Fluent Bit’s http output to forward data towards Observe’s HTTP endpoint. We can export data in Fluent Bit’s native msgpack format directly.

FAQ¶

Retry on failure¶

Fluent Bit retries on 5XX and 429 Too Many Requests errors. It will stop reading new log data when its buffer fills and resume when possible. Fluent Bit does not retry on other 4XX errors. See the Fluent Bit Scheduling and Retries documentation for more about retry configuration.