Fluent Bit

Fluent Bit provides lightweight log processor and forwarder services.

Installation

Fluent Bit provides detailed installation instructions on their website. For convenience, Observe provides pointers for the most frequently requested platforms:

Fluent Bit distributes td-agent-bit for officially supported distributions:

Alternatively, you can build from source.

Ensure you download Fluent Bit for Windows 1.9.1 or higher as older versions require a manual root certificate installation process.

  1. Install the appropriate Fluent Bit td-agent-bit package, available at https://docs.fluentbit.io/manual/installation/windows.

Fluent Bit maintains and regularly releases container images:

$ docker run -ti fluent/fluent-bit:1.7
Fluent Bit v1.7.2
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[ info] [engine] started (pid=1)
[ info] [storage] version=1.1.1, initializing...
[ info] [storage] in-memory
[ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[ info] [sp] stream processor started

You can obtain Fluent Bit through homebrew:

$ brew install fluent-bit

Configuration

The following snippet contains a minimal configuration to send data to Observe. It uses the dummy input plugin that generates sample events. (See Inputs in the Fluent Bit docs for a list of available plugins.)

Note

We strongly recommend that you configure a file location for the Fluent bit service logs, in order to assist with debugging potential ingest issues. You can do so by specifying a host file location of your choice in the [SERVICE] section of the Fluent Bit configuration. Example:

[SERVICE]
 ...
 log_file    some/host/path.log

[SERVICE]
    flush        5
    daemon       Off
    log_level    info

# Uncomment the below section if using AWS EC2
#[FILTER]
#    Name aws
#    Match *
#    imds_version v1
#    az true
#    ec2_instance_id true
#    ec2_instance_type true
#    account_id true
#    hostname true
#    vpc_id true

[INPUT]
    name  dummy
    dummy {"message":"Hello World!"}
    rate  1

[OUTPUT]
    name         http
    match        *
    host         ${OBSERVE_CUSTOMER}.collect.observeinc.com
    port         443
    tls          on

    uri          /v1/http/fluentbit

    format       msgpack
    header       Authorization     Bearer ${OBSERVE_TOKEN}
    header       X-Observe-Decoder fluent
    compress     gzip

This [INPUT] section reads from a log file using the tail plugin:

[INPUT]
    name        tail
    path        /var/log/my/service/*.log
    path_key    filename

    # Additional options for logs with very long lines
    # buffer_max_size  256k
    # skip_long_lines  on

For additional input plugins, see the Fluent Bit Inputs documentation.

We rely on Fluent Bit’s http output to forward data to Observe’s HTTP endpoint. We can export data in Fluent Bit’s native msgpack format directly.

FAQ

Retry on failure

Fluent Bit retries on 5XX and 429 Too Many Requests errors. It stops reading new log data when the buffer fills and resumes when possible. Fluent Bit does not retry on other 4XX errors. See the Fluent Bit Scheduling and Retries documentation for more about retry configuration.