Fluent Bit provides lightweight log processor and forwarder services.

Installation

Fluent Bit provides detailed installation instructions on the FluentBit website. For convenience, Observe provides pointers for the most frequently requested platforms:

Configuration

The following snippet contains a minimal configuration to send data to Observe. It uses the dummy input plugin that generates sample events. (See Inputs in the Fluent Bit docs for a list of available plugins.)

Observe strongly recommends configuring a file location for the Fluent bit service logs to assist with debugging potential ingest issues. You can do so by specifying a host file location of your choice in the [SERVICE] section of the Fluent Bit configuration. Example:

[SERVICE]
 ...
 log_file    some/host/path.log

[SERVICE]
    flush        5
    daemon       Off
    log_level    info

# Uncomment the below section if using AWS EC2
#[FILTER]
#    Name aws
#    Match *
#    imds_version v1
#    az true
#    ec2_instance_id true
#    ec2_instance_type true
#    account_id true
#    hostname true
#    vpc_id true

[INPUT]
    name  dummy
    dummy {"message":"Hello World!"}
    rate  1

[OUTPUT]
    name         http
    match        *
    host         ${OBSERVE_CUSTOMER}.collect.observeinc.com
    port         443
    tls          on

    uri          /v1/http/fluentbit

    format       msgpack
    header       Authorization     Bearer ${OBSERVE_TOKEN}
    header       X-Observe-Decoder fluent
    compress     gzip

The uri field in the [OUTPUT] section can be customized to help identify data properly. You can add words and slashes after http. For instance, /v1/http/fluentbit/prodsys and /v1/http/cali/yolo/davis are valid.

This [INPUT] section reads from a log file using the tail plugin:

[INPUT]
    name        tail
    path        /var/log/my/service/*.log
    path_key    filename

    # Additional options for logs with very long lines
    buffer_max_size  2MB
    skip_long_lines  on
    mem_buf_limit 10MB

    # Ignores files older than ignore_older.
    ignore_older "2d"

For additional input plugins, see the Fluent Bit Inputs documentation.

Observe relies on Fluent Bit's http output to forward data to the HTTP endpoint. Observe can export data in Fluent Bit's native msgpack format directly. Observe's Fluent Bit configurations compress output by default.

FAQ

Retry on failure

Fluent Bit retries on 5XX and 429 Too Many Requests errors. Fluent Bit does not retry on other 4XX errors. It stops reading new log data when the buffer fills and resumes when possible. See the Fluent Bit Scheduling and Retries documentation for more about retry configuration.