Fluent Bit¶
Fluent Bit is a lightweight log processor and forwarder.
Installation¶
Fluent Bit provides detailed installation instructions on their website. For convenience, we provide pointers for the most frequently requested platforms:
Fluent Bit distributes td-agent-bit
for officially supported distributions:
Alternatively, you can build from source.
Please ensure you download Fluent Bit for Windows 1.9.1 or higher - older versions require a manual root certificate installation process.
Install the appropriate Fluent Bit
td-agent-bit
package, available at https://docs.fluentbit.io/manual/installation/windows.
Fluent Bit maintains and regularly releases container images:
$ docker run -ti fluent/fluent-bit:1.7
Fluent Bit v1.7.2
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[ info] [engine] started (pid=1)
[ info] [storage] version=1.1.1, initializing...
[ info] [storage] in-memory
[ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[ info] [sp] stream processor started
Fluent Bit is available through homebrew:
$ brew install fluent-bit
Configuration¶
The following snippet contains a minimal configuration to send data to Observe. It uses the dummy
input plugin that generates sample events. (See Inputs in the Fluent Bit docs for a list of available plugins.)
[SERVICE]
flush 5
daemon Off
log_level info
# Uncomment the below section if using AWS EC2
#[FILTER]
# Name aws
# Match *
# imds_version v1
# az true
# ec2_instance_id true
# ec2_instance_type true
# account_id true
# hostname true
# vpc_id true
[INPUT]
name dummy
dummy {"message":"Hello World!"}
rate 1
[OUTPUT]
name http
match *
host collect.observeinc.com
port 443
tls on
http_user ${OBSERVE_CUSTOMER}
http_passwd ${OBSERVE_TOKEN}
uri /v1/http/fluentbit
format msgpack
header X-Observe-Decoder fluent
compress gzip
This [INPUT]
section reads from a log file using the tail
plugin:
[INPUT]
name tail
path /var/log/my/service/*.log
path_key filename
# Additional options for logs with very long lines
# buffer_max_size 256k
# skip_long_lines on
For additional input plugins, see the Fluent Bit Inputs documentation.
We rely on Fluent Bit’s http
output to forward data to Observe’s HTTP endpoint.
We can export data in Fluent Bit’s native msgpack format directly.
FAQ¶
Retry on failure¶
Fluent Bit retries on 5XX and 429 Too Many Requests
errors. It will stop reading new log data when its buffer fills and resume when possible. Fluent Bit does not retry on other 4XX errors. See the Fluent Bit Scheduling and Retries documentation for more about retry configuration.