Fluentd¶
Fluentd is a log processor and forwarder with an extensive plugin ecosystem. It is written in Ruby, and is therefore less efficient than Fluent Bit. We generally recommend using Fluent Bit for most usecases, unless you need a plugin only available for Fluentd.
Before Installation¶
Before installing Fluentd, prepare the system by following preinstall instructions
Installation¶
Fluentd provides detailed installation instructions on their website.
For convenience, we provide pointers for the most frequently requested platforms:
Fluentd distributes td-agent
for officially supported distributions:
Alternatively, you can install from ruby gem.
Fluentd is distributed as td-agent
on Windows.
Available as an MSI.
For Kubernetes deployments, it’s recommended to use Observe’s Kubernetes daemonset.
Fluentd maintains and regularly releases container images.
Fluentd is available through dmg package.
Configuration¶
The default locations of the configuration file are:
Linux/MacOS - /etc/td-agent/td-agent.conf
Windows - C:/opt/td-agent/etc/td-agent/td-agent.conf
The following snippet contains a minimal configuration to send a log file observe.log to Observe:
<source>
@type tail
tag logs
path /var/log/observe.log
path_key filename
<parse>
@type none
</parse>
<buffer>
chunk_limit_size 2MB
</buffer>
pos_file /var/log/td-agent/observe.log.pos
read_from_head true
limit_recently_modified 24h
</source>
<filter **>
@type record_transformer
<record>
hostname "${hostname}"
</record>
</filter>
<match **>
@type http
endpoint https://collect.observeinc.com/v1/http/fluentd
<auth>
method basic
username "#{ENV['OBSERVE_CUSTOMER']}"
password "#{ENV['OBSERVE_TOKEN']}"
</auth>
<buffer>
flush_interval 5s
num_threads 3
</buffer>
</match>
We rely on Fluentd’s http
output to forward data to Observe’s HTTP endpoint. Fluentd does not support compression for the http
output.
For the tail input plugin, you may wish to modify the following attributes:
pos_file
is used by Fluentd to track logs processed so far. This allows Fluentd to resume forwarding across restarts without submitting duplicate log entries. The file must be writable by Fluentd.read_from_head
should be enabled if you wish to begin ingesting a file from the head rather than tail. This can be useful when bulk uploading files on a first run.limit_recently_modified
restricts the files which are tailed by Fluentd to those modified recently. This protects against opening too many files concurrently when using a wildcard on a directory with many archived logs.
Note that this sample config gets values for username
and password
from OBSERVE_CUSTOMER
and OBSERVE_TOKEN
environment variables. Set these to your username and password.
FAQ¶
Retry on failure¶
Fluentd supports exponential backoff for retries. See the Fluentd Buffer Plugins documentation for more about buffer configuration.