Fluentd

Fluentd is a log processor and forwarder with an extensive plugin ecosystem. It is written in Ruby, and is therefore less efficient than Fluent Bit. We generally recommend using Fluent Bit for most usecases, unless you need a plugin only available for Fluentd.

Before Installation

Before installing Fluentd, prepare the system by following preinstall instructions

Installation

Fluentd provides detailed installation instructions on their website.

For convenience, we provide pointers for the most frequently requested platforms:

Fluentd distributes td-agent for officially supported distributions:

Alternatively, you can install from ruby gem.

Fluentd is distributed as td-agent on Windows.

Available as an MSI.

For Kubernetes deployments, it’s recommended to use Observe’s Kubernetes daemonset.

Fluentd maintains and regularly releases container images.

Fluentd is available through dmg package.

Configuration

The default locations of the configuration file are:

  • Linux/MacOS - /etc/td-agent/td-agent.conf

  • Windows - C:/opt/td-agent/etc/td-agent/td-agent.conf

The following snippet contains a minimal configuration to send a log file observe.log to Observe:

<source>
  @type tail
  tag logs
  path  /var/log/observe.log
  path_key filename
  <parse>
    @type none
  </parse>
  <buffer>
    chunk_limit_size 2MB
  </buffer>

  pos_file /var/log/td-agent/observe.log.pos
  read_from_head            true
  limit_recently_modified   24h
</source>

<filter **>
  @type record_transformer
  <record>
    hostname "${hostname}"
  </record>
</filter>

<match **>
  @type http
  endpoint https://collect.observeinc.com/v1/http/fluentd
  <auth>
    method basic
    username "#{ENV['OBSERVE_CUSTOMER']}"
    password "#{ENV['OBSERVE_TOKEN']}"
  </auth>
  <buffer>
    flush_interval 5s
    num_threads 3
  </buffer>
</match>

We rely on Fluentd’s http output to forward data to Observe’s HTTP endpoint. Fluentd does not support compression for the http output.

For the tail input plugin, you may wish to modify the following attributes:

  • pos_file is used by Fluentd to track logs processed so far. This allows Fluentd to resume forwarding across restarts without submitting duplicate log entries. The file must be writable by Fluentd.

  • read_from_head should be enabled if you wish to begin ingesting a file from the head rather than tail. This can be useful when bulk uploading files on a first run.

  • limit_recently_modified restricts the files which are tailed by Fluentd to those modified recently. This protects against opening too many files concurrently when using a wildcard on a directory with many archived logs.

Note that this sample config gets values for username and password from OBSERVE_CUSTOMER and OBSERVE_TOKEN environment variables. Set these to your username and password.

FAQ

Retry on failure

Fluentd supports exponential backoff for retries. See the Fluentd Buffer Plugins documentation for more about buffer configuration.