Fluentd¶

Fluentd provides a log processor and forwarder with an extensive plugin ecosystem. Written in Ruby, and is, therefore, less efficient than Fluent Bit. Observe generally recommends using Fluent Bit for most usecases, unless you need a plugin only available for Fluentd.

Before installing Fluentd¶

Before installing Fluentd, prepare the system by following these preinstall instructions.

Installation¶

Fluentd provides detailed installation instructions on their website.

For convenience, Observe provides pointers for the most frequently requested platforms:

Linux

Fluentd distributes td-agent for officially supported distributions:

Alternatively, you can install from ruby gem.

Windows

Fluentd is distributed as td-agent on Windows.

Available as an MSI.

Docker

For Kubernetes deployments, Observe recommends using Observe’s Kubernetes daemonset.

Fluentd maintains and regularly releases container images.

MacOS

Fluentd is available through dmg package.

Configuration¶

The default locations of the configuration file:

Linux/MacOS - /etc/td-agent/td-agent.conf
Windows - C:/opt/td-agent/etc/td-agent/td-agent.conf

The following snippet contains a minimal configuration to send a log file observe.log to Observe:

<source>
  @type tail
  tag logs
  path  /var/log/observe.log
  path_key filename
  <parse>
    @type none
  </parse>
  <buffer>
    chunk_limit_size 2MB
  </buffer>

  pos_file /var/log/td-agent/observe.log.pos
  read_from_head            true
  limit_recently_modified   24h
</source>

<filter **>
  @type record_transformer
  <record>
    hostname "${hostname}"
  </record>
</filter>

<match **>
  @type http
  endpoint https://#{ENV['OBSERVE_CUSTOMER']}.collect.observeinc.com/v1/http/fluentd
  headers {"Authorization": "Bearer #{ENV['OBSERVE_TOKEN']}"}
  <buffer>
    flush_interval 5s
    num_threads 3
  </buffer>
</match>

Observe relies on the Fluentd http output to forward data to the Observe’s HTTP endpoint. Fluentd does not support compression for the http output.

For the tail input plugin, you may want to modify the following attributes:

Fluentd uses the pos_file to track logs processed so far. This allows Fluentd to resume forwarding across restarts without submitting duplicate log entries. The file must be writable by Fluentd.
read_from_head should be enabled if you want to begin ingesting a file from the head rather than tail. This can be useful when bulk uploading files on a first run.
limit_recently_modified restricts the files tailed by Fluentd to those modified recently. This protects against opening too many files concurrently when using a wildcard on a directory with many archived logs.

Note that this sample configuration gets values for username and password from OBSERVE_CUSTOMER and OBSERVE_TOKEN environment variables. Set these to your username and password.

Retry on failure¶

Fluentd supports exponential backoff for retries. See the Fluentd Buffer Plugins documentation for more about buffer configuration.