Fluentd

Fluentd provides a log processor and forwarder with an extensive plugin ecosystem. It is written in Ruby, and is therefore less efficient than Fluent Bit. We generally recommend using Fluent Bit for most usecases, unless you need a plugin only available for Fluentd.

Before installing Fluentd

Before installing Fluentd, prepare the system by following preinstall instructions

Installation

Fluentd provides detailed installation instructions on their website.

For convenience, Observe provides pointers for the most frequently requested platforms:

Fluentd distributes td-agent for officially supported distributions:

Alternatively, you can install from ruby gem.

Fluentd is distributed as td-agent on Windows.

Available as an MSI.

For Kubernetes deployments, Observe recommends using Observe’s Kubernetes daemonset.

Fluentd maintains and regularly releases container images.

Fluentd is available through dmg package.

Configuration

The default locations of the configuration file:

  • Linux/MacOS - /etc/td-agent/td-agent.conf

  • Windows - C:/opt/td-agent/etc/td-agent/td-agent.conf

The following snippet contains a minimal configuration to send a log file observe.log to Observe:

<source>
  @type tail
  tag logs
  path  /var/log/observe.log
  path_key filename
  <parse>
    @type none
  </parse>
  <buffer>
    chunk_limit_size 2MB
  </buffer>

  pos_file /var/log/td-agent/observe.log.pos
  read_from_head            true
  limit_recently_modified   24h
</source>

<filter **>
  @type record_transformer
  <record>
    hostname "${hostname}"
  </record>
</filter>

<match **>
  @type http
  endpoint https://#{ENV['OBSERVE_CUSTOMER']}.collect.observeinc.com/v1/http/fluentd
  headers {"Authorization": "Bearer #{ENV['OBSERVE_TOKEN']}"}
  <buffer>
    flush_interval 5s
    num_threads 3
  </buffer>
</match>

Observe relies on the Fluentd http output to forward data to the Observe’s HTTP endpoint. Fluentd does not support compression for the http output.

For the tail input plugin, you may want to modify the following attributes:

  • Fluentd uses the pos_file to track logs processed so far. This allows Fluentd to resume forwarding across restarts without submitting duplicate log entries. The file must be writable by Fluentd.

  • read_from_head should be enabled if you want to begin ingesting a file from the head rather than tail. This can be useful when bulk uploading files on a first run.

  • limit_recently_modified restricts the files which are tailed by Fluentd to those modified recently. This protects against opening too many files concurrently when using a wildcard on a directory with many archived logs.

Note that this sample configuration gets values for username and password from OBSERVE_CUSTOMER and OBSERVE_TOKEN environment variables. Set these to your username and password.

Retry on failure

Fluentd supports exponential backoff for retries. See the Fluentd Buffer Plugins documentation for more about buffer configuration.