---
updatedAt: 2026-09-10T19:22:24.000Z
agentTools:
  projectIndex: https://docs.observeinc.com/llms.txt
---

# Fluentd

Fluentd provides a log processor and forwarder with an extensive plugin ecosystem. Fluentd is less efficient than [Fluent Bit](https://docs.observeinc.com/docs/fluent-bit). Observe by Snowflake generally recommends using Fluent Bit for most use cases, unless you need a plugin only available for Fluentd.

# Pre-install instructions

Before installing Fluentd, prepare the system by following the [Before Installation](https://docs.fluentd.org/installation/before-install) instructions in the Fluentd documentation.

# Install Fluentd

Fluentd provides detailed installation instructions in their [documentation](https://docs.fluentd.org/).

For convenience, Observe by Snowflake provides pointers for the most frequently requested platforms:

<Tabs>
  <Tab title="Linux">
    Fluentd distributes `td-agent` for officially [supported distributions](https://docs.fluentd.org/installation):

    * [Amazon Linux](https://docs.fluentd.org/installation/install-by-rpm#amazon-linux)
    * [Red Hat / CentOS](https://docs.fluentd.org/installation/install-by-rpm)
    * [Debian/Ubuntu](https://docs.fluentd.org/installation/install-by-deb)

    Alternatively, you can install from [ruby gem](https://docs.fluentd.org/installation/install-by-gem).
  </Tab>

  <Tab title="Windows">
    Fluentd is distributed as `td-agent` on Windows.

    Available as an [MSI](https://docs.fluentd.org/installation/install-by-msi).
  </Tab>

  <Tab title="Docker">
    For Kubernetes deployments, Observe recommends using Observe's [Kubernetes daemonset](../../integrations/kubernetes/kubernetes).

    Fluentd maintains and regularly releases [container images](https://docs.fluentd.org/container-deployment).
  </Tab>

  <Tab title="macOS">
    Fluentd is available through [dmg package](https://docs.fluentd.org/installation/install-by-dmg).
  </Tab>
</Tabs>

# Configure Fluentd

The default locations of the configuration file:

* Linux/MacOS - `/etc/td-agent/td-agent.conf`
* Windows - `C:/opt/td-agent/etc/td-agent/td-agent.conf`

The following snippet contains a minimal configuration to send a log file `observe.log` to Observe by Snowflake:

```text
<source>
  @type tail
  tag logs
  path  /var/log/observe.log
  path_key filename
  <parse>
    @type none
  </parse>
  <buffer>
    chunk_limit_size 2MB
  </buffer>

  pos_file /var/log/td-agent/observe.log.pos
  read_from_head            true
  limit_recently_modified   24h
</source>

<filter **>
  @type record_transformer
  <record>
    hostname "${hostname}"
  </record>
</filter>

<match **>
  @type http
  endpoint https://#{ENV['OBSERVE_CUSTOMER']}.collect.observeinc.com/v1/http/fluentd
  headers {"Authorization": "Bearer #{ENV['OBSERVE_TOKEN']}"}
  <buffer>
    flush_interval 5s
    num_threads 3
  </buffer>
</match>
```

Observe by Snowflake relies on the **Fluentd** [`http`](https://docs.fluentd.org/output/http) output to forward data to the Observe by Snowflake's [HTTP endpoint](https://docs.observeinc.com/docs/http). Fluentd does not support compression for the `http` output.

For the [tail input plugin](https://docs.fluentd.org/input/tail), you may want to modify the following attributes:

* **Fluentd** uses the `pos_file` to track logs processed so far. This allows **Fluentd** to resume forwarding across restarts without submitting duplicate log entries. The file must be writable by **Fluentd**.
* `read_from_head` should be enabled if you want to begin ingesting a file from the head rather than tail. This can be useful when bulk uploading files on a first run.
* `limit_recently_modified` restricts the files tailed by **Fluentd** to those modified recently. This protects against opening too many files concurrently when using a wildcard on a directory with many archived logs.

Note that this sample configuration gets values for `username` and `password` from `OBSERVE_CUSTOMER` and `OBSERVE_TOKEN` environment variables. Set these to your username and password.

# Retry on failure

**Fluentd** supports exponential back off for retries. See the [Fluentd Buffer Plugins](https://docs.fluentd.org/buffer) documentation for more about buffer configuration.

<br />