Buffer and retry OpenTelemetry exports during outages

In cases where Observe by Snowflake may become unavailable for a brief amount of time, you can configure your OTel exporter to buffer the data collected during the outage and try again when Observe by Snowflake becomes available.

• To retain a specific amount of data during an outage, such as 1 hour, set retry_on_failure.max_elapsed_time to the desired duration.

• Use sending_queue with retry_on_failure to survive an outage without losing buffered data, and set queue_size sufficiently large based on your data rate so it can store all observations for the outage window.

• Be sure to reevaluate your heap size requirement to align with the increased values for this custom configuration.

For example, on the otlphttp/observe exporter:

exporters:
    otlphttp/observe:
      retry_on_failure:
        enabled: true
        initial_interval: 5s
        max_interval: 30s
        max_elapsed_time: 300s

See the following documentation for more information:

• See Resiliency in the OpenTelemetry documentation for more information about OTel collector resiliency.
• See the Exporter Helper README for more information about retry on failure.