Observe on Iceberg allows you to use Observe and store your observability data as open Apache Iceberg^™ tables in your own Amazon S3 bucket, offering you the following benefits:

1. You own your data. Your Datasets are written as standard Iceberg tables in your S3 bucket. All existing Observe functionality works as normal, while reading from and writing data to your S3 bucket.
2. Open, engine-independent access. Query your open Iceberg data directly with external query engines, such as Apache Spark^™, DuckDB, Trino, or PyIceberg, without using the Observe query engine. Observe exposes a read-only Iceberg REST Catalog so those engines can discover and read your tables.

Who is this for? Teams that want to retain ownership and portability of their telemetry data, run their own analytics/ML on it with the engine of their choice, and avoid lock-in, while keeping the full Observe experience on top of the same data.

How it works

Your data flows into Iceberg tables in your bucket, and is reachable through Observe or directly through external query engines via the catalog:

flowchart LR
  subgraph Observe
    ING[Ingest pipelines] --> SRC[Source datasets]
    XF[OPAL transformations] --> DER[Derived datasets]
    QE[Observe query engine] --- SRC
    QE --- DER
    CAT[Observe REST API<br>Iceberg catalog endpoints<br>read-only]
  end

  subgraph "Your AWS account"
    S3[(Your S3 bucket<br>Iceberg tables)]
  end

  SRC -- write --> S3
  DER -- write --> S3
  QE -- read/write --> S3

  EXT[External engines<br>Spark, DuckDB, Trino] -- 1. metadata REST --> CAT
  CAT -- table location and schema --> EXT
  EXT -- 2. read data files directly --> S3

The following table defines some of the key concepts:

Data ownership and security

This section describes the data ownership and security measures in place for setting up Observe on Iceberg.

• Your data, your bucket. When a Dataset is stored on a data lake, its Iceberg tables are written to the S3 location you own, and external engines read them from there with your own credentials. Any Datasets that are not placed on a data lake use Observe-managed storage. If you remove the data lake, or revoke the IAM role, Observe loses access to that location.
• Scoped access. Observe accesses your S3 bucket only via the IAM role you create, restricted by your external ID and the permissions you grant.
• Read-only external access. The catalog only serves metadata for read operations; external engines can't modify your tables using the catalog.
• Direct data path. External engines read data files straight from your S3 bucket with your own credentials. Data does not go through Observe for external reads.

Requirements

Verify the following requirements are met:

• Your Observe tenant is enabled for the Observe on Iceberg private preview by Observe.
• You have an AWS account and S3 bucket you control. The private preview supports Amazon S3 only, with one AWS region per Data Lake.
• You have permission to create an IAM role and edit its trust policy in that AWS account.
• For external-engine access, you must have an engine that supports the Iceberg REST catalog, such as Apache Spark 3.x with the Iceberg runtime, DuckDB with the Iceberg extension, Trino, or PyIceberg.
• You have administrator privileges on the Observe platform.

First, create a data lake

A data lake is established with a one-time AWS handshake: you create an IAM role that trusts Observe (scoped by an external ID), then point the data lake at your bucket.

Perform the following steps in Observe to create a new data lake:

1. From the left navigation, select Data & integrations > Data lakes.
2. Click Add data lake.
3. Give your data lake a name and select Amazon S3 as the provider.
4. Click Next. You will see the trust and permission policies provided here to create an IAM role.

Access your IAM Console and perform the following steps to create a new IAM role. First, configure the trust policy:

1. From your AWS Console, click IAM.
2. Click Roles under Access Management.
3. Click Create role.
4. Choose the Custom trust policy card.
5. Copy the trust policy from Observe and paste it into your IAM custom trust policy.
6. Click Next.
7. Click Next again to skip the Permissions page.
8. Give the role a name and description, then click Create role.

Next, configure the permission policy:

1. From your AWS Console, click IAM.
2. Click Roles under Access Management.
3. Find and click the role you just created with the custom trust policy.
4. In the Permissions policies panel, select Add permissions > Create inline policy.
5. Select JSON in the Policy Editor.
6. Copy the permission policy from Observe and paste it into the Policy Editor. The <your-bucket-name> placeholder appears twice in the policy, be sure to replace both placeholders with the actual name of your S3 bucket.
7. Click Next.
8. Give your permission policy a name, then click Create policy.

Perform the rest of the steps in Observe.

1. After you create your IAM role, click Next.

2. In Step 3: connect your storage, provide the following information to grant Observe scoped access to your S3 bucket using the IAM role you just created.

   1. The URL to the S3 bucket where Iceberg data is stored. Use a clean directory without any pre-existing contents. Do not use a sub-directory of an existing data lake.

   2. The ARN of the IAM role you just created. To find this, click on IAM in your AWS Console, then click Roles, then click on your role name.

      

3. Click Create data lake.

Store your Datasets on the new data lake

When you create a Dataset in Observe, you can store the Dataset in the data lake you just created. Both Source Datasets and Derived Datasets can be saved to your data lake:

Perform the following steps to add a Source Dataset to a data lake directly from the Data & integrations menu:

1. From the left navigation rail, select Data & integrations > Add data, then select the Source Dataset card.
2. Select a data type and data lake. If you don't have any data lakes configured yet, you can create one here, then continue with creating a Source Dataset.
3. Specify a package name, Source Dataset name, and provide a description. In the Data Retention field, select how long you want your data to remain in our Snowflake tables.
4. Click Continue.

When you are working with any Dataset on top of a data lake, your daily activities and workflows don't change at all. The only difference is that the data now lives in your S3 bucket as Iceberg tables.

Datasets on separate data lakes are independent. A graph of Datasets built on top of one another doesn't require them to share a data lake, and data lakes don't interfere with each other.

Query your Iceberg tables with external engines

You can point any Iceberg-compatible query engine at the Observe catalog and read your Iceberg tables directly.

What your external query engine needs

Verify the following requirements for you external query engine:

1. Access to the Observe Iceberg REST Catalog (read-only), for metadata.
2. Access to the S3 bucket where the Iceberg table is stored. Your external engine reads the data files directly with its own AWS credentials. For Datasets stored on a data lake, that bucket is your own S3 bucket, so your existing AWS credentials apply.

Observe catalog endpoint and authentication

The Observe catalog is located at the following endpoint:

https://<your-tenant>.observeinc.com/v1/iceberg/catalog

To authenticate, use your existing Observe API token. You don't need a separate credential to use the catalog, and the token lifetime/rotation follow your normal Observe API token settings

Connect Apache Spark

Connect an Apache Spark environment to Observe using an Apache Iceberg REST catalog:

# Register the Observe catalog as a Spark Iceberg REST catalog
spark.sql.catalog.observe                 = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.observe.type            = rest
spark.sql.catalog.observe.uri             = https://<your-tenant>.observeinc.com/v1/iceberg/catalog
spark.sql.catalog.observe.token           = <your-customer-id> <token-secret>
# Data files live in YOUR bucket; Spark reads them with YOUR AWS credentials
spark.sql.catalog.observe.io-impl         = org.apache.iceberg.aws.s3.S3FileIO
spark.sql.extensions                      = org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

Run the following commands from Apache Spark to verify the connection and preview some of your data:

SHOW NAMESPACES IN observe;            -- → iceberg
SHOW TABLES IN observe.iceberg;
SELECT * FROM observe.iceberg.<table> LIMIT 100;

Connect DuckDB

Run the following commands from DuckDB to connect DuckDB to Observe and preview some of your data:

INSTALL iceberg; LOAD iceberg;
-- Attach the Observe REST catalog
ATTACH 'observe' AS observe (
  TYPE ICEBERG,
  ENDPOINT 'https://<your-tenant>.observeinc.com/v1/iceberg/catalog',
  TOKEN '<your-customer-id> <token-secret>'
);
SELECT * FROM observe.iceberg.<table> LIMIT 100;

Catalog API reference

The Observe REST API serves a read-only subset of the Apache Iceberg REST Catalog protocol under the v1/iceberg/catalog endpoint. These are ordinary Observe API endpoints, authenticated with your existing Observe API token.

Typically these API endpoints are called by you Iceberg client, such as your external query engine, but they are documented in the Catalog API reference for reference and troubleshooting.