Observe for Snowflake Components

O4S consists of two main components:

• First, a data gathering application that sends data from Snowflake to Observe

• Second, a data processing application/integration in Observe for exploring using dashboards, logs, metrics and traces

O4S Application in Snowflake - Send Data

Snowflake management and observability data is gathered from SNOWFLAKE shared database and event table(s) using Snowflake for Observe native application available from Snowflake Marketplace https://app.snowflake.com/marketplace/listing/GZTYZY3AR0U/observe-inc-observe-for-snowflake.

Snowflake for Observe native application uses scheduled tasks executed by virtual warehouse OR serverless tasks to send records from SNOWFLAKE shared database views and Event Tables to Observe.

In order to send data from Snowflake, Snowflake for Observe native app uses External Integration which binds with Network Rule (opening access to Observe ingest endpoint) with a pair of Secrets (storing Observe ingest endpoint and ingest token variables). These are stored in a small database with just those configuration values, and no data.

The data is collected by scheduled tasks, with schedules are adjustable for each object:

• History views (like QUERY_HISTORY, LOGIN_HISTORY, METERING_HISTORY and others) are collected in ongoing manner by keeping a last-retrieved watermark from the timestamp column in the view (like END_TIME for QUERY_HISTORY or EVENT_TIMESTAMP for LOGIN_HISTORY)

• Object views (like DATABASES, TABLES, USERS and others) are collected in their entirety while filtering to only active items (rows where DELETED_ON column is null)

• Event tables, custom tables and views are collected by creating a CDC stream on their contents and consuming new records on regular interval

The data for rows is packaged into JSON objects using OBJECT_CONSTRUCT, bundled into batches of ~8MB in size and sent to Observe HTTP endpoint with decorations that are expected by Snowflake app/integration in Observe.

Optionally, for Event Table rows that represent metrics (RECORD_TYPE=METRIC), the data is converted to OpenTelemetry metric protobuf format, packaged into sizeable batches and sent to Observe OpenTelemetry endpoint.

O4S Application in Observe - Management Interface

Data sent from Snowflake by Snowflake for Observe native application is processed by Snowflake app/integration in Observe.

Using Observe data transformation features, Snowflake app/integration builds a curated data graph of connected datasets, including things like query and login history, usage and billing metrics, and traces.

The full list is visible in the Snowflake app/integration.

The app also provides prebuilt dashboards for common Snowflake management features as well as logs (query history, task history), metrics (billing, data loading, data transfer, query scans) and traces (Snowpark code, SPCS, cortex, UDFs).