Snowflake data in Observe¶
Snowflake Data in Observe - Datasets¶
Observe creates a connected graph of the Snowflake management views and observability data in Event Table data in the following datasets:
Event Datasets¶
Event type datasets:
Name |
Purpose |
Links to |
Origin |
|---|---|---|---|
COPY_HISTORY |
Data loading history [COPY INTO and Snowpipe] |
Account, Database, Schema, Table |
|
DATABASE_STORAGE_USAGE_ HISTORY |
Storage usage summary (daily) |
Account, Database |
|
LISTING_AUTO_FULFILLMENT_DATABASE _STORAGE_DAILY |
The data storage used by Cross-Cloud Auto-Fulfillment |
Account |
|
LISTING_AUTO_FULFILLMENT_REFRESH _DAILY |
The data refreshes performed by Cross-Cloud Auto-Fulfillment |
Account |
|
LISTING_AUTO_FULFILLMENT_ USAGE_HISTORY |
Cross-Cloud Auto-Fulfillment costs |
Account, Organizational Account |
|
LISTING_EVENTS_DAILY |
Activity of the marketplace listings |
Account, Marketplace Application |
|
LOAD_HISTORY |
Data loading history [COPY INTO] |
Account, Database, Schema, Table |
|
LOCK_WAIT_HISTORY |
Transaction lock wait history |
Account, Database, Schema, Table, Query |
|
LOGIN_HISTORY |
User logins |
Account, User |
|
MARKETPLACE_DISBURSEMENT_ REPORT |
History of earnings from paid listings in the Snowflake Marketplace |
Account, Marketplace Application |
|
MARKETPLACE_LISTING_ INVOICE_STATUS |
History of invoices related to paid listings in the Snowflake Marketplace |
Account, Marketplace Application |
|
METERING_DAILY_HISTORY |
Snowflake features usage and billing history daily |
Account |
|
SNOWPIPE_STREAMING_ CLIENT_HISTORY |
Snowpipe Streaming client usage history |
Account |
|
STAGE_STORAGE_USAGE_ HISTORY |
Stage storage usage summary (daily) |
Account |
|
STORAGE_USAGE |
Storage usage summary (daily) |
Account |
|
TABLE_STORAGE_METRICS |
Snowflake table-level storage utilization information |
Account, Database, Schema, Table |
|
WAREHOUSE_EVENTS_HISTORY |
Virtual warehouse lifecycle events |
Account, Role, User, Warehouse, Query |
Interval Datasets¶
Interval type datasets:
Name |
Purpose |
Links to |
Origin |
|---|---|---|---|
AUTOMATIC_CLUSTERING_HISTORY |
Automatic clustering usage and billing history |
Account, Database, Schema, Table |
|
COMPLETE_TASK_GRAPHS |
Completed task graph runs usage and history |
Account, Database, Schema, Task |
|
DATA_TRANSFER_HISTORY |
Cloud to cloud data transfer usage history |
Account |
|
DATABASE_REPLICATION_ USAGE_HISTORY |
Database replication history |
Account, Database |
|
DOCUMENT_AI_USAGE_HISTORY |
Document AI service usage history |
Account, Query |
|
DYNAMIC_TABLE_REFRESH_HISTORY |
Dynamic table refresh history |
Account, Database, Schema, Table, Query |
|
EVENT_USAGE_HISTORY |
Event table usage and billing |
Account |
|
HYBRID_TABLE_USAGE_HISTORY |
Hybrid table usage history |
Account, Hybrid Table |
|
INTERNAL_DATA_TRANSFER_HISTORY |
Snowpark Container Services internal data transfers usage history |
Account |
|
LISTING_CONSUMPTION_DAILY |
Snowflake native app or share consumption |
Account, Marketplace Application |
|
LISTING_TELEMETRY_DAILY |
Telemetry of the marketplace listings |
Account |
|
MATERIALIZED_VIEW_REFRESH_HISTORY |
Materialized views refresh billing history |
Account, Database, Schema, Table |
|
METERING_HISTORY |
Snowflake features usage and billing history |
Account |
|
MARKETPLACE_PAID_USAGE_DAILY |
Daily history of your usage of a specific paid listing in the Snowflake Marketplace |
Account |
|
MONETIZED_USAGE_DAILY |
History of daily consumer queries per listing, including charges accumulated for the usage |
Account, Marketplace Application |
|
PIPE_USAGE_HISTORY |
Data load usage and billing [Snowpipe] |
Account, Pipe |
|
QUERY_HISTORY |
Snowflake queries |
Account, Database, Schema, Table, Role, User, Session, Session Activity |
|
REPLICATION_GROUP_REFRESH_HISTORY |
Refresh history for a replication or failover group |
Account, Query |
|
REPLICATION_GROUP_USAGE_HISTORY |
Refresh usage and costs for a replication or failover group |
Account |
|
SEARCH_OPTIMIZATION_BENEFITS |
Search optimization service benefits |
Account, Table |
|
SEARCH_OPTIMIZATION_HISTORY |
Search optimization service usage history |
Account, Table |
|
SERVERLESS_TASK_HISTORY |
Serverless tasks billing and usage history |
Account, Database, Schema |
|
Session |
Snowflake user session |
Account, User |
|
SNOWPARK_CONTAINER_SERVICES _HISTORY |
Snowpark Container Services usage and billing history |
Account |
|
SNOWPIPE_STREAMING_FILE _MIGRATION_HISTORY |
Snowpipe Streaming file to table migration history |
Account, Database, Schema, Table |
|
TABLE_DML_HISTORY |
Table data manipulation history and effects |
Account, Database, Schema, Table |
|
TABLE_PRUNING_HISTORY |
Table partition pruning history |
Account, Database, Schema, Table |
|
TASK_HISTORY |
Task runs usage and history |
Account, Database, Schema, Task, Query |
|
WAREHOUSE_LOAD_HISTORY |
Virtual warehouse workload history |
Account, Warehouse |
|
WAREHOUSE_METERING_HISTORY |
Virtual warehouse usage and billing history |
Account, Warehouse |
Resource Datasets¶
Resource type datasets:
Name |
Purpose |
Links to |
Origin |
|---|---|---|---|
Account |
Snowflake account |
||
Database |
Snowflake database |
Account, Role (owner) |
|
File Format |
Snowflake file format |
Account, Database, Schema, Role (owner) |
|
Function |
Snowflake function |
Account, Database, Schema, Role (owner) |
|
Grant to Role |
Privileges that have been granted to a role |
Account, Role (Grantee), Role (Granted By) |
|
Grant to User |
Roles that have been granted to a user |
Account, Role (Granted), Role (Granted By), User |
|
Hybrid Table |
Snowflake hybrid table |
Account, Database, Schema, Role (owner) |
|
Marketplace Application |
Snowflake Apps installed from listings in marketplace |
Account |
|
Network Policy |
Snowflake network policy |
Account, Role (owner) |
|
Network Rule |
Snowflake network rule |
Account, Database, Schema, Role (owner) |
|
Organization Rate Sheet |
Snowflake Organization rate sheet |
Account |
|
Organizational Account |
Snowflake organization account |
Account |
|
Password Policy |
Snowflake password policy |
Account, Database, Schema, Role (owner) |
|
Pipe |
Snowflake pipe |
Account, Database, Schema, Role (owner) |
|
Procedure |
Snowflake procedure |
Account, Database, Schema, Role (owner) |
|
Role |
Snowflake role |
Account |
|
Schema |
Snowflake schema |
Account, Database, Role (owner) |
|
Secret |
Snowflake secret |
Account, Database, Schema, Role (owner) |
|
Sequence |
Snowflake sequence |
Account, Database, Schema, Role (owner) |
|
Service |
Snowflake compute service |
Account, Database, Schema, Role (owner) |
|
Stage |
Snowflake stage |
Account, Database, Schema, Role (owner) |
|
Table |
Snowflake table |
Account, Database, Schema, Role (owner), User |
|
Task |
Snowflake task |
Account, Database, Schema |
|
User |
Snowflake user |
Account, Role (default), Role (secondary), Role (owner) |
|
View |
Snowflake view |
Account, Database, Schema, Role (owner), User |
|
Warehouse |
Snowflake virtual warehouse |
Account |
Metrics and Sessions¶
Metrics type and sessions datasets:
Name |
Purpose |
Links to |
Origin |
|---|---|---|---|
Automatic Clustering Metering Metrics |
Automatic clustering metering metrics |
Account, Database, Schema, Table |
AUTOMATIC_CLUSTERING_HISTORY |
Compute Pool Metering Metrics |
Snowpark Container Services metering metrics |
Account |
SNOWPARK_CONTAINER_SERVICES_HISTORY |
Data Transfer Metering Metrics |
Data transfer metering metrics |
Account |
DATA_TRANSFER_HISTORY |
Event Table Metering Metrics |
Event table metering metrics |
Account |
EVENT_USAGE_HISTORY |
Hybrid Tables Metering Metrics |
Hybrid table metering metrics |
Account, Hybrid Table |
HYBRID_TABLE_USAGE_HISTORY |
Query Metrics |
Query metrics |
Account, Database, Schema, Table, Role, User, Session, Session Activity |
QUERY_HISTORY |
Serverless Task Metering Metrics |
Serverless task metering metrics |
Account, Database, Schema |
SERVERLESS_TASK_HISTORY |
Service Metering Metrics |
All Snowflake services metering metrics |
Account, Table, Pipe, Warehouse |
METERING_HISTORY |
Snowpipe Metering Metrics |
Snowpipe streaming metering metrics |
Account, Pipe |
PIPE_USAGE_HISTORY |
Table Storage Metrics |
Snowflake table-level storage utilization information as metrics |
Account, Database, Schema, Table |
TABLE_STORAGE_METRICS |
Warehouse Metering Metrics |
Virtual warehouse metering metrics |
Account, Warehouse |
WAREHOUSE_METERING_HISTORY |
Session Activity |
Snowflake user session queries activity |
Account, User |
QUERY_HISTORY |
Snowflake Event Table Datasets¶
Event table related datasets contain data from Snowflake ❄️Event Table:
Name |
Purpose |
Links to |
Origin |
|---|---|---|---|
Event Table Event |
Snowflake event table - events |
Account, Database, Schema |
RECORD_TYPE=EVENT |
Event Table Logs |
Snowflake event table - logs |
Account, Database, Schema, Warehouse, Role, User, Query |
RECORD_TYPE=LOG |
Event Table Metrics |
Snowflake event table - metrics |
Account, Database, Schema |
RECORD_TYPE=METRICS |
Event Table OTEL Metrics |
Snowflake event table - metrics (via OTEL endpoint) |
Account |
RECORD_TYPE=METRICS |
Event Table Spans |
Snowflake event table - spans |
Account, Database, Schema, Warehouse, Role, User, Query |
RECORD_TYPE=SPAN |
Span Event |
Snowflake event table - span events |
Account, Database, Schema, Warehouse, Role, User, Query, Event Table, Span |
RECORD_TYPE=SPAN_EVENT |
Dashboards¶
Observe provides following dashboards for display of Snowflake data:
Name |
Purpose |
Uses |
|---|---|---|
Account |
Summary of objects in account and some core metrics |
Account, Object views, QUERY_HISTORY |
Data Export Activity |
Create external integration, create stage and COPY commands |
QUERY_HISTORY |
Database |
Summary of objects in database and some core metrics, including queries |
Account, Object views, Metric datasets, QUERY_HISTORY |
Login Activity |
View into all the user logins |
LOGIN_HISTORY |
Pipe |
Consumption and usage for pipes |
Object views, Metric datasets |
Query History |
Overview of query activity |
QUERY_HISTORY |
Query History Metrics |
Overview of query activity via precomputed metrics |
Query Metrics |
Schema |
Summary of objects in schema and some core metrics, including queries |
Account, Object views, Metric datasets, QUERY_HISTORY |
Service Metering |
Usage of all compute features |
SERVICE_METERING |
Session Activity |
Overview of session activity |
Session |
Table |
Summary of objects in schema and some core metrics, including queries |
Account, Object views, Metric datasets, QUERY_HISTORY |
Warehouse |
Warehouse usage and activity |
Warehouse, WAREHOUSE_METERING_HISTORY |
Snowflake Data in Observe - Metrics¶
Metering Metrics¶
These service metering metrics provide details about consumption of compute and storage:
Dataset |
Name |
|---|---|
Automatic Clustering Metering Metrics |
snowflake.metering.automatic_clustering.credits_used |
Automatic Clustering Metering Metrics |
snowflake.metering.automatic_clustering.num_bytes_reclustered |
Automatic Clustering Metering Metrics |
snowflake.metering.automatic_clustering.num_rows_reclustered |
Compute Pool Metering Metrics |
snowflake.metering.compute_pool.credits_used |
Data Transfer Metering Metrics |
snowflake.metering.data_transfer.bytes_transferred |
Event Table Metering Metrics |
snowflake.metering.event_table.bytes_ingested |
Event Table Metering Metrics |
snowflake.metering.event_table.credits_used |
Hybrid Tables Metering Metrics |
snowflake.metering.hybrid_tables.credits_used |
Serverless Task Metering Metrics |
snowflake.metering.serverless_task.credits_used |
Service Metering Metrics |
snowflake.metering.bytes |
Service Metering Metrics |
snowflake.metering.credits_cloud |
Service Metering Metrics |
snowflake.metering.credits_compute |
Service Metering Metrics |
snowflake.metering.credits_total |
Service Metering Metrics |
snowflake.metering.files |
Service Metering Metrics |
snowflake.metering.rows |
Snowpipe Metering Metrics |
snowflake.metering.pipe.bytes_inserted |
Snowpipe Metering Metrics |
snowflake.metering.pipe.credits_used |
Snowpipe Metering Metrics |
snowflake.metering.pipe.files_inserted |
Table Storage Metrics |
snowflake.storage.active_bytes |
Table Storage Metrics |
snowflake.storage.failsafe_bytes |
Table Storage Metrics |
snowflake.storage.retained_for_clone_bytes |
Table Storage Metrics |
snowflake.storage.time_travel_bytes |
Warehouse Metering Metrics |
snowflake.metering.warehouse.credits_cloud |
Warehouse Metering Metrics |
snowflake.metering.warehouse.credits_compute |
Warehouse Metering Metrics |
snowflake.metering.warehouse.credits_total |
Query History Metrics¶
These metrics are derived from ❄️QUERY_HISTORY view:
Dataset |
Name |
|---|---|
Query Metrics |
snowflake.query.bytes_deleted.count |
Query Metrics |
snowflake.query.bytes_deleted.max |
Query Metrics |
snowflake.query.bytes_deleted.min |
Query Metrics |
snowflake.query.bytes_deleted.sum |
Query Metrics |
snowflake.query.bytes_read_from_result.count |
Query Metrics |
snowflake.query.bytes_read_from_result.max |
Query Metrics |
snowflake.query.bytes_read_from_result.min |
Query Metrics |
snowflake.query.bytes_read_from_result.sum |
Query Metrics |
snowflake.query.bytes_scanned.count |
Query Metrics |
snowflake.query.bytes_scanned.max |
Query Metrics |
snowflake.query.bytes_scanned.min |
Query Metrics |
snowflake.query.bytes_scanned.sum |
Query Metrics |
snowflake.query.bytes_sent_over_network.count |
Query Metrics |
snowflake.query.bytes_sent_over_network.max |
Query Metrics |
snowflake.query.bytes_sent_over_network.min |
Query Metrics |
snowflake.query.bytes_sent_over_network.sum |
Query Metrics |
snowflake.query.bytes_spilled_local.count |
Query Metrics |
snowflake.query.bytes_spilled_local.max |
Query Metrics |
snowflake.query.bytes_spilled_local.min |
Query Metrics |
snowflake.query.bytes_spilled_local.sum |
Query Metrics |
snowflake.query.bytes_spilled_remote.count |
Query Metrics |
snowflake.query.bytes_spilled_remote.max |
Query Metrics |
snowflake.query.bytes_spilled_remote.min |
Query Metrics |
snowflake.query.bytes_spilled_remote.sum |
Query Metrics |
snowflake.query.bytes_written_to_result.count |
Query Metrics |
snowflake.query.bytes_written_to_result.max |
Query Metrics |
snowflake.query.bytes_written_to_result.min |
Query Metrics |
snowflake.query.bytes_written_to_result.sum |
Query Metrics |
snowflake.query.bytes_written.count |
Query Metrics |
snowflake.query.bytes_written.max |
Query Metrics |
snowflake.query.bytes_written.min |
Query Metrics |
snowflake.query.bytes_written.sum |
Query Metrics |
snowflake.query.child_queries_wait_time.count |
Query Metrics |
snowflake.query.child_queries_wait_time.max |
Query Metrics |
snowflake.query.child_queries_wait_time.min |
Query Metrics |
snowflake.query.child_queries_wait_time.sum |
Query Metrics |
snowflake.query.compilation_time.count |
Query Metrics |
snowflake.query.compilation_time.max |
Query Metrics |
snowflake.query.compilation_time.min |
Query Metrics |
snowflake.query.compilation_time.sum |
Query Metrics |
snowflake.query.execution_time.count |
Query Metrics |
snowflake.query.execution_time.max |
Query Metrics |
snowflake.query.execution_time.min |
Query Metrics |
snowflake.query.execution_time.sum |
Query Metrics |
snowflake.query.list_external_files_time.count |
Query Metrics |
snowflake.query.list_external_files_time.max |
Query Metrics |
snowflake.query.list_external_files_time.min |
Query Metrics |
snowflake.query.list_external_files_time.sum |
Query Metrics |
snowflake.query.queued_overload_time.count |
Query Metrics |
snowflake.query.queued_overload_time.max |
Query Metrics |
snowflake.query.queued_overload_time.min |
Query Metrics |
snowflake.query.queued_overload_time.sum |
Query Metrics |
snowflake.query.queued_provisioning_time.count |
Query Metrics |
snowflake.query.queued_provisioning_time.max |
Query Metrics |
snowflake.query.queued_provisioning_time.min |
Query Metrics |
snowflake.query.queued_provisioning_time.sum |
Query Metrics |
snowflake.query.queued_repair_time.count |
Query Metrics |
snowflake.query.queued_repair_time.max |
Query Metrics |
snowflake.query.queued_repair_time.min |
Query Metrics |
snowflake.query.queued_repair_time.sum |
Query Metrics |
snowflake.query.rows_deleted.count |
Query Metrics |
snowflake.query.rows_deleted.max |
Query Metrics |
snowflake.query.rows_deleted.min |
Query Metrics |
snowflake.query.rows_deleted.sum |
Query Metrics |
snowflake.query.rows_inserted.count |
Query Metrics |
snowflake.query.rows_inserted.max |
Query Metrics |
snowflake.query.rows_inserted.min |
Query Metrics |
snowflake.query.rows_inserted.sum |
Query Metrics |
snowflake.query.rows_produced.count |
Query Metrics |
snowflake.query.rows_produced.max |
Query Metrics |
snowflake.query.rows_produced.min |
Query Metrics |
snowflake.query.rows_produced.sum |
Query Metrics |
snowflake.query.rows_unloaded.count |
Query Metrics |
snowflake.query.rows_unloaded.max |
Query Metrics |
snowflake.query.rows_unloaded.min |
Query Metrics |
snowflake.query.rows_unloaded.sum |
Query Metrics |
snowflake.query.rows_updated.count |
Query Metrics |
snowflake.query.rows_updated.max |
Query Metrics |
snowflake.query.rows_updated.min |
Query Metrics |
snowflake.query.rows_updated.sum |
Query Metrics |
snowflake.query.rows_written_to_result.count |
Query Metrics |
snowflake.query.rows_written_to_result.max |
Query Metrics |
snowflake.query.rows_written_to_result.min |
Query Metrics |
snowflake.query.rows_written_to_result.sum |
Query Metrics |
snowflake.query.total_elapsed_time.count |
Query Metrics |
snowflake.query.total_elapsed_time.max |
Query Metrics |
snowflake.query.total_elapsed_time.min |
Query Metrics |
snowflake.query.total_elapsed_time.sum |
Query Metrics |
snowflake.query.transaction_blocked_time.count |
Query Metrics |
snowflake.query.transaction_blocked_time.max |
Query Metrics |
snowflake.query.transaction_blocked_time.min |
Query Metrics |
snowflake.query.transaction_blocked_time.sum |
Event Table Metrics - Platform¶
Snowflake ❄️ Platform Metrics will be visible in Observe in either Event Table Metrics or Event Table OTEL Metrics datasets:
Dataset |
Name |
|---|---|
Event Table Metrics |
container.cpu.limit |
Event Table Metrics |
container.cpu.requested |
Event Table Metrics |
container.cpu.usage |
Event Table Metrics |
container.gpu.limit |
Event Table Metrics |
container.gpu.memory.capacity |
Event Table Metrics |
container.gpu.memory.usage |
Event Table Metrics |
container.gpu.requested |
Event Table Metrics |
container.gpu.utilization |
Event Table Metrics |
container.memory.limit |
Event Table Metrics |
container.memory.requested |
Event Table Metrics |
container.memory.usage |
Event Table Metrics |
container.restarts |
Event Table Metrics |
container.state.finished |
Event Table Metrics |
container.state.last.finished.reason |
Event Table Metrics |
container.state.pending |
Event Table Metrics |
container.state.pending.reason |
Event Table Metrics |
container.state.running |
Event Table Metrics |
container.state.started |
Event Table Metrics |
process.cpu.utilization |
Event Table Metrics |
process.memory.usage |
Event Table Metrics |
network.egress.received.bytes |
Event Table Metrics |
network.egress.received.packets |
Event Table Metrics |
network.egress.transmitted.bytes |
Event Table Metrics |
network.egress.transmitted.packets |
Event Table Metrics |
volume.capacity |
Event Table Metrics |
volume.io.inflight |
Event Table Metrics |
volume.read.iops |
Event Table Metrics |
volume.read.throughput |
Event Table Metrics |
volume.usage |
Event Table Metrics |
volume.write.iops |
Event Table Metrics |
volume.write.throughput |
Event Table Metrics - Custom¶
If you have an application that emits its own ❄️ Application Metrics to the Event Table, you will see these metrics show up in either Event Table Metrics or Event Table OTEL Metrics datasets.