Send Data from Snowflake to Observe¶

Send Data from Observe Snowflake Application¶

Send Data from Observe for Snowflake App - Via UI¶

Snowflake for Observe native application is ready to send data to Observe. Now you need to specify which management views and tables to send.

Log into your Snowflake instance in Snowsight console
Change to the ACCOUNTADMIN role
Navigate to Data Products\Apps
Click Observe for Snowflake app
In ACCOUNT_USAGE History section, choose the view you want in Add History View dropdown and click Add button
- You can add all views using Add All button
- Note: some views require Snowflake Enterprise Edition or higher
- Note: not all views may contain data, the contents of many depend on your usage of those features

Observe For Snowflake - Add History View

In ACCOUNT_USAGE Object section, choose the view you want in Add Object View dropdown and click Add button
- You can add all views using Add All button

In EVENT TABLE section, add either default SNOWFLAKE.TELEMETRY.EVENTS table or the custom event table you have configured for your account

Send Data from Observe for Snowflake App - Via Console¶

Log into your Snowflake instance in Snowsight console
Navigate to Project\Worksheets
Use previously created o4s scripted install worksheet
Run:
- Uncomment each view you want to submit

--------------------------------------------------------------------------------
-- [ Streamlit UI ]
-- Step 9: Add various supported ACCOUNT_USAGE views for collection
--------------------------------------------------------------------------------
USE ROLE ACCOUNTADMIN;

USE APPLICATION OBSERVE_FOR_SNOWFLAKE;

-- History views
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.ACCESS_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.ALERT_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.AUTOMATIC_CLUSTERING_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.BLOCK_STORAGE_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.COMPLETE_TASK_GRAPHS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.COPY_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.DATA_CLASSIFICATION_LATEST');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.DATA_QUALITY_MONITORING_USAGE_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.DATA_TRANSFER_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.DATABASE_REPLICATION_USAGE_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.DATABASE_STORAGE_USAGE_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.DOCUMENT_AI_USAGE_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.DYNAMIC_TABLE_REFRESH_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.EVENT_USAGE_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.HYBRID_TABLE_USAGE_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.INTERNAL_DATA_TRANSFER_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.LOAD_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.LOCK_WAIT_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.LOGIN_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.MATERIALIZED_VIEW_REFRESH_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.METERING_DAILY_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.METERING_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.PIPE_USAGE_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.QUERY_ACCELERATION_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.QUERY_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.REPLICATION_GROUP_REFRESH_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.REPLICATION_GROUP_USAGE_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.SEARCH_OPTIMIZATION_BENEFITS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.SEARCH_OPTIMIZATION_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.SERVERLESS_TASK_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.SESSIONS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.SNOWPARK_CONTAINER_SERVICES_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.SNOWPIPE_STREAMING_CLIENT_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.SNOWPIPE_STREAMING_FILE_MIGRATION_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.STAGE_STORAGE_USAGE_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.STORAGE_USAGE');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.TABLE_DML_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.TABLE_PRUNING_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.TASK_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.WAREHOUSE_EVENTS_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.WAREHOUSE_LOAD_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY');
-- These only work when you have reader accounts
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('READER_ACCOUNT_USAGE.LOGIN_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('READER_ACCOUNT_USAGE.QUERY_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('READER_ACCOUNT_USAGE.STORAGE_USAGE');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('READER_ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY');
-- These work only in the Org admin account
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ORGANIZATION_USAGE.LISTING_AUTO_FULFILLMENT_USAGE_HISTORY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('ORGANIZATION_USAGE.RATE_SHEET_DAILY');
-- These work only if you you are in the marketplace account and have apps
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('DATA_SHARING_USAGE.LISTING_AUTO_FULFILLMENT_DATABASE_STORAGE_DAILY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('DATA_SHARING_USAGE.LISTING_AUTO_FULFILLMENT_REFRESH_DAILY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('DATA_SHARING_USAGE.LISTING_CONSUMPTION_DAILY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('DATA_SHARING_USAGE.LISTING_EVENTS_DAILY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('DATA_SHARING_USAGE.LISTING_TELEMETRY_DAILY');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('DATA_SHARING_USAGE.MARKETPLACE_DISBURSEMENT_REPORT');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('DATA_SHARING_USAGE.MARKETPLACE_LISTING_INVOICE_STATUS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_HISTORY('DATA_SHARING_USAGE.MONETIZED_USAGE_DAILY');

-- Object views
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.DATABASES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.FILE_FORMATS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.FUNCTIONS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.GRANTS_TO_ROLES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.GRANTS_TO_USERS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.HYBRID_TABLES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.MASKING_POLICIES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.NETWORK_POLICIES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.NETWORK_RULES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.PASSWORD_POLICIES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.PIPES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.PROCEDURES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.ROW_ACCESS_POLICIES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.ROLES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.SCHEMATA');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.SECRETS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.SEQUENCES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.SERVICES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.SESSION_POLICIES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.STAGES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.TABLES');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.TABLE_STORAGE_METRICS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.TAGS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.USERS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ACCOUNT_USAGE.VIEWS');
-- These work only in the Org admin account
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('ORGANIZATION_USAGE.ACCOUNTS');
-- These work only if you you are in the marketplace account and have apps
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_ACCOUNT_USAGE_OBJECT('DATA_SHARING_USAGE.APPLICATION_STATE');

--------------------------------------------------------------------------------
-- [ Streamlit UI ]
-- Step 10: Add EVENT_TABLE to collection and adjust schedule
-- variables to replace below: <event_table_db>, <event_table_schema>, <event_table_name>
--------------------------------------------------------------------------------
USE ROLE ACCOUNTADMIN;

USE APPLICATION OBSERVE_FOR_SNOWFLAKE;

-- Choose default or non-default event table based on your account configuration
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_EVENT_TABLE('SNOWFLAKE.TELEMETRY.EVENTS');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_EVENT_TABLE('<event_table_db>.<event_table_schema>.<event_table_name>');

--------------------------------------------------------------------------------
-- [ Streamlit UI ]
-- Step 11: Add any table or view
-- variables to replace below: <table_or_view_db>, <table_or_view_schema>, <table_or_view_name>
--------------------------------------------------------------------------------
USE ROLE ACCOUNTADMIN;

USE APPLICATION OBSERVE_FOR_SNOWFLAKE;

-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.ADD_REGULAR_TABLE('<table_or_view_db>.<table_or_view_schema>.<table_or_view_name>');

Adjust Schedule of Sending Data From Observe Snowflake Application¶

Snowflake for Observe native application uses tasks (running on XS warehouse or serverless) to send data. Those tasks consume compute credits. The default settings strive to strike a balance between data freshness and low runtime costs.

When tasks run on XS warehouse, the recommended settings suspend the XS warehouse after 1 minute of inactivity, which minimizes idle time.

When tasks run on serverless warehouse, you are only charged for the time that unit of compute is being used.

The data egressed will be billed in https://docs.snowflake.com/en/sql-reference/organization-usage/usage_in_currency_daily under SERVICE_TYPE=DATA_TRANSFER, with costs expected to be extremely minimal.

Either way, the more frequently these tasks run, the more credits would be consumed.

Adjust Schedule of Sending Data - Via UI¶

Snowflake for Observe native application is now sending your Snowflake management data to Observe. You can adjust the frequency of sending for each management view and event table.

Log into your Snowflake instance in Snowsight console
Change to the ACCOUNTADMIN role
Navigate to Data Products\Apps
Click Observe for Snowflake app
Modifying the schedule of any configured task by providing a new CRON value in SCHEDULE setting
Click Save Changes button

Observe For Snowflake - Adjust Send Schedule

To disable task, uncheck RUNNING checkbox and click Save Changes button
To disable all tasks, click Suspend All button
To start all disabled tasks, click Start All button

Adjust Schedule of Sending Data - Via Console¶

Log into your Snowflake instance in Snowsight console
Navigate to Project\Worksheets
Use previously created o4s scripted install worksheet
Run:

-- Adjust schedule for the ACCOUNT_USAGE views
-- Replace the schema and the view name accordingly
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.UPDATE_TASK_SCHEDULE('*/15 * * * * UTC', 'TASKS.O4S_TASK_COLLECT_ACCOUNT_USAGE_QUERY_HISTORY_0');

-- Adjust schedule for the SNOWFLAKE.TELEMETRY.EVENTS event table to collect less frequently if so desired
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.UPDATE_TASK_SCHEDULE('*/15 * * * * UTC', 'TASKS.O4S_TASK_SEND_EVENT_TABLE_' || SHA2('SNOWFLAKE.TELEMETRY.EVENTS') || '_0');
-- Adjust schedule for a custom event table
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.UPDATE_TASK_SCHEDULE('*/15 * * * * UTC', 'TASKS.O4S_TASK_SEND_EVENT_TABLE_' || SHA2('<event_table_db>.<event_table_schema>.<event_table_name>') || '_0');

-- Adjust schedule for a custom table or a view
-- Choose the right statement based on the object type
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.UPDATE_TASK_SCHEDULE('*/15 * * * * UTC', 'TASKS.O4S_TASK_SEND_TABLE_' || SHA2('<table_or_view_db>.<table_or_view_schema>.<table_or_view_name>') || '_0');
-- CALL OBSERVE_FOR_SNOWFLAKE.PUBLIC.UPDATE_TASK_SCHEDULE('*/15 * * * * UTC', 'TASKS.O4S_TASK_SEND_VIEW_' || SHA2('<table_or_view_db>.<table_or_view_schema>.<table_or_view_name>') || '_0');

Validate Snowflake Data is Received in Observe¶

To validate that Snowflake data is going to Observe, check the status of previously created ingest token in Snowflake app/integration.

Log into your Observe instance using an admin account
Click Data\Applications in left-side navigation
Find Snowflake application and open it
Click on Connections tab
The previously created token should show a green status light and have some observations

Observe For Snowflake - Token Receiving Data