Accidental ingestion of sensitive data

What happens if sensitive data is accidentally ingested by Observe?

Customers are responsible for preventing the transmission of sensitive data such as personally identifiable information (PII) or financial data to Observe. If accidental ingestion occurs, Observe offers support to mitigate and remediate the issue. This policy outlines customer actions and the process for requesting data deletion.

Identifying Sensitive Data

Customers are responsible for identifying when sensitive data has been ingested and which datasets contain that data. Observe does not scan or identify datasets containing PII or sensitive values on behalf of customers.

Once the affected datasets are identified and the appropriate OPAL filters are provided, Observe can apply dataset query filters to hide sensitive data from query results.

Customer actions to mitigate accidental ingestion

To prevent further transmission and restrict access to sensitive data, customers should take the following steps before requesting deletion:

  • Stop sending sensitive data: Immediately halt the transmission of sensitive data to Observe. Customers are responsible for implementing preventive measures.

  • Check data retention settings: Verify if the sensitive data is within your configured retention period. Data will automatically be deleted once the retention period expires. Refer to Data Retention in Observe Documentation for details.

  • Implement RBAC policies: Restrict access to datastreams or datasets by applying Role-Based Access Control (RBAC) policies, limiting queries to authorized users only.

  • Apply Dataset Query Filters (Immediate Mitigation): The only way to immediately mitigate accidental sensitive data exposure is through dataset query filters or RBAC. Query filters hide sensitive records from query results by matching an OPAL filter.

    If you need Observe support to apply dataset query filters to help mitigate exposure, please provide the following for each affected dataset including any downstream datasets in the relevant support ticket:

    • Confirmation that ingestion has stopped
    • Worksheet with OPAL filters identifying the sensitive data (no make_column or similar verbs).

    These filters take effect immediately and prevent sensitive values from appearing in query results.

Dataset Query Filters

Dataset query filters protect sensitive data from appearing in search results during querying and serve as a soft-delete, but do not affect the source data stored in Observe's Snowflake tables.

Request data deletion

If sensitive data cannot be sufficiently mitigated using RBAC and dataset query filters, customers may request deletion of the data via the Observe Support Portal. Data deletion is a resource-intensive process and is often not required once proper mitigation is in place.

Provide the following information to submit a deletion request:

  • Confirmation that sensitive data ingestion has stopped.
  • Why RBAC or query filters are insufficient to mitigate the issue
  • OPAL queries for each dataset or datastream identifying sensitive data (≤ 7 days per query)
  • The total number of records to be deleted per dataset for verification.
  • Confirmation that the requester is an admin of the Observe deployment.

Data deletion process and timeline

The data deletion timeline depends on your level of support with Observe:

  • Standard support: Deletion requests are targeted for completion within 30 days of submission.
  • Premier support: Completion is targeted within 14 days of submission.

Deletion requests are processed as described below:

  • Deletions under 10,000 rows can be done surgically; 10,000 to 1 million require customer approval and cost assessment.
  • Deletions over 1 million rows are not supported; customers must use timestamp-based time-range deletions instead.
  • Deletions are based on coarse-grained timestamp filters, not precise query matches, potentially deleting more data than requested.
  • Support targets deletion completion within 30 days (standard) or 14 days (premier).

Important notes

  • Customers should prioritize access controls (filters or RBAC) to prevent unauthorized access, as deletion may not be immediate.
  • Observe will confirm completion of the deletion process upon request.

For further assistance, contact us using the Observe Support Portal.