Track user activity with audit trails

This page describes Observe audit logs for CRUD operations.

📘
Note
Audit logs for CRUD operations is GA and documented here. Query audit logs is still in private preview. Please open a support request to have query audit logs enabled in your tenant.

As an administrator, you can use enable the audit trail to understand who is using Observe within your organization and what actions they are performing. See What is audited? for a summary of the objects and actions audited by Observe.

Using audit logs, you can understand changes in your Observe tenant, such as:

A dashboard or monitor is now broken and needs to be fixed, due to a change in the input Dataset. Who modified the input Dataset, and when?
An alert never went out during my on-call shift, because a monitor was disabled. Who disabled the monitor, and when?

For security teams, audit trail events help with compliance checks and maintaining audit trails of who did what, and when, for your Observe resources. For example, maintaining an audit trail:

When someone changes the access controls on a dataset, providing too much access.
For user logins, account, or role changes in your organization.

This audit trail can be exported to cloud object storage via the Observe Data Export feature, where your compliance requirements for retention are beyond your Observe retention, or for analysis by your SIEM.

What is audited?

The following table shows the Observe objects that are audited, and specifically the actions audited for each object:

Object Type	Create	Read	Update	Delete	Disable	Enable
Dataset	✓	✓	✓	✓
Monitor	✓	✓	✓	✓	✓	✓
Worksheet	✓	✓	✓	✓
Dashboard	✓	✓	✓	✓
Datastream	✓	✓	✓	✓
Reference table	✓	✓	✓	✓
Datastream token	✓	✓	✓	✓	✓	✓
API token	✓		✓	✓	✓	✓
User	✓	✓	✓	✓	✓	✓
Report	✓	✓	✓	✓	✓	✓
RBAC statement	✓		✓	✓
RBAC group	✓		✓	✓
RBAC group member	✓			✓
Drop filter	✓	✓	✓	✓	✓	✓
Document	✓	✓	✓	✓
Data export job	✓	✓		✓

In addition, login and logout logs are also audited, including the login type, such as SAML or local, and whether the login attempt was successful or not. These logs are not associated with any Observe objects.

Enable audit logging

You can enable audit logging in your Observe instance settings. See Instance settings.

Configuration

Observe Usage reporting app

Observe has bundled the dataset usage/Audit Events with the Observe Usage reporting app focused on capturing audit trail data. You can also control access to the audit trail by adding access controls to the System datastream/root dataset as well as the usage/Audit Events dataset. You must be on version 0.22.0 or higher of the Observe Usage Reporting app to take advantage of this content.

Archive your audit trail

Archiving is an optional feature for the audit trail. You can use archiving to write to Amazon S3 and have your SIEM system read events from it, or store them for compliance reasons. Note that Data Export is currently in public preview.

Explore audit events

The Observe Usage reporting app now bundles audit trail content.

Audit Events dashboard: This dashboard provides examples for monitoring authentication events, overall CRUD operations per user, as well as content popularity.

usage/Audit Events Dataset: This Dataset enriches the raw audit data in the System Datastream with user information.

If you want to just view the raw events, you can open the System Datastream and run the following OPAL query, which shows all the audit events by their operation and object type. The example_events column will contain newline separated samples of your audit events.

make_col schema:string(EXTRA.schema)
filter schema = "audit_log"
statsby example_events:string_agg_distinct(string(FIELDS),"
"), group_by(FIELDS.object_kind,FIELDS.operation)

Create a monitor

Because the underlying raw data is exposed it is very simple to create monitors for specific audit events. For example, set a monitor that triggers when a specific user logs in, or set a monitor for anytime a dashboard is deleted. In the below example you can filter to failed login events by user, and alert when the number of failed logins per user surpasses a count over a specific timerange.

Create a scheduled report

You can use Observe dashboard reports to create scheduled report of the Audit Events dashboard. See Generate dashboard reports.

Download audit events as a CSV file

Observe Audit Trail allows you to download audit events as a CSV or JSON file locally. These events can then be analyzed locally, uploaded to a different tool for further analytics, or shared with appropriate team members as part of a security and compliance exercise.

To export audit events as CSV:

Open the usage/Audit_Trail Dataset in a Worksheet.
Filter the worksheet to just the audit events you want. For example, to view just the LLM audit logs:
```
filter EXTRA.schema = "audit_log" and FIELDS.operation = "llm_invocation"
```
Click Export, then select CSV or JSON as the format.
Select the number of events to export.

JSON examples

This section contains some examples of LLM audit logs.

o11y.ai Help

{
  "ai_feature": "o11y_help",
  "ai_model": "gpt-4.1-2025-04-14",
  "ai_provider": "openai",
  "input_prompt": "what is the opal verb for getting the topk aggregate",
  "operation": "llm_invocation",
  "time": "2026-02-06T16:24:56.86286968Z",
  "user_id": 1952
}

Explain a log message

{
  "ai_feature": "explain_log_message",
  "ai_provider": "openai",
  "input_prompt": "Can you explain this log message? E20260202 07:43:56.026131    189 scanner/src/tags.cpp:742] msg=\\"Metric name contains colon, ignoring\\" metric_name=workload:istio_tcp_connections_closed_total",
  "operation": "llm_invocation",
  "time": "2026-02-02T07:47:03.372435965Z",
  "user_id": 1792
}

OPAL co-pilot

{
  "ai_feature": "opal_copilot",
  "ai_model": "Qwen2.5-7B",
  "ai_provider": "self_hosted",
  "input_prompt": "# Parameters\\n\\n# Linkable Datasets\\n\\n# Input Datasets\\nName: \\"Tracing/Span\\", Label:  \\"Tracing/Span\\", Role: Data\\nKind: Interval\\nIndexes: \\nPrimary Keys: trace_id, span_id\\nForeign Keys:\\nColumns:\\n    start_time: timestamp\\n    end_time: timestamp\\n    duration: duration\\n    response_status: string\\n    service_name: string\\n    span_name: string\\n    status_code: int64\\n    error: bool\\n    status_message: string\\n    span_type: string\\n    service_namespace: string\\n    service_instance_id: string\\n    environment: string\\n    service_version: string\\n    attributes: object\\n    resource_attributes: object\\n    instrumentation_library: object\\n    parent_span_id: string\\n    span_id: string\\n    trace_id: string\\n    customer_id: int64\\n    user_id: int64\\n    k8s_cluster_uid: string\\n    k8s_namespace_name: string\\n    k8s_pod_name: string\\n    fields: object\\n    meta: object\\n\\n\\nName: \\"Tracing/Span_s3ji7\\", Label:  \\"Tracing/Span\\", Role: Data\\nKind: Interval\\nIndexes: \\nPrimary Keys: trace_id, span_id\\nForeign Keys:\\nColumns:\\n    start_time: timestamp\\n    end_time: timestamp\\n    duration: duration\\n    response_status: string\\n    service_name: string\\n    span_name: string\\n    status_code: int64\\n    error: bool\\n    status_message: string\\n    span_type: string\\n    service_namespace: string\\n    service_instance_id: string\\n    environment: string\\n    service_version: string\\n    attributes: object\\n    resource_attributes: object\\n    instrumentation_library: object\\n    parent_span_id: string\\n    span_id: string\\n    trace_id: string\\n    customer_id: int64\\n    user_id: int64\\n    k8s_cluster_uid: string\\n    k8s_namespace_name: string\\n    k8s_pod_name: string\\n    fields: object\\n    meta: object\\n\\n\\n# Additional Context\\nObject Kind: Worksheet\\n\\n# OPAL Query\\n// get the name of the parent_service on the same row as the spans service_name\\n",
  "operation": "llm_invocation",
  "source_ip": "73.61.69.6",
  "time": "2026-02-06T16:15:43.868735451Z",
  "user_id": 1921
}

Extract regex

{
  "ai_feature": "regex_extract",
  "ai_model": "gpt-4.1-2025-04-14",
  "ai_provider": "openai",
  "input_prompt": "I0206 18:00:26.428699                  745 tempo/bind.go:583] msg=\"calculate effective window\" query_id=b7311624-0385-11f1-85d4-c5073a492ced op_index=0 policy=BestEffortFinalSlice best_effort_materialized_windows=",
  "operation": "llm_invocation",
  "time": "2026-02-06T18:01:53.635177705Z",
  "user_id": 1792
}

Trace AI analysis

{
  "ai_feature": "trace_analysis",
  "ai_model": "claude-sonnet-4-5-20250929",
  "ai_provider": "anthropic",
  "input_prompt": "This trace contains 8 spans.\\n[\\n  {\\n    \\"start_time\\": 1770396534059.2705,\\n    \\"end_time\\": 1770396534059.3293,\\n    \\"span_name\\": \\"missing\\",\\n    \\"service_name\\": \\"\\",\\n    \\"duration\\": 0.058837890625,\\n    \\"span_id\\": \\"a55f2ac271feb44e\\",\\n    \\"parent_span_id\\": \\"\\"\\n  },\\n  {\\n    \\"start_time\\": 1770396534059.2705,\\n    \\"end_time\\": 1770396534059.3293,\\n    \\"span_name\\": \\"observe/sched/taskmgr.(*TaskManager).handleTaskRequest\\",\\n    \\"service_name\\": \\"scheduler\\",\\n    \\"duration\\": 0.058837890625,\\n    \\"span_id\\": \\"3b5f2c6c8942505d\\",\\n    \\"parent_span_id\\": \\"a55f2ac271feb44e\\",\\n    \\"response_status\\": \\"Ok\\",\\n    \\"span_type\\": \\"Service entry point\\"\\n  },\\n  {\\n    \\"start_time\\": 1770396534059.295,\\n    \\"end_time\\": 1770396534059.3223,\\n    \\"span_name\\": \\"observe/sched/taskmgr.(*TaskManager).handleTask\\",\\n    \\"service_name\\": \\"scheduler\\",\\n    \\"duration\\": 0.02734375,\\n    \\"span_id\\": \\"55be5c5ae258d825\\",\\n    \\"parent_span_id\\": \\"a55f2ac271feb44e\\",\\n    \\"response_status\\": \\"Ok\\",\\n    \\"span_type\\": \\"Service entry point\\"\\n  },\\n  {\\n    \\"start_time\\": 1770396534059.309,\\n    \\"end_time\\": 1770396534059.3193,\\n    \\"span_name\\": \\"observe/sched/taskmgr.(*TaskManager).addTaskToQueue\\",\\n    \\"service_name\\": \\"scheduler\\",\\n    \\"duration\\": 0.01025390625,\\n    \\"span_id\\": \\"0dfad42dfe1ed41d\\",\\n    \\"parent_span_id\\": \\"55be5c5ae258d825\\",\\n    \\"response_status\\": \\"Ok\\",\\n    \\"span_type\\": \\"Internal operation\\"\\n  },\\n  {\\n    \\"start_time\\": 1770396534059.3157,\\n    \\"end_time\\": 1770396534072.8037,\\n    \\"span_name\\": \\"TaskQueuing\\",\\n    \\"service_name\\": \\"scheduler\\",\\n    \\"duration\\": 13.488037109375,\\n    \\"span_id\\": \\"347fbb29f5653d01\\",\\n    \\"parent_span_id\\": \\"55be5c5ae258d825\\",\\n    \\"response_status\\": \\"Ok\\",\\n    \\"span_type\\": \\"Internal operation\\"\\n  },\\n  {\\n    \\"start_time\\": 1770396534072.8137,\\n    \\"end_time\\": 1770396534072.8174,\\n    \\"span_name\\": \\"observe/sched/taskmgr.(*TaskManager).scheduleOneAndLogSuccess\\",\\n    \\"service_name\\": \\"scheduler\\",\\n    \\"duration\\": 0.003662109375,\\n    \\"span_id\\": \\"2baf7797eb7de402\\",\\n    \\"parent_span_id\\": \\"55be5c5ae258d825\\",\\n    \\"response_status\\": \\"Ok\\",\\n    \\"span_type\\": \\"Internal operation\\"\\n  },\\n  {\\n    \\"start_time\\": 1770396534072.9053,\\n    \\"end_time\\": 1770396534072.9128,\\n    \\"span_name\\": \\"missing\\",\\n    \\"service_name\\": \\"\\",\\n    \\"duration\\": 0.007568359375,\\n    \\"span_id\\": \\"54f50980eab35c12\\",\\n    \\"parent_span_id\\": \\"\\"\\n  },\\n  {\\n    \\"start_time\\": 1770396534072.9053,\\n    \\"end_time\\": 1770396534072.9128,\\n    \\"span_name\\": \\"observe/sched/warehouse.(*warehouse).openConn\\",\\n    \\"service_name\\": \\"scheduler\\",\\n    \\"duration\\": 0.007568359375,\\n    \\"span_id\\": \\"6dc4d40c9eb18872\\",\\n    \\"parent_span_id\\": \\"54f50980eab35c12\\",\\n    \\"response_status\\": \\"Ok\\",\\n    \\"span_type\\": \\"Internal operation\\"\\n  }\\n]",
  "operation": "llm_invocation",
  "user_id": 1792
}

Updated 6 days ago