Observe Datasets and Time#
Observe collects all data, system and application logs, metrics, and tracing spans, into observations, which then transform into datasets. Datasets are structured with times or time intervals, as well as relations linking to or from other datasets. These relations between different parts of the system provide Observe with superpowers for discovering the hidden meaning in your data.
A dataset lives within a named project, and also has a name. Project names must be unique for your customer, and dataset names must be unique within their project. When you log into Observe, the Explore page lets you browse the different datasets that exist for your customer ID.
A Dataset has a schema, which is a set of named columns and definitions of data types stored in those columns (such as event or resource).
If an incident occurs “at a time” and has a well-defined timestamp, then the dataset is an Event Dataset. Events have a single point in time, and typically link or relate to one or more other tables in the system. For example, “user X logged into system Y at time Z” is an event, which also links to the “user” dataset and the “system” Dataset.
Creating Log Datasets from Event Datasets#
In order for a Log Dataset to appear in the list of available Log Datasets, you must add OPAL code to the desired Event Dataset.
interface code to the dataset allows you to display it in the Log Explorer. The Observe interface can then interpret the dataset as logs. For example, adding fields allows you to search and expand the fields.
To add a specific column of data to the Log Dataset, use the following OPAL code:
interface "log", "log":<nameOfLogField>
The column must have the
string designation. If not, convert the column to an object using OPAL before adding the
log interface from the Dataset, use the
For more information about
interface, see the OPAL verb,
Finally, objects with permanence over time, and whose state changes over time, are stored in resource datasets. Any field value for a resource has a valid time interval — a start time, and an end time. For a resource, you can ask questions like “what was the name at time T?” Additionally, a primary key identifies a resource.