Observe Datasets and Time

Observe collects all data, system and application logs, metrics, and tracing spans, into observations, which then transform into datasets. Datasets are structured with times or time intervals, as well as links, relations, to or from other datasets. These relations, between different parts of the system provides Observe with superpowers when discovering the hidden meaning in the data.

Datasets

A dataset lives within a named project, and also has a name. Project names must be unique for your customer, and dataset names must be unique within their project. When you log into Observe, the Explore page lets you browse the different datasets that exist for your customer ID.

A dataset has a schema, a set of named columns and the type of data stored in those columns, and a type, such as event or resource.

Event Datasets

If an incident occurs “at a time” and has a well-defined timestamp, then the dataset is an “event dataset.” Events have a single point in time, and typically link or relate to one or more other tables in the system. For example, “user X logged into system Y at time Z” is an event, which also links to the “user” dataset and the “system” dataset.

Foreign Keys

Resource Datasets

Finally, objects with permanence over time, and whose state changes over time, are stored in resource datasets. Any field value for a resource has a validity time interval — a start time, and an end time. For a resource, you can ask questions like “what was the name at time T?” Additionally, a primary key identifies a resource . Resource Times

Resource Primary Keys