Observe Datasets and Time

Observe collects all data (system and application logs, metrics, and tracing spans) into observations, which are transformed into datasets. Datasets are structured with times or time intervals, as well as links (relations) to/from other datasets. Having these links (relations) between different parts of the system is what gives Observe superpowers when discovering the meaning hidden in the data.

Datasets

A dataset lives within a named project, and in turn has a name. Project names must be unique within your customer, and dataset names must be unique within their project. When you log into Observe, you are shown the “Explore” page which lets you browse the different datasets that exist for your customer id.

A dataset has a schema (a set of named columns and the type of data stored in those columns) and a type: “table,” “event,” or “resource.” The type is determined mainly by how much time-related information is available about the dataset.

Table Datasets

If information is not related to time and changes over time are not tracked, a dataset is a “table.” This is like a normal look-up table in most systems, but is the kind that is used the least in Observe because it doesn’t allow tracking changes over time.

Event Datasets

If something happens “at a time” and has a well-defined timestamp, then the dataset is an “event dataset.” Events have a single point in time, and typically link (relate) to one or more other tables in the system. For example, “user X logged into system Y at time Z” is an event, which also links to the “user” dataset and the “system” dataset.

Foreign Keys

Resource Datasets

Finally, objects that have permanence over time, and whose state changes over time, are stored in “resource datasets.” Any field value for a resource has a validity time interval — a start time, and an end time. For a resource, you can ask questions like “what was the name at time T?” Additionally, a resource is identified by a primary key.

Resource Times

Resource Primary Keys