About Queries and On-demand Acceleration

What is a query?

Queries send requests for data stored in Observe and allow you to view that data. Essentially any time you view a dataset in Observe, you perform a query. Under the hood, a query consists of multiple stages, with each stage using a separate OPAL script. An OPAL script consists of a sequence of statements where the output of one statement is the input for the next. This could be a single line with one statement or many lines of complex shaping and filtering.

A script contains the following types of elements:

• Query - your query contains a single time window shared across all stages, and each stage consists of a set of input elements, such as a block of OPAL script.

• Inputs - can be datasets or other stages in the same query. Each stage contains a single output with an OPAL script consisting of OPAL verbs and functions.

• Verbs - define the processing to perform with the data in the stage.

A complete OPAL script consists of inputs, verbs, functions, and outputs defining the desired result. When you create a query in a worksheet using the console and click Run, Observe executes all stages in a query on the datasets. In a Dashboard, Observe runs the query in the background.

Understanding On-demand Acceleration

Datasets in Observe have an acceleration window. An acceleration window contains the time range for the queries on each dataset to be fast and efficient.

Observe automatically accelerates the dataset for the prior seven(7) days when you edit or create a dataset. Therefore, Observe performs queries in recent time ranges quickly and efficiently.

Observe also automatically extends this acceleration window forward as new data arrives, so queries on any time range after creating a dataset also perform quickly and efficiently.

Observe does not automatically accelerate access to historical data from before dataset creation. As a result, queries on newly created or edited datasets, such as those created a few days ago, using a time range from far in the past, such as this day last year, cannot be satisfied with accelerated data.

When you select a time range with the Select Time Range option, some options may have a yellow icon next to them. These are time ranges extending past the datasets’ acceleration window in your query.

If you select a time range with next to it, two things can occur in Observe:

1. Because you selected a time range beyond the acceleration window of the datasets you want to query, older data from before the beginning of the acceleration window may be missing from the result.

   • For example, if a queried Dataset was only accelerated 14 days ago, and you select Past 30 days in the Select Time Range, your query results only reflects the past 14 days, even though you requested the past 30 days.

2. You see a message indicating that your results may only reflect a partial range of the requested time range.

If you can use partial results, you can dismiss the message.

You can accelerate the dataset by selecting Accelerate full range if you need results for the whole time range.

If you perform this action, Observe initiates a background process to accelerate the datasets in your query for the entire time range you selected from the Select Time Range menu.

You can monitor the acceleration progress on your dataset from the Query Overview window. It can take anywhere from a few seconds to a few minutes for the dataset to update.

Note

Acceleration of historical time ranges for datasets consumes Observe Compute Credits (OCC). You can monitor your credit usage by reviewing the Usage Dashboard for credits attributed to backfill activity.

After completing the background process, your next query returns results for the full time range that you requested. Because it used an accelerated dataset, the query becomes much faster and more cost-efficient compared to querying your raw data. Also, because you accelerated the dataset, all other users in your account enjoy the same performance and efficiency benefit when querying this dataset.

Accelerating Datasets

When you open a worksheet, dashboard, or dataset, Observe automatically updates the query, and you can see the Query in progress icon . Click the Stop icon to end the query process.

After completing the background process, your following query returns results for the requested time range. Because it uses an accelerated dataset, the query becomes much faster and more cost-efficient than querying your raw data. Also, because you accelerated the dataset, all other users in your account enjoy the same performance and efficiency benefit when querying this dataset.

If the query returns results successfully, the icon changes to the Query Overview icon . You can refresh the query using the Refresh icon.

Figure 1 - Query results for a dataset

You can change the time range for the query by using the Select Time Range icon. After you select it, you may see some time ranges with a yellow icon next to them.

Figure 2 - Query range for a dataset

These time ranges may include a window of time where the underlying datasets for your query are not accelerated. If you query these ranges, your query may return partial results compared to a query on raw data ingested into Observe.

You may elect to accelerate the unaccelerated time range to obtain complete results.

Figure 3 - Dataset duration beyond the accelerated range

Clicking on the yellow icon allows you to choose to accelerate the range of data.

Figure 4 - Unaccelerated results for a dataset

To accelerate the full range of data, click Accelerate full range.

Figure 5 - Unable to accelerate data for the selected time range

Acceleration may be unavailable for a selected time range because the dataset on demand materialization has reached the upper limit. Contact Observe support if you encounter this error and need assistance to accelerate a more extended time range.

Data Acceleration and Stale Data

You may encounter instances where Data Acceleration has overridden the Freshness Goal of a Dataset or Datasets. The following information displays in the Query Overview:

Figure 6 - Excessively Stale Data

Datasets may hibernate due to a long period of inactivity which automatically occurs to help reduce credit usage. You can wait a few minutes and refresh the page to receive fresh data.

Figure 7 - Data Hibernation

Data can be stale due to a high freshness goal. You must manually decrease the freshness goal in order to refresh the data.

Figure 8 - High Freshness Goal

Data cannot be accelerated to meet the Freshness Goal because the Dataset contains too much data or the OPAL for the Dataset requires optimization.

Figure 9 - Dataset Contains a High Volume of Data