documentationUsing Acceleration Manager with Datasets¶
Use the Acceleration Manager to diagnose and fine-tune Acceleration Credit consumption for datasets and monitors in your Observe account.
Observe periodically precomputes the latest contents of datasets in order to make queries faster and more efficient than queries on the raw data ingested into Observe. Observe calls this process data acceleration.
The data acceleration process is tuned with a setting called a freshness goal. The freshness goal of a dataset or monitor states the maximum time delay between the periodic updates that the platform makes to that dataset or monitor. A tighter freshness goal asks the platform to perform these updates more frequently in order to keep data fresher and more up-to-date, and a looser freshness goal allows the platform to perform these updates on a slower schedule. In general, a tighter freshness goal generally results in a higher rate of Acceleration Credit consumption, and a looser freshness goal results in a lower rate of Acceleration Credit consumption. Thus, tuning the freshness goal allows you to make a trade-off between the freshness of data in these datasets and the cost of accelerating queries on them.
For example, regular reviews of your Usage Dashboard reveal that the Dataset, kubernetes/Pod Metrics, consumes 4.7% of your Acceleration Credits with a one (1) minute freshness goal. You determine that you can accept a freshness goal of five (5) minutes and reduce your Acceleration Credit consumption.
To lower the freshness goal, navigate to Acceleration Manager and search for the kubernetes/Pod Metrics Dataset. Select the Settings icon, and then Edit Freshness Goal. Increase the freshness goal to five (5) minutes. Your dataset now accelerates once every five minutes and decreases the number of consumed Acceleration Credits.
To understand more about data acceleration and the impact on your Datasets, see Advanced Observe Concepts and About Queries and On-demand Acceleration. For details on freshness goals for datasets, see Transform Queries and Freshness Goals.
The Acceleration Manager page displays the following information about your Datasets and Monitors:
Ongoing Acceleration Jobs¶
Name - displays the name of the Dataset with an icon to indicate the Dataset type as Event Stream or Resource.
Last Update - displays the last time the Dataset was updated.
Freshness Goal - displays how often you want Observe to update the Dataset. Observe schedules Dataset acceleration jobs to satisfy the freshness goals of all Datasets. Longer freshness goals allow Observe to include more data in an acceleration job, providing more efficiency and less credit usage overall.
Effective Freshness - displays the current freshness of the Dataset.
Status - displays the current status of the Dataset such as Accelerated, Cannot Be Accelerated, or Backfill in progress.
Acceleration Range - displays the length of time to apply acceleration to the Dataset. This is the range of data currently accelerated.
Last Modified By - displays the name of the member who last modified the Dataset definition.
Last Modified - displays the date of the last modification.
Settings - click to edit the freshness goal, open the Dataset, or edit the Dataset definition.
Figure 1 - Acceleration Manager
You may see one of two icons displayed next to the Effective Freshness value:
- The freshness goal of this Dataset is suppressed due to your Acceleration Credits limit in Credit Manager. Data may be less fresh than you desire. See Credit Manager for more information on Acceleration Credits.
- There is a downstream Dataset or Monitor with a tighter freshness goal than this object, preventing the set goal from matching the effective goal.
Use these steps to change the freshness goal for a selected Dataset:
Click the Settings
icon at the end of a Dataset row to open the menu.
Figure 2 - Settings Options
2. Select Edit Freshness Goal, and click Apply. The message “Checking downstream Datasets…” displays on the panel.
3. If the set freshness goal matches the effective freshness goal, Observe displays the following message:
Figure 3 - Freshness goal matches effective freshness goal
4. If a downstream Dataset has a tighter goal than the set goal of the selected Dataset, Observe returns the following message:
Figure 4 - Effective goal tighter than desired freshness goal
In this case, you must adjust the freshness goals of the downstream Datasets. Observe displays the affected Dataset in the message, for example, Server/Host. The minimum dataset freshness goals and any downstream freshness goals dictate how often Observe executes acceleration of the data.
5. If you use Acceleration Credits limits in the Credit Manager feature to budget your accelerations, Observe returns the following message:
Figure 5 - Freshness suppressed by Credit Manager
On-Demand Acceleration Jobs¶
The On-Demand tab displays the status of Datasets undergoing acceleration or were accelerated.
Figure 6 - On-Demand Acceleration Jobs
You can select to display them by the following types:
All states - displays Acceleration Jobs in any state.
Running - displays Acceleration Jobs currently undergoing the acceleration process.
Completed - displays Acceleration Jobs that finished the acceleration process.
Cancelled - displays Acceleration Jobs canceled during the acceleration process.
On-Demand Acceleration Jobs also displays the following information:
Source - displays the Dataset group. Clicking on the arrow next to the group to display the underlying Datasets.
Progress - displays the status of the Acceleration Jobs.
Job ID - displays the Job ID of the Acceleration Jobs.
User - displays the user who requested the Acceleration Jobs for the Dataset.
Datasets Involved - displays the number of Datasets accelerated.
Time Triggered - displays the time when the Acceleration Jobs was requested.
Actions - displays Cancel during the Acceleration Jobs to allow you to stop the acceleration of the Dataset.