Credit Manager

Credit Manager helps you manage the consumption of your Observe Compute Credits (OCC) to stay within your desired budget.

Your organization consumes OCCs when the Observe platform performs computations on ingested data, such as performing a query or accelerating a Dataset. Use the settings on the Credit Manager page to set a quota for the maximum number of OCCs your organization consumes for queries and data acceleration.

For more information on data acceleration and queries, see About Queries and On-demand Acceleration.

Using Credit Manager

On the Credit Manager page, specify the Query Credit Limit and the Acceleration Credit Limit separately. Add the two numbers together to determine your overall OCC limit.

For example, if you set the Query Credit Limit to 100 credits per day and the Acceleration Credit Limit to 100 credits per day, your overall OCC budget consists of 200 credits per day or 6000 credits per month.

Query Credit Limit and Acceleration Credit Limit have two distinct behaviors.

Query Credits

To configure the Query Credit Limit, select Custom to define the number of Query Credits consumed per day as your budget, or select Unlimited to disable Query Credit Manager.

Figure 1 - Query Credits

When the Query Credit Limit becomes 80% exhausted in a 24-hour rolling window, your users begin to see warnings alerting them that your account has begun approaching your desired budgetary goal. The notifications function as a cue for users to consider their use of Observe and adjust their usage to reduce credit consumption.

Figure 2 - 80% of available credits consumed

Figure 3 - Recent usage warning message

Figure 4 - Temporarily override the limit once or override for an hour

The warnings explicitly alert users about exceeding your intended budget for the day. Because Observe is a critical monitoring and investigation tool, Observe provides several options to your users to continue using Observe without interruption:

• Allow Once - temporarily override the limit once for the current user query.

• Allow for the next hour - temporarily override the limit for user queries over the next hour.

Note

Either selection affects only the current user and not the organization. 

As an administrator, you can set a higher Query Credit Limit, or users can wait a short time until more Query Credits accrue at a rate defined by the daily Query Credit Limit.

Acceleration Credits

To configure the Acceleration Credit Limit, select Custom to add the number of Acceleration Credits per day for your budget, or select Unlimited to disable Acceleration Credit Manager.

Figure 5 - Acceleration Credits

Unlike Queries, Observe performs Datset Acceleration as a background activity, so your users do not receive explicit warnings or alerts when exhausting the Acceleration Credit usage. Instead, when the Observe account exhausts the Acceleration Credit Limit in seven (7) days, Observe automatically raises the freshness goals for Datasets in Observe to reduce cost and stay within credit limits.

Observe automatically retunes the freshness goal to the original value when the Acceleration Credit consumption returns below the set limit. For example, suppose you set the freshness goal for the Container Logs Dataset to 60 seconds. If the Acceleration Credit usage exceeds the limit for an extended period of time, Observe may automatically adjust this freshness goal to 120 seconds up to a maximum goal of one hour.

With a more relaxed freshness goal, Observe performs Data Acceleration on each dataset less often, reducing the Acceleration Credit consumption rate. However, users may experience delays until data becomes available in the datasets. Optimizing Acceleration Credit consumption trades off cost and freshness goals, for example, latency between data ingest and the ability to query the data.

Using Credit Manager Effectively

The following tips guide you to effectively using and configuring Credit Manager:

• Use Credit Manager as a “safety net” to prevent excessive credit consumption and unexpected costs. When you use Credit Manager for the first time, start by setting the limit to twice the number of credits you plan to consume. If you forecast that you may consume 3,000 Query Credits in a month, set the Query Credit Limit to 3,000/30 x 2 = 200, which minimizes disruption to your users.

• The Credit Manager page contains a few statistics to help guide the configuration of your credit limits, such as Daily avg credits in past 30d and Credits used past 24hr. Use the Usage Dashboard to view credit usage in more detail and over longer ranges of time to assist with configuring your limits.

• If you find that your users reach the Query Credit Limit daily, use the following suggestions:

  • Use shorter query time ranges such as Past 24 hours rather than long time ranges such as Past 30 days. When Observe queries with wider time ranges on large Datasets, a vast amount of data must be scanned. To make queries more efficient, use a shorter time range necessary to satisfy your query.

  • If your users query a large Dataset, for example, Container Logs, you should create smaller Datasets derived from the large Dataset to target a common use case such as API Server Errors. Although creating a Dataset may consume Acceleration Credits, users share the accelerated Dataset and query it. This can result in a lower net consumption of total Credits. Also, queries on smaller datasets can be faster and provide a better user experience.

Ideally, you should adjust the freshness goals of your Datasets rather than allow Credit Manager to retune them for you. For example, suppose you have a Dataset that you infrequently query, such as VPC Flow Logs. In that case, you can set the freshness goal to 30 minutes to reserve more of your Acceleration Credit budget for more critical Datasets with higher freshness requirements. Reach out to your Observe Data Engineer for assistance with retuning your Dataset freshness goals.

For more information on either Query Credits or Acceleration Credits, see the Usage Dashboard.