Using Credit Manager to Manage Compute Usage

The Credit Manager helps you manage the consumption of your Observe Compute Credits (OCC) to stay within your desired budget.

Your organization consumes OCCs when the Observe platform performs computations on ingested data, such as performing a query or accelerating a Dataset. Use the settings on the Credit Manager page to separately set a quota for the maximum number of OCCs your organization may consume for queries and the acceleration of Datasets. Add the two numbers together to determine your overall OCC limit.

For example, if you set the Query Credit Limit to 100 credits per day and the Acceleration Credit Limit to 100 credits per day, your overall OCC budget consists of 200 credits per day.

In addition to these global limits, you can set user credit limits. This helps you to prevent a single user or group of users from accidentally consuming a larger amount of credits than expected.

  • Daily per user throttle limit

  • Daily per user hard limit

  • Individual daily throttle limit

  • Individual daily hard limit

Note

The per-user credit limit is currently in private preview, please work with your Observe team to enable this feature flag.

When a throttle limit is exceeded, Observe slows usage for the user and displays a warning at the lower left corner.

For more information on the acceleration of Datasets and the execution of user queries, see About Queries and On-demand Acceleration.

To access the Credit Manager UI, you can navigate to the URL https://${observe_tenant}.observeinc.com/settings/credit-manager, replacing ${observe_tenant} with your observe tenant ID.

Limiting Query Credits

When a Query Credit Limit is set, the Query Credit Manager enforces that the daily credit usage for user queries lies below that limit. The query credit limit is a hard limit. When the Query Credit Limit is exceeded, queries are blocked and users are notified of this. Because Observe is a critical monitoring and investigation tool, Observe provides several options to your users to continue using Observe without interruption:

  • Allow Once - temporarily override the limit once for the current user query.

  • Allow for the next hour - temporarily override the limit for user queries over the next hour.

Note

Either selection affects only the current user and not the organization.

As an administrator, you can set a higher Query Credit Limit, or users can wait a short time until more Query Credits accrue at a rate defined by the daily Query Credit Limit.

A Note on the Implementation

To smoothly limit the amount of Query credits used, Observe employs the token bucket algorithm. This offers the advantage of building up a burst budget at times of low utilization. Assume for example that there was no query activity on the weekend. On Monday the Credit Token Bucket, storing the credits available to be consumed, will be filled to 100%. The credits contained in the Credit Token Bucket can then be consumed at an arbitrary rate. Rate limiting will only take place after all credits in the Credit Token Bucket have been consumed. This means that a credit usage spike over daily credit limits, such as an urgent investigation, will not cause limiting until the built up balance is consumed. If your organization’s users are being blocked from using Observe unexpectedly, please reach out to your Observe Data Engineer.

Limiting Acceleration Credits

Unlike Queries, Observe performs Dataset Acceleration as a background activity, so your users do not receive explicit warnings or alerts when exhausting the Acceleration Credit usage. Instead, when the Observe account exhausts the Acceleration Credit Limit in seven (7) days, Observe automatically loosens the freshness goals for Datasets in Observe to reduce cost and stay within credit limits.

Observe automatically re-tunes the freshness goal to the original value when the Acceleration Credit consumption returns below the set limit. For example, suppose you set the freshness goal for the Container Logs Dataset to 60 seconds. If the Acceleration Credit usage exceeds the limit for an extended period of time, Observe may automatically adjust this freshness goal to 120 seconds up to a maximum goal of one hour.

With a looser freshness goal, Observe performs Data Acceleration on each dataset less often, reducing the Acceleration Credit consumption rate. However, users may experience delays until data becomes available in the datasets. Optimizing Acceleration Credit consumption trades off cost and freshness goals, for example, latency between data ingest and the ability to query the data.

Using Credit Manager Effectively

The following tips guide you to effectively using and configuring Credit Manager:

  • Use Credit Manager as a “safety net” to prevent excessive credit consumption and unexpected costs. When you use Credit Manager for the first time, start by setting the limit to twice the number of credits you plan to consume. If you forecast that you may consume 3,000 Query Credits in a month, set the Query Credit Limit to 3,000/30 x 2 = 200, which minimizes disruption to your users.

  • The Credit Manager page contains a few statistics to help guide the configuration of your credit limits, such as Daily avg credits in past 30d and Credits used past 24hr. Use the Usage Dashboard to view credit usage in more detail and over longer ranges of time to assist with configuring your limits.

  • If you find that your users reach the Query Credit Limit daily, use the following suggestions:

    • Use shorter query time ranges such as Past 24 hours rather than long time ranges such as Past 30 days. When Observe queries with wider time ranges on large Datasets, a vast amount of data must be scanned. To make queries more efficient, use a shorter time range necessary to satisfy your query.

    • If your users query a large Dataset, for example, Container Logs, you should create smaller Datasets derived from the large Dataset to target a common use case such as API Server Errors. Although creating a Dataset may consume Acceleration Credits, users share the accelerated Dataset and query it. This can result in a lower net consumption of total Credits. Also, queries on smaller datasets can be faster and provide a better user experience.

Ideally, you should adjust the freshness goals of your Datasets rather than allow Credit Manager to re-tune them for you. For example, suppose you have a Dataset that you infrequently query, such as VPC Flow Logs. In that case, you can set the freshness goal to 30 minutes to reserve more of your Acceleration Credit budget for more critical Datasets with tighter freshness requirements. Reach out to your Observe Data Engineer for assistance with re-tuning your Dataset freshness goals.

For more information on Query Credits and Acceleration Credits usage consult the Usage Dashboard.