Helpful Hints¶
Sometimes we have handy little tips that haven’t yet made it to a documentation page. The suggestions here may be updated or moved, if there’s something you are looking for, try Quick Search.
Account details¶
Customer ID¶
When you log into Observe, your Customer ID is the subdomain of the URL you use to access Observe. Example: 123456789012.observeinc.com
System Data¶
The Observe system stores information about itself in the System datastream and dataset. This datastream contains many different types of observations about activity in your workspace. The Observe Usage reporting dashboards and monitors are a useful way to interact with System data.
The System dataset can also be useful for troubleshooting issues ingesting data into Observe. To investigate ingest issues, open the System dataset and filter for OBSERVATION_KIND equals ingest_error
. Use the Extract from JSON
tool on the FIELDS column to find error types, messages, and more useful information. For assistance with the ingest, contact your data engineer.
OPAL¶
Changing a field type¶
Change the type of an existing field by creating a new one with the desired type. You may keep both fields or replace the existing one by giving it the same name.
colmake foo:float64(foo)
Aggregation Options¶
Here’s a way to pick the correct aggregation term for the task at hand:
If you need to align metrics by time, use align. Alignment regularizes time series data into regularly spaced bins on a grid so it can be compared with other aligned time series.
If you need to aggregate numbers by space, use aggregate. Aggregation computes a single numeric value for each of the points in an aligned series so that series can be compared. If you look at visualizations automatically created by the Metrics Explorer, you will often see
align
andaggregate
used together.If you need to align any type of data across time and space across the whole query window, use statsby. The
statsby
command aligns and aggregates in a single command. It is not accelerable.If you need to align any type of data across time and space, either bucketized “for every X minutes” or across the whole query window, use timechart. The
timechart
command aligns and aggregates in a single command.
Customized metric aggregation¶
Do common metric aggregation operations with the aggregate
verb:
rollup options(buckets:100), cpu_usage:metric("cpu_usage_total", rollup:"rate", type:"cumulativeCounter")
aggregate avg_cpu_usage:avg(cpu_usage), group_by(cluster_uid, node_name, cpu_id)
You can also form more advanced aggregation operations with it as well. For example, create a weighted average using the following code:
rollup options(buckets:100), cpu_usage:metric("cpu_usage_total", rollup:"rate", type:"cumulativeCounter")
colmake weight:case(
contains(cpu_type, "expensive"), 2.0,
contains(cpu_type, "normal"), 1.0)
aggregate avg_cpu_usage:avg(cpu_usage * weight), group_by(cluster_uid, node_name)
Filter¶
Comparisons
filter temperature > 60 and temperature < 80
filter temperature < 30 or temperature > 100
filter hostname="www" or (hostname="api" and user="root")
filter not severity="DEBUG"
Operators vs Functions
Construct expressions with either operators or functions. For example, these two statements are equivalent:
filter abc < 100
filter lt(abc, 100)
if_null¶
For example, a source error resulted in JSON data with similar values but different key names.
FIELDS
{"data":"abc123"}
{"payload":"def456"}
{"data":"ghi789"}
Use if_null
to get the value from payload
if there is no
value for data
. Note that both values must be the same type.
colmake data:if_null(string(FIELDS.data), string(FIELDS.payload))
Performance¶
Limit your query window to 1 hour or less while actively modeling¶
By default, worksheets read 4 hours of data. Depending on the input dataset, that can be a lot of data. Consider reducing the query window to 1 hour or less while actively modeling.
Create intermediate event datasets when shaping data¶
Where possible, create an intermediate event dataset by publishing partially shaped data as a new event dataset. Queries and further derived datasets typically have to read less data than if you create them directly on top of the original input dataset.
This technique is especially effective if the intermediate dataset applies a selective filter to the input dataset, picks only a subset of input columns, or extracts JSON paths from an input column and then drops the original column.
Avoid defining datasets directly on a Datastream dataset.
Use options(expiry)
to reduce the time range read by make_resource
¶
By default, the make_resource
verb reads a large time range of input events: 24 hours. The reason for this behavior is that make_resource
must compute the state of each resource at the beginning of the query time range, and, by default, it looks for events up to 24 hours in the past. Thus, a query with make_resource
with a query time range of 4 hours actually reads at least 28 hours of input data.
24+ hours can be a lot of data, especially if the input dataset is a source dataset from a Datastream. Avoid defining resource datasets directly on a Datastream dataset and create a filtered intermediary instead.
Most resource types receive events much more frequently than every 24 hours. Observe recommends adding options(expiry:duration_hr(...))
to your make_resource
command to reduce the lookback where appropriate.
For example, if you know that the live instances of some resource dataset receive events at least every 15 minutes, it would be appropriate to set the resource expiration to 1 hour, thereby greatly reducing the amount of data read by make_resource
:
make_resource options(expiry:duration_hr(1)), col1:col1, primarykey(pk1, pk2)
Shaping data¶
Field name allowed characters¶
In most cases, field (column) names may contain any character except double quote
"
, period .
, or colon :
. Underscores are displayed as spaces in the UI.
colmake "ΔT":float64(field3)
colmake "占用率":float64(field4)
colmake "0_3µm":float64(um03)
To reference a field with non-alphanumeric characters in an OPAL statement, use
double quotes and prepend @.
.
colmake temp_difference:@."ΔT"
Regex extracted columns (either Extract From Text or extract_regex
) are limited to alphanumeric
characters (A-Z, a-z, 0-9).
UI¶
Supported Web Browsers¶
Observe works best with the latest versions of Chrome, Edge, Firefox, and Safari.
Understanding Time window selection¶
Dashboards, Worksheets, and Explorers operate on a time window set by the time picker in the top right. Many of the options in that picker are relative to the present, such as “Past 15 minutes”, “Last week”, or “Since $DATE”. After your initial selection, the start and end times will transition to static points so that the data in the window does not change. For example, a dashboard with a time filter of “Past 15 minutes” on load will set start and end times, and after a few minutes will adjust the time display to show that the observed time is a fifteen minute window in the past.
Change the number of results displayed¶
By default, an events table shows the first 1000 rows of results. You can change the number displayed in the Limit tab of the Table Controls menu.
Hide, show, or reorder columns¶
Also use Table Controls to hide, show, or change the order of columns displayed. In the Columns tab, click to show or hide, and drag to reorder.
Video instructions
Data Table Settings¶
For each table in a Worksheet, Log Events, Metrics, and Datasets, you can adjust the table to suit your viewing needs. The following settings can be changed when you click the Table settings icon:
Columns: hide or unhide the columns in this data
View: change how many lines are displayed per row of data
Limit: change the maximum number of rows displayed