Working with the Log Explorer¶
Your organization may generate large amounts of log data and events through applications, networks, systems, and users, and requires a systematic process to manage and monitor different data across log files. Log management requires a continuous process of centrally collecting, parsing, storing, analyzing, and disposing of data to provide actionable insights for supporting troubleshooting, performance enhancement, or security monitoring.
Computer-generated data that serves as a primary source for information about activities, usage patterns, and operations on your applications, servers, or other devices can be stored as log files. Use log files to identify and analyze situations where applications, networks, and systems experience bottlenecks or performance issues. Log files provide detailed information about every action and provide insights into identifying root causes of anomalies or problems.
Managing log files requires collecting data from multiple sources of logs, and these are the most common types of log files:
System logs - logs that record events generated within an OS, such as driver errors or CPU usage.
Application logs - logs generated when an event occurs inside an application. Use application logs to measure and understand how your application functions after releasing it or during the development cycle.
Security logs - logs generated when security events such as unsuccessful login attempts, failed authentication requests, or password changes occur in your organization.
In the Getting Started with Observe tutorial, you investigated log events for a Kubernetes container Dataset using a Worksheet and OPAL. Using Log Explorer allows you to locate the logs in a Dataset without initially creating a Worksheet and filtering for the logs.
To start using the Log Explorer, log into your Observe instance and locate the Logs icon on the left navigation bar. Click the icon to display the following interface:
![Log Explorer Interface](../../_images/log-explorer1.png)
Figure 1 - Log Explorer Interface
Quick Start Using Kubernetes Container Logs¶
This quick tutorial builds on the Getting Started with Observe material, but instead of looking for errors in the logs, you want to know about the errors responses that occur in the logs.
Log into your Observe instance and use the following steps to search for errors in your Kubernetes Container Logs:
From the left navigation bar, under Investigate, click Logs.
In the Search log datasets field, enter Container Logs, and select it from the search results.
From the Filters list for the Dataset, select frontend to view only errors from the front end.
In the Filter field, enter error. Corresponding entries in the Container Logs appear highlighted in the log column.
![Container Logs with frontend and error Filters](../../_images/frontend-filter.png)
Figure 2 - Container Logs with frontend and error Filters
Filter out Unimplemented as you don’t need this data for analysis.
Filter for
log=customerID
and review errors associated with customers.
Live Mode¶
Note
Only customers with usage-based pricing can access this feature.
You can click Live Mode when viewing logs and see your logs stream into Observe. Filter your logs and generate visualizations that continuously update with new data.
![Live Mode](../../_images/live-mode.png)
Figure 5 - Enabling Live Mode
Since Live Mode increases your credit usage, you may want to disable it unless actively working on troubleshooting an ongoing issue. Live Mode automatically becomes disabled after 15 minutes. Using the Time Scrubber feature also automatically disables Live Mode.
For Log Explorer, you can select from 5 minutes, 10 minutes, or 15 minutes.
When you enable Live Mode, and click the Query icon, you see information about the query similar to the following image.
![Live Mode Query Details](../../_images/live-data.png)
Figure 6 - Live Mode Query Details
Latest data received - the time that data required for the query most recently arrived on the Observe instance but has not yet been processed.
Latest data available to query - the latest system time at which new data was processed and became available for Observe to query it. Live Mode users can typically expect between 30 and 90 seconds of latency from source to screen, depending on data rate and agent configuration.
These two status messages may have slightly different times as the first one designates the time that the data required for the query most recently arrived on the Observe instance but has not yet been processed. The second message designates the time the data became available for Observe to query it.
Token Search¶
Note
To enable this feature, please contact Observe Support.
When working with a dataset, searching for a specific ID or keyword with logs accumulated over time may only return a few results. The search accesses a lot of data and needs considerable computation time for scanning the logs, which results in considerable costs and high query latency.
To make your search return results more quickly, Observe supports Token Search by building an underlying token index. This feature supports only complete tokens and is case-insensitive.
The Tokenization uses a combination of major and minor separators to divide the log strings into tokens.
Major Separators¶
List of major separators:
'[', ']', '<', '>', '(', ')', '{', '}',
'|', '!', ';', ',', '*', '&', '?', '+', '\'', '"', '\n',
'\r', '\t', ' ', '=',
0x1D, // Group Separator ASCII
0x1E, // Record Separator ASCII
0x1F, // Unit Separator ASCII
// Multi byte
'␝', // U+241D - Group Separator UTF-8
'␞', // U+241E - Record Separator UTF-8
'␟', // U+241F - Unit Separator UTF-8
Minor Separators¶
List of minor separators:
'/', ':', '@', '.', '-', '$', '#', '%', '\\', '_'
Tokenization Rules¶
The indexing is following these tokenization rules::
Divide the input string using only major separators. All substrings become tokens.
Hello World-OutThere → ["Hello", "World-OutThere"]
2. Divide the input string using major and minor separators. All substrings become tokens.
Hello World-OutThere → ["Hello", "World-OutThere", "World", "OutThere"]
3. Divide the input string using major separators and then divide the output string to only include tokens with the same minor separator.
Hello 127.0.0.1:8080 → ["Hello", "127.0.0.1:8080", "127", "0", "0", "1", "8080", "127.0.0.1", "1:8080"]
Note
Tokens smaller than five (5) characters are not indexed as they are not selective enough for the search to be effective.
Testing the Token Rules
make_col tokenized_logs:test_index_tokenization(log, "major_minor_v1")
make_col tokenized_string:test_index_tokenization("Hello world", "major_minor_v1")
Observe only supports the tokenizer ‘major_minor_v1’, and you don’t need to specify the tokenizer if the dataset already has an index.
Querying with a Token or Tokens
filter column ~ <token>
requires a column name and at lesat a single token. The OPAL function identifies when a token contains a major separator and if there is not a real token. The function can only be used with filter
.
filter log ~ <hello>
---
filter log ~ <hello> and <world>
---
filter log ~ <hello world>
---
filter log ~ <hello>
filter log ~ <world>
Case Insensitive
The following queries will return the same results.
filter log ~ <hello>
filter log ~ <Hello>
filter log ~ <HELLO>
filter log ~ <"hello">
filter log ~ <"Hello">
filter log ~ <"HELLO">
filter contains_token(log, "hello")
Tokens are not in order
The following queries will return the same results.
filter log ~ <hello world>
filter log ~ <world hello>
If you need a phrase search, use the following query to search for a token first, and then check the order.
filter log ~ <hello world> and log ~ "hello world"
Token search with conjunctions and disjunctions
You can combine token search with boolean expressions queries
filter log ~ <hello> and log ~ <world>
filter log ~ <hello> or log ~ <aloha>
filter log ~ <aloha> or (log ~ <hello> and log ~ <world>)
Token search can also be combined with other predicates
filter log ~ <hello> and column = 100
filter log ~ <hello> or column = 100
Accelerating boolean expression queries with index
For queries that use conjunctions of predicates (i.e., AND), query can be accelerated by index if any of the predicates can use token index, equality index, or substring index.
For queries that use disjunctions of predicates (i.e., OR), query can be accelerated by index if all predicates can use token index, equality index, or substring index.
Example Errors
filter log !~ <hello>
--> token search can only be combined using `and` with other token search functions.
---
filter stream ~ <hello>
--> no index found on column "stream", available indexed column(s): "log".
Use the following examples to create queries on your tokenized dataset:
All indexed columns
filter * ~ <token>
Single column
filter column ~ <token>
Log Explorer Overview¶
The left menu displays a Search that allows you to search for specific Log Datasets.
Log Explorer displays a list of available Log Datasets logs. If you don’t see the desired Datasets, use the Search function to locate it.
The left menu also displays a list of Filters to use with the currently selected Logs dataset. When you select a Filter, the Filter appears in the Query Builder.
In the center panel, you can build filters using the Query Builder and select from a dropdown list of parameters, or you can select OPAL and use OPAL to build your filter list.
Click Logs to display the datasets as log events. Selecting a row from the list of Logs displays the Log details in the right panel.
While in Logs mode, you can use column formatting tools to filter, sort, and visualize the data. Use a right-click on the column header to display options for working with data in a single column. The type of options depends on the data type in the column.
Filter - filter data by a single parameter.
Remove empty cells - remove empty cells from the column.
Create as visualization - create a visual representation of data in a column.
Create summary - add a summary of the data in a column.
Extract from string - extract parameters depending on the type of string.
Sort A -> Z (descending order)
Sort Z -> A (ascending order)
Conditional Formatting - Apply color and style to string or numeric column types based on conditional formatting rules
Hide column - hide the column from view.
Convert -
int - integer
float - floating-point type
time - timestamp, date, time, interval
JSON - JavaScript Object Notation type
Add parameters
Existing - add existing parameters to the column data.
Create new - create a new parameter.
Add to resource
Datasets - add the column data to an existing Dataset.
Add to new - add the column data to a new Dataset
Link to other dataset
Datasets - link the column data to a listed Dataset.
View more - list more Datasets.
Click Table to display the Log Events in a read-only table format.
Click Visualize to display the Log Events as a visualization of the events. When you click Visualize, you can then build an expression from the displayed parameters.
From the Type menu, select from the following (see the Visualization Types Reference for more detail):
Line Chart
Bar Chart
Stacked Area
Single Stat
Pie Chart
List
Value Over Time
Geographic
Choropleth
From the Plot menu, select from the following:
over time
summary
as is
From the using function menu, select from the following:
Any
Any Not Null
Array Aggregation
Array Aggregation with duplicates remembered
Average
Count Distinct Exact
Count Distinct Fast
Count Values
Deriv
First
First Not Null
Last
Last Not Null
Maximum
Median
Median Exact
Minimum
Percentile
Rate
Standard Deviation
Sum
From the of menu, select from the displayed parameters. This list varies from Dataset to Dataset.
From the by menu, select from the displayed parameters. This list varies from Dataset to Dataset.
The top right contains a button, Actions, that allows you to perform the following tasks:
Create monitor - create a Monitor to watch your Log Dataset for desired events.
Add to dashboard - use the Log Dataset to create a new Dashboard.
Open data in worksheet - open the data in a Worksheet to further model and refine the data.
![Actions Menu Options](../../_images/actions-menu.png)
Figure 7 - Select From a list of Actions
You can also select a time range for your data using the menu to the left of Actions.
![Select a time range for your data](../../_images/time-picker.png)
Figure 8 - Select a Time Range for your data
When you right-click on a line in a Time Series visualization, this displays a menu that allows you to select from two options:
Show this data only - This allows you to display only that graph line in the visualization.
Exclude this data - Remove the data from the visualization.
Copy - Copy the graph line.
Inspect - Inspect the data for the graph line.
For selected resource - Displays the related resource which you can open in a new window.
View related - View the following related data in new windows:
Dashboard
Metrics
Logs
![Options for a single graph line](../../_images/line-options.png)
Figure 9 - Select From a List of Options for a Single Graph Line
Exporting Data¶
To download the data displayed in Log Explorer, click the Export button. You may select CSV or JSON format, and a maximum size limit (one thousand, ten thousand, or one hundred thousand rows). Note that hidden fields will be included. Use the pick_col OPAL verb to reduce the width of downloaded data.