Microsoft Azure¶
The Observe Azure app allows you to collect data from your Azure resources within your subscription. Based on the region, the Azure app simultaneously ingests logs and metrics from standard Azure services.
The Observe Azure app works with the datasets within the application. Observe can automate many everyday data modeling tasks, ensuring an accurate picture of your infrastructure. Observe can also update your workspace with new datasets when new functionality releases for this integration.
If you currently utilize Azure data, consult with your Observe Data Engineer to see if the Azure Integration could enhance your existing data collection strategy.
What data does Observe ingest?¶
Observe catalogs resources deployed at a given location (region) and collects logs and metrics from those resources.
Figure 1 - Overview of Observe Azure data collection using the Observe Function App
The Observe Function App contains three functions responsible for data collection:
event_hub_telemetry_func
- captures the data sent to the Event Hub from the Azure resources Diagnostic settings. Event Hub triggers this function and forwards the data to Observe.timer_resources_func
- returns all resources within the location (region) and the corresponding metadata. It runs on an assigned NCRONTAB schedule and set to every 10 minutes by default.timer_vm_metrics_func
- returns virtual machine metrics from the hypervisor. It runs on an assigned NCRONTAB schedule and set to every 5 minutes by default.
Standard Ingest Sources¶
The Observe Azure app ingests the following types of data from a single location (region):
Note
Microsoft Azure does not provide operating system (OS) metrics for virtual machines. Consider installing the Observe Host Monitoring app.
Using Azure Integration Data¶
After shaping, the incoming data populates datasets such as the following:
AD
Active Directory - Active Directory Logs which includes Audit Logs, Sign-In logs and Provisioning Logs
AKS
AKS Clusters - AKS Cluster Logs and Metrics
App Services
App Services - Overall App Service performance, usage, and tracing
Cognitive Services
Cognitive Service - Usage and Performance of Cognitive Services, such as OpenAI services
Functions
Function App - Overall FunctionApp performance and usage
Functions - Function performance
SQL Database
SQL Server - Servers SQL Databases employ
SQL Database - Logs audit, deadlocks, error, query, timeout activity, and metrics such as query store runtime and wait statistics.
SQL Managed Instances
SQL Managed Instance - Status and basic metrics of instances
SQL Managed Instance Databases - Logs of errors and metrics, including query store runtime and wait statistics for databases running within SQL Managed Instances
Storage Account
Storage Account - Basic info about storage account, logs about error and performance related to a Storage Account, blob and containers.
Blob Service - Overall info about blob service, performance metrics and associated storage account.
Blob Container - Overall info about blob container.
Virtual Machines
Virtual Machine - Status and basic metrics of your virtual machines
Virtual Machine Disk - Status and usage of allocated disks for your virtual machines
Installation¶
Observe Azure Collection Setup¶
Registering Resource Providers
In order for the collection to work and install properly, you must ensure that the following resource providers are registered for your subscription:
Microsoft.Insights
Microsoft.KeyVault
Microsoft.EventHub
Microsoft.Web
Verify that the following resource providers are also registered, as these are accessed by the timer_resources_func
function for sending resource metadata to Observe.
Microsoft.Compute
Microsoft.Storage
Microsoft.Network
Microsoft.Sql
Microsoft.ContainerService
Microsoft.Resources
Microsoft Documentation provides guidance on how to register resource providers using the Azure Portal, Azure CLI, and Azure PowerShell.
User Principal Access
The setup below shows how to install the collection by authenticating as a user principal via az login
The collection will create resources, such as an application registration and its associated service principal, which will require elevated privileges.
Ensure your user principal has the following subscription scoped roles at minimum:
Contributor
(Privileged administrator role)User Access Administrator
orApplication Administrator
(Privileged administrator role)
See Azure documentation for more information on how to assign roles.
Note
You may see an error similar to below during install if you do not have the required roles set correctly:
ApplicationsClient.BaseClient.Post(): unexpected status 403 with OData error: Authorization_RequestDenied: Insufficient privileges to complete the operation
Location
Note
The collection must be deployed to each region in each subscription.
When you create an Azure data collection using Terraform or Azure Resource Manager, you need to specify an Azure location. Find the location of your Azure services by looking at the Resource groups page or each Azure service page, such as Azure Virtual Machines. Use the following Azure CLI command to find the right naming convention for your Azure services:
$ az account list-locations --output table
DisplayName Name Regional Display Name
------------ ---- ---------------------
East US eastus (US) East US
East US 2 eastus2 (US) East US 2
South Central US southcentralus (US) South Central US
For example, if you have a Virtual Machine (VM) located in the East US, use location = "eastus"
in the Terraform configuration, or --location eastus
in Azure Resource Manager.
You can install the Azure app on the Apps page to install the Azure integration.
Click Configuration and select the Azure integrations to install using the app.
Select Connections.
Click Create connection. This creates an ingest token to use with the data stream.
Follow the instructions to create the ingest token.
Figure 2 - List of available Azure integrations
Before you begin, you should, at a minimum, have a datastream available to send Azure data, including an ingest token to be employed by the Data Collection Module terraform-azure-collection.
Install the Azure CLI and Azure’s Functions Core Tools
Be sure the Azure CLI installs correctly by logging into Azure.
az login
3. You should receive a token from your browser that looks like this:
[
{
"cloudName": "AzureCloud",
"homeTenantId": "########-####-####-####-############",
"id": "########-####-####-####-############",
"isDefault": true,
"managedByTenants": [],
"name": "Acme Inc",
"state": "Enabled",
"tenantId": "########-####-####-####-############",
"user": {
"name": "[email protected]",
"type": "user"
}
}
]
4. If you have multiple subscription / tenant, then use below command to set correct subscription/tenant where you would like to deploy Observe Azure collection stack -
az account set --subscription "My Demos"
OR
az login --tenant <tenant>
For details on Azure subscription/tenant follow here.
5. Clone the Observe Terraform Collection Module repository locally.
git clone [email protected]:observeinc/terraform-azure-collection.git
6. Assign the Application Variables.
Inside the root of the terraform-azure-collection, create a file called azure.auto.tfvars
. The file contains the following content:
observe_customer = "{OBSERVE_CUSTOMER_ID}"
observe_token = "{DATASTREAM_INGEST_TOKEN}"
observe_domain = "{OBSERVE_DOMAIN}" #i.e. observeinc.com
timer_resources_func_schedule = "{TIMER_TRIGGER_FUNCTION_SCHEDULE}"
timer_vm_metrics_func_schedule = "{TIMER_TRIGGER_FUNCTION_SCHEDULE}"
location = "{AZURE_REGIONAL_NAME}"
Your Observe Customer ID is the 12 digit number at the beginning of the URL that you access Observe with, such as
https://**123456789012**.observeinc.com
.Some Observe instances may optionally use a name instead of Customer ID; if this is the case for your instance, contact your Observe Data Engineer to discuss implementation. A stem name will work as is, but a DNS redirect name may require special configuration.
Note
The Azure app assigns these default values:
timer_resources_func_schedule
is the schedule used to collect data from your Azure resources.timer_vm_metrics_func_schedule
- is the schedule used to collect metrics on your Virtual Machines.
Both values based on NCRONTAB.
The location
value sets the Azure Regional Name to eastus
by default.
7. Deploy the Azure collection module.
Inside the root directory of the terraform-azure-collection module, run the following commands:
terraform init
terraform apply -auto-approve
Data collection should begin shortly.
Note
To capture logs and metrics of most resources, you must add diagnostic settings. Resource metadata collection is done automatically although data can take up to 20 minutes to show up in Observe. Check the Microsoft Azure function logs to ensure desired results.
8. Collecting data from multiple locations - To collect data from multiple locations, re-deploy the Observe Azure Collection with the updated location you want to collect data.
Create an App Registration by searching for App Registrations in the Search bar, and then clicking New registration. Create for the Registration.
Figure 1 - App Registration Overview after Creation
2. Create a Client Secret by clicking “Add a certificate or secret”.
Figure 2 - App Registration Client Secret
3. Record the following 3 values:
Secret Value (K3j8… in the above screenshot)
Application ID (58339 in the first screenshot)
Object ID (31a56… in the first screenshot)
4. Search for Enterprise Applications in the top search bar.
Figure 3 - Enterprise App Overview Page
5. Record the Enterprise App Object ID (3aca… in the above screenshot)
6. Initialize Azure Cloud Shell or another shell environment.
az login
7. If you have multiple subscription / tenant, then use below command to set correct subscription/tenant where you would like to deploy Observe Azure collection stack -
az account set --subscription "My Demos"
OR
az login --tenant <tenant>
For details on Azure subscription/tenant follow here.
8. Download the Observe Azure Resource Manager template.
Using Bash
curl -Lo resourcemanager-latest.json https://observeinc.s3.us-west-2.amazonaws.com/azure/resourcemanager-latest.json
Using PowerShell
Invoke-WebRequest -Uri https://observeinc.s3.us-west-2.amazonaws.com/azure/resourcemanager-latest.json -OutFile resourcemanager-latest.json
9. Enter the following text in the shell. Run the command to create the resource.
az deployment sub create \
--name observe \
--location <THE_AZURE_LOCATION_TO_MONITOR> \
--template-file resourcemanager-latest.json \
--parameters \
observe_customer=<YOUR_CUSTOMER_ID> \
observe_token=<YOUR_DATASTREAM_TOKEN> \
objectId=31a56... \
applicationId=58339... \
clientSecretValue=K3j8... \
enterpriseAppObjectId=3aca8...
10. After a few minutes, the command generates the subscription.
Figure 4 - Successful Deployment
Data collection should begin shortly.
Note
To capture logs and metrics of most resources, you must add diagnostic settings as shown below. Resource metadata collection is done automatically although data can take up to 20 minutes to show up in Observe.
11. Collecting data from multiple locations To collect data from multiple locations, re-deploy the Observe Azure Collection with the updated location you want to collect data.
Azure Resource Configuration¶
To capture logs and metrics of most resources, you must add diagnostic settings. Expand the desired Azure service name for instructions on adding diagnostic settings.
Active Directory
Log into the Azure portal.
Search for the Active Directory in the search field.
Figure 1 - Search for Active Directory
3. Go to Monitoring and Diagnostic settings. Select Diagnostic setting.
Figure 2 - Active Directory - Add Diagnostic Settings
4. Select the following options:
Diagnostic setting name - Observe
AuditLogs
SignInLogs
NonInteractiveUserSignInLogs
ServicePrincipalSignInLogs
ManagedIdentitySignInLogs
ProvisioningLogs
Stream to an event hub
5. Select the event hub created by the Observe collection function.
6. Click Save.
AKS
Note
For comprehensive Kubernetes monitoring, install the Observe Kubernetes App
Log into the Azure portal.
In the Azure search field, enter Kubernetes Services.
Figure 1 - Search for Kubernetes Services
3. Choose the service you want to monitor.
4. From the menu, select Diagnostic Settings.
Figure 2 - Kubernetes Services - Select Diagnostic Settings
5. Click Add diagnostic setting.
Figure 3 - Kubernetes Services - Add Diagnostic Setting
6. Create a name for your setting. Select all of the metrics listed, then click Stream to an Event hub. Be sure to select the Event hub namespace and Event hub name with your Observe customer ID.
Figure 4 - Kubernetes Services - Diagnostic Setting
7. Click Save.
App Services
Log into the Azure portal.
Search for the App Services in the search field and select the App Services.
Figure 1 - Search for App Services
3. Select the App Service you want to monitor.
Figure 2 - App Service - Select App Service
4. Search for Diagnostic settings and select it.
Figure 3 - App Service- Select Diagnostic Settings
5. Select Add diagnostic setting.
Figure 4 - App Services - Add Diagnostic Settings
6. Select the following options:
Diagnostic setting name - Observe
HTTP logs
App Service Console Logs
App Service Application Logs
Access Audit Logs
IPSecurity Audit logs
App Service Platform logs
All Metrics
Stream to an event hub
7. Select the event hub created by the Observe collection function.
8. Click Save.
If you have Application Insights enabled and want to get detailed info for your App Service:
1. Search for Application Insights and select it.
Figure 5 - App Services- Search Application Insights
2. Click View Application Insights data.
3. Search for Diagnostic settings and select it.
Figure 6 - App Services - Search Diagnostic Settings
4. Select Add diagnostic setting.
5. Select the following options:
Diagnostic setting name - Observe
All logs
All Metrics
Stream to an event hub
6. Select the event hub created by the Observe collection function.
7. Click Save.
Figure 7 - App Service - Diagnostic Setting
Cognitive Services
Log into the Azure portal.
Search for the Cognitive Services in the search field and select it.
Figure 1 - Search for Cognitive Services
3. Select the Cognitive Service Type you want to monitor.
Figure 2 - Cognitive Services - Select Service Type
4. Select the Cognitive Service you want to monitor.
Figure 3 - Cognitive Services - Select Service
5. Search for Diagnostic settings and select it.
Figure 4 - Cognitive Services - Select Diagnostic Settings
6. Select Add diagnostic setting.
Figure 5 - Cognitive Services - Add Diagnostic Settings
7. Select the following options:
Diagnostic setting name - Observe
All Logs
All Metrics
Stream to an event hub
8. Select the event hub created by the Observe collection function.
9. Click Save.
Figure 6 - Cognitive Services - Diagnostic Setting
Functions
Log into the Azure portal.
Search for the Function App in the search field and select the Function App.
Figure 1 - Search for Function App
3. Select the Function App you want to monitor.
Figure 2 - Function App - Select Function App
4. Search for Diagnostic settings and select it.
Figure 3 - Function App - Select Diagnostic Settings
5. Select Add diagnostic setting.
Figure 4 - Function App - Add Diagnostic Settings
6. Select the following options:
Diagnostic setting name - Observe
Function Application Logs
All Metrics
Stream to an event hub
7. Select the event hub created by the Observe collection function.
8. Click Save.
Figure 5 - Function App - Diagnostic Setting
SQL Database
Log into the Azure portal.
In the Azure search field, enter SQL Database.
Select your database.
Figure 1 - SQL Database - Select SQL Database
4. From the left menu, choose Diagnostic Settings and click Add diagnostic setting.
Figure 2- SQL Database - Add Diagnostic Setting
5. Choose all Logs, Audit Logs, and all Metrics. Send to the event hub with your customer ID.
Figure 3 - SQL Database - Diagnostic Setting
6. Create a name for Diagnostic Setting and click Save.
SQL Managed Instances
Log into the Azure portal.
In the Azure search field, enter SQL Managed Instances.
Select your SQL Managed Instance.
Figure 1 - SQL Managed Instances - Select SQL Managed Instance
4. From the left menu, choose Diagnostic Settings and click Add diagnostic setting.
Figure 2 - SQL Managed Instances - Add Diagnostic Setting
5. Select all Logs, Audit logs, and all Metrics. Send to event hub with your customer-id.
Figure 3 - SQL Managed Instances - Diagnostic Setting
6. Create a name for the diagnostic setting and click Save.
Storage Account
Log into the Azure portal.
Search for the Storage Accounts in the search field.
Figure 1 - Search for Storage Account
3. Go to the Monitoring section and select Diagnostic setting. This enables logs and metrics at the Storage Account level.
Figure 2 - Storage Account - Diagnostic Settings
4. Select Add Diagnostics Settings
Figure 2 - Storage Account - Add Diagnostic Settings
5. Select Metrics -> Transaction and select the event hub created by the Observe collection function.
6. Click Save.
Blob
1. Go to the Monitoring and Diagnostic settings. Select Diagnostic setting. Select blob under the Storage Account.
Figure 2 -Blob - Diagnostic Settings
2. Select Add Diagnostics Settings
3. Select Metrics -> Transaction. Select Logs -> Categories -> StorageRead, StorageWrite, StorageDelete.
4. Select the event hub created by the Observe collection function.
5. Click Save.
Virtual Machines
Currently, the Azure app collects this data using the timer_resources_func
and timer_vm_metrics_func
functions within the Observe Function app deployed. It does not require any diagnostic settings.
Creating Diagnostic Settings at Scale
Creating Azure Policies¶
If you want to create Diagnostic Settings for all resources of a resource type, you can configure an Azure Policy. Create custom policy definitions for each type of resource to monitor. Microsoft provides documentation on how to accomplish this.
Once you create the policies in your Azure account, Observe recommends creating a policy initiative, assigning the relevant policies, assigning that policy to a Management Group, and then adding your subscriptions to that management group.
Note
Complete these steps before moving on to the next section.
Creating a Management Group¶
Log into the Azure portal.
In the Azure search field, enter Management Groups.
Click Create.
Give the policy an ID such as observeDiagnosticSettings.
Set the policy display name such as Observe Diagnostic Settings.
Figure 1 - Settings for creating a management group
6. Click Submit.
Populating the Management Group¶
Once you create the management group, you need to move any other management groups or subscriptions to apply under the Observe Diagnostic Settings management group. Use the following steps:
Log into the Azure portal.
In the Azure search field, enter Management Groups.
Click on the three dots on the right of the management group or subscription.
Click Move.
Select the Observe Diagnostic Settings management group as the destination.
Click Save.
Creating an Initiative¶
An initiative consists of a collection of one or more policies that allow for simpler policy management. To create an initiative, use the following steps:
Log into the Azure portal.
In the Azure search field, enter Policy.
Select Definitions from the left menu of the page.
Click Initiative Definition at the top of the page.
Choose the management group created in the previous step for Initiative location.
Set the name of the initiative to Send to Observe.
Set the category to Monitoring after selecting on Use existing.
Figure 2 - Basic settings for creating initiative
8. Click Next.
9. Select all of the policies for the diagnostic settings for the resource types to monitor.
10. Click Initiative Parameters at the top of the page.
11. Click Create initiative parameter to create a new initiative parameter.
12. Create an initiative parameter called azureRegions. Set the type to array, and add a strong type of location. Add a default value, and then click Save.
Figure 3 - Azure Regions initiative parameter
Create an initiative parameter called eventHubName. Set the type to string, and give it a strong type of Generic - Microsoft.EventHub/Namespaces/EventHubs.
Select yes for Assign Permissions, and then click Save.
Figure 4 - Azure event hub name initiative parameter
3. Create an initiative parameter called eventHubRuleId.
4. Set the type to string, and add a strong type of Generic - Microsoft.EventHub/Namespaces/AuthorizationRules.
5. Select yes for Assign Permissions, and then click Save.
Figure 5 - Azure event hub authorization rule initiative parameter
Create an initiative parameter called metricsEnabled.
Set the type to string, and then set the allowed value to [“True”, “False”].
Set the default value to True and then click Save.
Figure 6 - Azure metrics enabled initiative parameter
Create an initiative parameter called logsEnabled. Set the type to string, and set the allowed value to [“True”, “False”].
Set the default value to True and then click Save.
Figure 7 - Azure logs enabled initiative parameter
Create an initiative parameter named profileName.
Set the type to string, and then set the default value to SendToObserve. 3. Click Save.
Figure 8 - Azure profile name initiative parameter
Navigate to the Policy Parameters section.
Clear the Only show parameters that need input or review checkbox.
Set every Value Type to Use Initiative Parameter and select the corresponding initiative parameter.
Figure 9 - Azure Initiative Policy Parameter Settings
4. Click Review and then Create.
Assigning the Initiative¶
Assign the initiative to each region you want to monitor using the following steps:
Log into the Azure portal.
In the Azure search field, enter Policy.
Select Assignments.
Click Assign Initiative.
Choose the management group created in the previous step as the Scope.
Select the Send to Observe initiative created in the previous section.
Name the assignment Send to Observe -
.
Figure 10 - Initiative Assignment Basic Settings
Click Parameters.
Clear the Only show parameters that need input or review checkbox.
Select the region to deploy for the azureRegions parameter.
Select the eventhub name from the region.
Figure 11 - Initiative Assignment Parameters Event Hub name
5. Select the Eventhub authorization rule for the Eventhub.
Figure 12 - Initiative Assignment Event Hub Authorization Rule
Figure 13 - Initiative Assignment Full Parameter Listing
6. Click Review + create and then Create.
Once you create the assignment, any new resources of the resource type automatically receive the diagnostic settings to send to Observe created a short period after you create the resource.
Remediation¶
To remediate non-compliant resources from the Azure portal, perform the following steps:
Log into the Azure portal.
In the Azure search field, enter Policy.
Click Remediation.
Find the policy for the resource type you want to remediate and click the 3 dots.
Click Remediate.
Click Remediate again on the Remediation page.
This creates the diagnostic settings for all of those resources sent to Observe.
FAQ¶
Troubleshooting Azure¶
Note
Be sure you install Diagnostic Settings for your integration (metrics and logs) before you begin to troubleshoot your instance. If none are configured, then only resource metadata will show up in Observe which can first take up to 20 minutes.
Q: I’ve installed via Azure Resource Manager or the Observe collection module terraform-observe-collection, but I don’t see any data in Observe. How can I troubleshoot the app?
Check to see if your deployment settings are correct, whether through parameters of Azure Resource Manager or through variables in
azure.auto.tfvars
if using Terraform.It’s common to swap values for
objectId
andenterpriseAppObjectId
, or to introduce spaces or other extra characters in these settings.Check Observe ingest token
observe_token
is valid in the Observe instance and the corresponding datastream has not been deleted.Enable Azure’s Application Insights and select your Observe Resource Group. Ensure that the application functions perform correctly.
Figure 4 - Enabling Application Insights On Your Resource Group
Figure 5 - Select Application Insights Inside Your Resource Group
Figure 6 - Review Observe Application Insights
Note
If you decide to remove the Observe collection module, remove Azure Application Insights from the portal first.
If the functions exist, try restarting the instance for your Observe Function App.
Figure 7 - Restart Function App Machine
Q: I see logs & metrics data in Observe but I don’t see any resource metadata in Observe. How can I troubleshoot?
Navigate to
timer_resources_func
function in the function app and ensure that successful invocations are taking place as in the below image:
If there are failing invocations, navigate to the “Monitor” tab and click on an invocation to view failure details.
By default, Terraform or ARM will automatically attempt to register any resource providers that it supports. Verify that the following resource providers are at least registered, as these are accessed by the
timer_resources_func
function. See here for more information.Microsoft.Compute
Microsoft.Storage
Microsoft.Network
Microsoft.Insights
Microsoft.Sql
Microsoft.Web
Microsoft.KeyVault
Microsoft.ContainerService
Microsoft.Resources
Microsoft.EventHub
Q: How do I send Azure collection function logs into Observe?
To send Azure collection function logs into Observe for an Observe engineer to troubleshoot, enable the diagnostic settings on the Observe function app:
Log into the Azure portal.
Search for the Function App in the search field and select the Function App.
Figure 1 - Search for Function App
3. Select the Observe collection function called observeApp-<OBSERVE-CUSTOMER-ID>-<AZURE-LOCATION>-<LAST-8-CHARS-OF-SUBSCRIPTION>
.
4. Search for Diagnostic settings and select it.
Figure 2- Function App- Select Diagnostic Settings
5. Select Add diagnostic setting.
Figure 3 - Function App - Add Diagnostic Settings
6. Select the following options:
Diagnostic setting name - observe
Function Application Logs
AllMetrics
Stream to an event hub
7. Select the event hub created by the Observe collection function. This will look like observeeventhub-<OBSERVE-CUSTOMER-ID>-<AZURE-LOCATION>-<LAST-8-CHARS-OF-SUBSCRIPTION>
.
8. Click Save.
Q: I’m receiving HTTP status code 429 “Too many requests” errors in Function Logs. What should I do?
The 429 HTTP status is commonly used to reject a request because a call rate limit is reached. The client will auto retry requests but may raise an exception after repeated retries. This usually happens in the timer_resources_func
function by Microsoft.Storage
client’s API calls, which has stricter rate limits for management operations.
Rate limiting may happen when there are large numbers of blob services and/or blob containers in a storage account. It is recommended to disable their collection by adding a feature flag as part of the function app environment variables, under App Settings.
FEATURE_FLAGS=!blob_services,!blob_containers
Figure 4 - Function App - Add additional environment variables
If managing via Terraform, this can be set by adding the following app_settings
variable to your azure.auto.tfvars
file
app_settings = {
FEATURE_FLAGS = "!blob_services,!blob_containers"
}
If managing via ARM, this can be set by adding the feature_flags
parameter to the ARM deployment command as below:
az deployment sub create \
--name observe \
--location <THE_AZURE_LOCATION_TO_MONITOR> \
--template-file resourcemanager-latest.json \
--parameters \
observe_customer=<YOUR_CUSTOMER_ID> \
observe_token=<YOUR_DATASTREAM_TOKEN> \
objectId=31a56... \
applicationId=58339... \
clientSecretValue=K3j8... \
enterpriseAppObjectId=3aca8... \
feature_flags='!blob_services,!blob_containers'
Note
Before enabling feature flags, please ensure you’re on the latest version of function app.
Q: Why don’t some of the messages in Function Logs link back to the Python Function Resource?
Any logging performed by functions in Python should use the logging module and not the print()
function. Using the print()
function generates separate events for each line and does not include the required metadata to tie it back to a function, or, in the case of multi-line stack traces, the other lines of the stack trace.