Microsoft Azure

The Observe Azure app allows you to collect data from your Azure resources within your subscription. Based on the region, the Azure app simultaneously ingests logs and metrics from standard Azure services.

The Observe Azure app works with the datasets within the application. Observe can automate many everyday data modeling tasks, ensuring an accurate picture of your infrastructure. Observe can also update your workspace with new datasets when new functionality releases for this integration.

If you currently utilize Azure data, consult with your Observe Data Engineer to see if the Azure Integration could enhance your existing data collection strategy.

What data does Observe ingest?

Observe catalogs resources deployed at a given location (region) and collects logs and metrics from those resources.

Observe app in Okta

Figure 1 - Overview of Observe Azure data collection using the Observe Function App

The Observe Function App contains three functions responsible for data collection:

  • event_hub_telemetry_func - captures the data sent to the Event Hub from the Azure resources Diagnostic settings. Event Hub triggers this function and forwards the data to Observe.

  • timer_resources_func - returns all resources within the location (region) and the corresponding metadata. It runs on an assigned NCRONTAB schedule and set to every 10 minutes by default.

  • timer_vm_metrics_func - returns virtual machine metrics from the hypervisor. It runs on an assigned NCRONTAB schedule and set to every 5 minutes by default.

Standard Ingest Sources

The Observe Azure app ingests the following types of data from a single location (region):

Note

Microsoft Azure does not provide operating system (OS) metrics for virtual machines. Consider installing the Observe Host Monitoring app.

Using Azure Integration Data

After shaping, the incoming data populates datasets such as the following:

  • AD

    • Active Directory - Active Directory Logs which includes Audit Logs, Sign-In logs and Provisioning Logs

  • AKS

    • AKS Clusters - AKS Cluster Logs and Metrics

  • App Services

    • App Services - Overall App Service performance, usage, and tracing

  • Cognitive Services

    • Cognitive Service - Usage and Performance of Cognitive Services, such as OpenAI services

  • Functions

    • Function App - Overall FunctionApp performance and usage

    • Functions - Function performance

  • SQL Database

    • SQL Server - Servers SQL Databases employ

    • SQL Database - Logs audit, deadlocks, error, query, timeout activity, and metrics such as query store runtime and wait statistics.

  • SQL Managed Instances

    • SQL Managed Instance - Status and basic metrics of instances

    • SQL Managed Instance Databases - Logs of errors and metrics, including query store runtime and wait statistics for databases running within SQL Managed Instances

  • Storage Account

    • Storage Account - Basic info about storage account, logs about error and performance related to a Storage Account, blob and containers.

    • Blob Service - Overall info about blob service, performance metrics and associated storage account.

    • Blob Container - Overall info about blob container.

  • Virtual Machines

    • Virtual Machine - Status and basic metrics of your virtual machines

    • Virtual Machine Disk - Status and usage of allocated disks for your virtual machines

Installation

Observe Azure Collection Setup

Registering Resource Providers

In order for the collection to work and install properly, you must ensure that the following resource providers are registered for your subscription:

  • Microsoft.Insights

  • Microsoft.KeyVault

  • Microsoft.EventHub

  • Microsoft.Web

Verify that the following resource providers are also registered, as these are accessed by the timer_resources_func function for sending resource metadata to Observe.

  • Microsoft.Compute

  • Microsoft.Storage

  • Microsoft.Network

  • Microsoft.Sql

  • Microsoft.ContainerService

  • Microsoft.Resources

Microsoft Documentation provides guidance on how to register resource providers using the Azure Portal, Azure CLI, and Azure PowerShell.

User Principal Access

The setup below shows how to install the collection by authenticating as a user principal via az login

The collection will create resources, such as an application registration and its associated service principal, which will require elevated privileges.

Ensure your user principal has the following subscription scoped roles at minimum:

  • Contributor (Privileged administrator role)

  • User Access Administrator or Application Administrator (Privileged administrator role)

See Azure documentation for more information on how to assign roles.

Note

You may see an error similar to below during install if you do not have the required roles set correctly: ApplicationsClient.BaseClient.Post(): unexpected status 403 with OData error: Authorization_RequestDenied: Insufficient privileges to complete the operation

Location

Note

The collection must be deployed to each region in each subscription.

When you create an Azure data collection using Terraform or Azure Resource Manager, you need to specify an Azure location. Find the location of your Azure services by looking at the Resource groups page or each Azure service page, such as Azure Virtual Machines. Use the following Azure CLI command to find the right naming convention for your Azure services:

$ az account list-locations --output table

DisplayName        Name             Regional Display Name 
------------       ----             ---------------------
East US		      eastus            (US) East US
East US 2         eastus2          (US) East US 2
South Central US  southcentralus   (US) South Central US

For example, if you have a Virtual Machine (VM) located in the East US, use location = "eastus" in the Terraform configuration, or --location eastus in Azure Resource Manager.

You can install the Azure app on the Apps page to install the Azure integration.

  1. Click Configuration and select the Azure integrations to install using the app.

  2. Select Connections.

  3. Click Create connection. This creates an ingest token to use with the data stream.

  4. Follow the instructions to create the ingest token.

List of available Azure integrations for Observe

Figure 2 - List of available Azure integrations

Before you begin, you should, at a minimum, have a datastream available to send Azure data, including an ingest token to be employed by the Data Collection Module terraform-azure-collection.

  1. Install the Azure CLI and Azure’s Functions Core Tools

  2. Be sure the Azure CLI installs correctly by logging into Azure.

  az login

3. You should receive a token from your browser that looks like this:

    [
      {
        "cloudName": "AzureCloud",
        "homeTenantId": "########-####-####-####-############",
        "id": "########-####-####-####-############",
        "isDefault": true,
        "managedByTenants": [],
        "name": "Acme Inc",
        "state": "Enabled",
        "tenantId": "########-####-####-####-############",
        "user": {
          "name": "[email protected]",
          "type": "user"
        }
      }
    ]

4. If you have multiple subscription / tenant, then use below command to set correct subscription/tenant where you would like to deploy Observe Azure collection stack -

  az account set --subscription "My Demos"

OR

  az login --tenant <tenant>

For details on Azure subscription/tenant follow here.

5. Clone the Observe Terraform Collection Module repository locally.

    git clone [email protected]:observeinc/terraform-azure-collection.git

6. Assign the Application Variables.

Inside the root of the terraform-azure-collection, create a file called azure.auto.tfvars. The file contains the following content:

    observe_customer = "{OBSERVE_CUSTOMER_ID}"
    observe_token = "{DATASTREAM_INGEST_TOKEN}"
    observe_domain = "{OBSERVE_DOMAIN}" #i.e. observeinc.com
    timer_resources_func_schedule = "{TIMER_TRIGGER_FUNCTION_SCHEDULE}" 
    timer_vm_metrics_func_schedule = "{TIMER_TRIGGER_FUNCTION_SCHEDULE}"
    location = "{AZURE_REGIONAL_NAME}"
  • Your Observe Customer ID is the 12 digit number at the beginning of the URL that you access Observe with, such as https://**123456789012**.observeinc.com.

  • Some Observe instances may optionally use a name instead of Customer ID; if this is the case for your instance, contact your Observe Data Engineer to discuss implementation. A stem name will work as is, but a DNS redirect name may require special configuration.

Note

The Azure app assigns these default values:

  • timer_resources_func_schedule is the schedule used to collect data from your Azure resources.

  • timer_vm_metrics_func_schedule - is the schedule used to collect metrics on your Virtual Machines.

Both values based on NCRONTAB.

The location value sets the Azure Regional Name to eastus by default.

7. Deploy the Azure collection module.

Inside the root directory of the terraform-azure-collection module, run the following commands:

      terraform init
      terraform apply -auto-approve

Data collection should begin shortly.

Note

To capture logs and metrics of most resources, you must add diagnostic settings. Resource metadata collection is done automatically although data can take up to 20 minutes to show up in Observe. Check the Microsoft Azure function logs to ensure desired results.

8. Collecting data from multiple locations - To collect data from multiple locations, re-deploy the Observe Azure Collection with the updated location you want to collect data.

  1. Create an App Registration by searching for App Registrations in the Search bar, and then clicking New registration. Create for the Registration.

    App Registration Overview after Creation

    Figure 1 - App Registration Overview after Creation

2. Create a Client Secret by clicking “Add a certificate or secret”.

App Registration Client Secret

Figure 2 - App Registration Client Secret

3. Record the following 3 values:

  • Secret Value (K3j8… in the above screenshot)

  • Application ID (58339 in the first screenshot)

  • Object ID (31a56… in the first screenshot)

4. Search for Enterprise Applications in the top search bar.

Enterprise App Overview Page

Figure 3 - Enterprise App Overview Page

5. Record the Enterprise App Object ID (3aca… in the above screenshot)

6. Initialize Azure Cloud Shell or another shell environment.

  az login

7. If you have multiple subscription / tenant, then use below command to set correct subscription/tenant where you would like to deploy Observe Azure collection stack -

  az account set --subscription "My Demos"

OR

  az login --tenant <tenant>

For details on Azure subscription/tenant follow here.

8. Download the Observe Azure Resource Manager template.

Using Bash

  curl -Lo resourcemanager-latest.json https://observeinc.s3.us-west-2.amazonaws.com/azure/resourcemanager-latest.json

Using PowerShell

  Invoke-WebRequest -Uri https://observeinc.s3.us-west-2.amazonaws.com/azure/resourcemanager-latest.json -OutFile resourcemanager-latest.json

9. Enter the following text in the shell. Run the command to create the resource.

az deployment sub create \
      --name observe \
      --location <THE_AZURE_LOCATION_TO_MONITOR> \
      --template-file resourcemanager-latest.json \
      --parameters \
        observe_customer=<YOUR_CUSTOMER_ID> \      
        observe_token=<YOUR_DATASTREAM_TOKEN> \
        objectId=31a56... \
        applicationId=58339... \
        clientSecretValue=K3j8... \
        enterpriseAppObjectId=3aca8... 

10. After a few minutes, the command generates the subscription.

Successful Deployment

Figure 4 - Successful Deployment

Data collection should begin shortly.

Note

To capture logs and metrics of most resources, you must add diagnostic settings as shown below. Resource metadata collection is done automatically although data can take up to 20 minutes to show up in Observe.

11. Collecting data from multiple locations To collect data from multiple locations, re-deploy the Observe Azure Collection with the updated location you want to collect data.

Azure Resource Configuration

To capture logs and metrics of most resources, you must add diagnostic settings. Expand the desired Azure service name for instructions on adding diagnostic settings.

Active Directory
  1. Log into the Azure portal.

  2. Search for the Active Directory in the search field.

Search for Active Directory

Figure 1 - Search for Active Directory

3. Go to Monitoring and Diagnostic settings. Select Diagnostic setting.

Add Active Directory Diagnostic Settings

Figure 2 - Active Directory - Add Diagnostic Settings

4. Select the following options:

  • Diagnostic setting name - Observe

  • AuditLogs

  • SignInLogs

  • NonInteractiveUserSignInLogs

  • ServicePrincipalSignInLogs

  • ManagedIdentitySignInLogs

  • ProvisioningLogs

  • Stream to an event hub

5. Select the event hub created by the Observe collection function.

6. Click Save.

AKS

Note

For comprehensive Kubernetes monitoring, install the Observe Kubernetes App

  1. Log into the Azure portal.

  2. In the Azure search field, enter Kubernetes Services.

Kubernetes search

Figure 1 - Search for Kubernetes Services

3. Choose the service you want to monitor.

4. From the menu, select Diagnostic Settings.

Diagnostic Settings

Figure 2 - Kubernetes Services - Select Diagnostic Settings

5. Click Add diagnostic setting.

Adding Diagnostic Settings

Figure 3 - Kubernetes Services - Add Diagnostic Setting

6. Create a name for your setting. Select all of the metrics listed, then click Stream to an Event hub. Be sure to select the Event hub namespace and Event hub name with your Observe customer ID.

Diagnostic Settings

Figure 4 - Kubernetes Services - Diagnostic Setting

7. Click Save.

App Services
  1. Log into the Azure portal.

  2. Search for the App Services in the search field and select the App Services.

Search for App Services

Figure 1 - Search for App Services

3. Select the App Service you want to monitor.

Select App Service

Figure 2 - App Service - Select App Service

4. Search for Diagnostic settings and select it.

Select Diagnostic Settings

Figure 3 - App Service- Select Diagnostic Settings

5. Select Add diagnostic setting.

Add App Services Diagnostic Settings

Figure 4 - App Services - Add Diagnostic Settings

6. Select the following options:

  • Diagnostic setting name - Observe

  • HTTP logs

  • App Service Console Logs

  • App Service Application Logs

  • Access Audit Logs

  • IPSecurity Audit logs

  • App Service Platform logs

  • All Metrics

  • Stream to an event hub

7. Select the event hub created by the Observe collection function.

8. Click Save.

If you have Application Insights enabled and want to get detailed info for your App Service:

1. Search for Application Insights and select it.

Search for Application Insights

Figure 5 - App Services- Search Application Insights

2. Click View Application Insights data.

3. Search for Diagnostic settings and select it.

Search for Diagnostic Settings

Figure 6 - App Services - Search Diagnostic Settings

4. Select Add diagnostic setting.

5. Select the following options:

  • Diagnostic setting name - Observe

  • All logs

  • All Metrics

  • Stream to an event hub

6. Select the event hub created by the Observe collection function.

7. Click Save.

Search for Function App

Figure 7 - App Service - Diagnostic Setting

Cognitive Services
  1. Log into the Azure portal.

  2. Search for the Cognitive Services in the search field and select it.

Search for Cognitive Services

Figure 1 - Search for Cognitive Services

3. Select the Cognitive Service Type you want to monitor.

Select a Cognitive Services Type

Figure 2 - Cognitive Services - Select Service Type

4. Select the Cognitive Service you want to monitor.

Select your Cognitive Services

Figure 3 - Cognitive Services - Select Service

5. Search for Diagnostic settings and select it.

Select Diagnostics for Cognitive Service

Figure 4 - Cognitive Services - Select Diagnostic Settings

6. Select Add diagnostic setting.

Add Diagnostics Setting for Cognitive Service

Figure 5 - Cognitive Services - Add Diagnostic Settings

7. Select the following options:

  • Diagnostic setting name - Observe

  • All Logs

  • All Metrics

  • Stream to an event hub

8. Select the event hub created by the Observe collection function.

9. Click Save.

Save Diagnostics Setting for Cognitive Service

Figure 6 - Cognitive Services - Diagnostic Setting

Functions
  1. Log into the Azure portal.

  2. Search for the Function App in the search field and select the Function App.

Search for Function App

Figure 1 - Search for Function App

3. Select the Function App you want to monitor.

Search for Function App

Figure 2 - Function App - Select Function App

4. Search for Diagnostic settings and select it.

Search for Function App

Figure 3 - Function App - Select Diagnostic Settings

5. Select Add diagnostic setting.

Search for Function App

Figure 4 - Function App - Add Diagnostic Settings

6. Select the following options:

  • Diagnostic setting name - Observe

  • Function Application Logs

  • All Metrics

  • Stream to an event hub

7. Select the event hub created by the Observe collection function.

8. Click Save.

Search for Function App

Figure 5 - Function App - Diagnostic Setting

SQL Database
  1. Log into the Azure portal.

  2. In the Azure search field, enter SQL Database.

  3. Select your database.

Search for Function App

Figure 1 - SQL Database - Select SQL Database

4. From the left menu, choose Diagnostic Settings and click Add diagnostic setting.

Search for Function App

Figure 2- SQL Database - Add Diagnostic Setting

5. Choose all Logs, Audit Logs, and all Metrics. Send to the event hub with your customer ID.

Search for Function

Figure 3 - SQL Database - Diagnostic Setting

6. Create a name for Diagnostic Setting and click Save.

SQL Managed Instances
  1. Log into the Azure portal.

  2. In the Azure search field, enter SQL Managed Instances.

  3. Select your SQL Managed Instance.

Search for Function

Figure 1 - SQL Managed Instances - Select SQL Managed Instance

4. From the left menu, choose Diagnostic Settings and click Add diagnostic setting.

Search for Function

Figure 2 - SQL Managed Instances - Add Diagnostic Setting

5. Select all Logs, Audit logs, and all Metrics. Send to event hub with your customer-id.

SQL managed instances Diagnostic Settings

Figure 3 - SQL Managed Instances - Diagnostic Setting

6. Create a name for the diagnostic setting and click Save.

Storage Account
  1. Log into the Azure portal.

  2. Search for the Storage Accounts in the search field.

Search for Storage Account

Figure 1 - Search for Storage Account

3. Go to the Monitoring section and select Diagnostic setting. This enables logs and metrics at the Storage Account level.

Storgae Account Diagnostic Settings

Figure 2 - Storage Account - Diagnostic Settings

4. Select Add Diagnostics Settings

Storgae Account Add Diagnostic Settings

Figure 2 - Storage Account - Add Diagnostic Settings

5. Select Metrics -> Transaction and select the event hub created by the Observe collection function.

6. Click Save.

Blob

1. Go to the Monitoring and Diagnostic settings. Select Diagnostic setting. Select blob under the Storage Account.

Blob Diagnostic Settings

Figure 2 -Blob - Diagnostic Settings

2. Select Add Diagnostics Settings

3. Select Metrics -> Transaction. Select Logs -> Categories -> StorageRead, StorageWrite, StorageDelete.

4. Select the event hub created by the Observe collection function.

5. Click Save.

Virtual Machines

Currently, the Azure app collects this data using the timer_resources_func and timer_vm_metrics_func functions within the Observe Function app deployed. It does not require any diagnostic settings.

Creating Diagnostic Settings at Scale

Creating Azure Policies

If you want to create Diagnostic Settings for all resources of a resource type, you can configure an Azure Policy. Create custom policy definitions for each type of resource to monitor. Microsoft provides documentation on how to accomplish this.

Once you create the policies in your Azure account, Observe recommends creating a policy initiative, assigning the relevant policies, assigning that policy to a Management Group, and then adding your subscriptions to that management group.

Note

Complete these steps before moving on to the next section.

Creating a Management Group

  1. Log into the Azure portal.

  2. In the Azure search field, enter Management Groups.

  3. Click Create.

  4. Give the policy an ID such as observeDiagnosticSettings.

  5. Set the policy display name such as Observe Diagnostic Settings.

Management Group Settings

Figure 1 - Settings for creating a management group

6. Click Submit.

Populating the Management Group

Once you create the management group, you need to move any other management groups or subscriptions to apply under the Observe Diagnostic Settings management group. Use the following steps:

  1. Log into the Azure portal.

  2. In the Azure search field, enter Management Groups.

  3. Click on the three dots on the right of the management group or subscription.

  4. Click Move.

  5. Select the Observe Diagnostic Settings management group as the destination.

  6. Click Save.

Creating an Initiative

An initiative consists of a collection of one or more policies that allow for simpler policy management. To create an initiative, use the following steps:

  1. Log into the Azure portal.

  2. In the Azure search field, enter Policy.

  3. Select Definitions from the left menu of the page.

  4. Click Initiative Definition at the top of the page.

  5. Choose the management group created in the previous step for Initiative location.

  6. Set the name of the initiative to Send to Observe.

  7. Set the category to Monitoring after selecting on Use existing.

Initiative Basic Settings

Figure 2 - Basic settings for creating initiative

8. Click Next.

9. Select all of the policies for the diagnostic settings for the resource types to monitor.

10. Click Initiative Parameters at the top of the page.

11. Click Create initiative parameter to create a new initiative parameter.

12. Create an initiative parameter called azureRegions. Set the type to array, and add a strong type of location. Add a default value, and then click Save.

Initiative Parameter Regions

Figure 3 - Azure Regions initiative parameter

  1. Create an initiative parameter called eventHubName. Set the type to string, and give it a strong type of Generic - Microsoft.EventHub/Namespaces/EventHubs.

  2. Select yes for Assign Permissions, and then click Save.

Initiative Parameter EventHub Name

Figure 4 - Azure event hub name initiative parameter

3. Create an initiative parameter called eventHubRuleId.
4. Set the type to string, and add a strong type of Generic - Microsoft.EventHub/Namespaces/AuthorizationRules. 5. Select yes for Assign Permissions, and then click Save.

Initiative Parameter EventHub Rule

Figure 5 - Azure event hub authorization rule initiative parameter

  1. Create an initiative parameter called metricsEnabled.

  2. Set the type to string, and then set the allowed value to [“True”, “False”].

  3. Set the default value to True and then click Save.

Initiative Parameter Metrics Enabled

Figure 6 - Azure metrics enabled initiative parameter

  1. Create an initiative parameter called logsEnabled. Set the type to string, and set the allowed value to [“True”, “False”].

  2. Set the default value to True and then click Save.

Initiative Parameter Logs Enabled

Figure 7 - Azure logs enabled initiative parameter

  1. Create an initiative parameter named profileName.

  2. Set the type to string, and then set the default value to SendToObserve. 3. Click Save.

Initiative Parameter Profile Name

Figure 8 - Azure profile name initiative parameter

  1. Navigate to the Policy Parameters section.

  2. Clear the Only show parameters that need input or review checkbox.

  3. Set every Value Type to Use Initiative Parameter and select the corresponding initiative parameter.

Initiative Policy Parameter Settings

Figure 9 - Azure Initiative Policy Parameter Settings

4. Click Review and then Create.

Assigning the Initiative

Assign the initiative to each region you want to monitor using the following steps:

  1. Log into the Azure portal.

  2. In the Azure search field, enter Policy.

  3. Select Assignments.

  4. Click Assign Initiative.

  5. Choose the management group created in the previous step as the Scope.

  6. Select the Send to Observe initiative created in the previous section.

  7. Name the assignment Send to Observe - .

Initiative Assignment Basic Settings

Figure 10 - Initiative Assignment Basic Settings

  1. Click Parameters.

  2. Clear the Only show parameters that need input or review checkbox.

  3. Select the region to deploy for the azureRegions parameter.

  4. Select the eventhub name from the region.

Initiative Assignment Parameters Event Hub name

Figure 11 - Initiative Assignment Parameters Event Hub name

5. Select the Eventhub authorization rule for the Eventhub.

Initiative Assignment Event Hub Authorization Rule

Figure 12 - Initiative Assignment Event Hub Authorization Rule

Initiative Assignment Full Parameter Listing

Figure 13 - Initiative Assignment Full Parameter Listing

6. Click Review + create and then Create.

Once you create the assignment, any new resources of the resource type automatically receive the diagnostic settings to send to Observe created a short period after you create the resource.

Remediation

To remediate non-compliant resources from the Azure portal, perform the following steps:

  1. Log into the Azure portal.

  2. In the Azure search field, enter Policy.

  3. Click Remediation.

  4. Find the policy for the resource type you want to remediate and click the 3 dots.

  5. Click Remediate.

  6. Click Remediate again on the Remediation page.

This creates the diagnostic settings for all of those resources sent to Observe.

FAQ

Troubleshooting Azure

Note

Be sure you install Diagnostic Settings for your integration (metrics and logs) before you begin to troubleshoot your instance. If none are configured, then only resource metadata will show up in Observe which can first take up to 20 minutes.

Q: I’ve installed via Azure Resource Manager or the Observe collection module terraform-observe-collection, but I don’t see any data in Observe. How can I troubleshoot the app?

  • Check to see if your deployment settings are correct, whether through parameters of Azure Resource Manager or through variables in azure.auto.tfvars if using Terraform.

  • It’s common to swap values for objectId and enterpriseAppObjectId, or to introduce spaces or other extra characters in these settings.

  • Check Observe ingest token observe_token is valid in the Observe instance and the corresponding datastream has not been deleted.

  • Enable Azure’s Application Insights and select your Observe Resource Group. Ensure that the application functions perform correctly.

Enabling Application Insights On Your Resource Group

Figure 4 - Enabling Application Insights On Your Resource Group

Select Application Insights Inside Your Resource Group

Figure 5 - Select Application Insights Inside Your Resource Group

Review Observe Application Insights

Figure 6 - Review Observe Application Insights

Note

If you decide to remove the Observe collection module, remove Azure Application Insights from the portal first.

If the functions exist, try restarting the instance for your Observe Function App.

Restart Function App Machine

Figure 7 - Restart Function App Machine

Q: I see logs & metrics data in Observe but I don’t see any resource metadata in Observe. How can I troubleshoot?

  • Navigate to timer_resources_func function in the function app and ensure that successful invocations are taking place as in the below image:

Timer Resources Function
  • If there are failing invocations, navigate to the “Monitor” tab and click on an invocation to view failure details.

Timer Resources Function Error
  • By default, Terraform or ARM will automatically attempt to register any resource providers that it supports. Verify that the following resource providers are at least registered, as these are accessed by the timer_resources_func function. See here for more information.

    • Microsoft.Compute

    • Microsoft.Storage

    • Microsoft.Network

    • Microsoft.Insights

    • Microsoft.Sql

    • Microsoft.Web

    • Microsoft.KeyVault

    • Microsoft.ContainerService

    • Microsoft.Resources

    • Microsoft.EventHub

Q: How do I send Azure collection function logs into Observe?

To send Azure collection function logs into Observe for an Observe engineer to troubleshoot, enable the diagnostic settings on the Observe function app:

  1. Log into the Azure portal.

  2. Search for the Function App in the search field and select the Function App.

Search for Function App

Figure 1 - Search for Function App

3. Select the Observe collection function called observeApp-<OBSERVE-CUSTOMER-ID>-<AZURE-LOCATION>-<LAST-8-CHARS-OF-SUBSCRIPTION>.

4. Search for Diagnostic settings and select it.

Select Diagnostic Settings

Figure 2- Function App- Select Diagnostic Settings

5. Select Add diagnostic setting.

Add Function App Diagnostic Settings

Figure 3 - Function App - Add Diagnostic Settings

6. Select the following options:

  • Diagnostic setting name - observe

  • Function Application Logs

  • AllMetrics

  • Stream to an event hub

7. Select the event hub created by the Observe collection function. This will look like observeeventhub-<OBSERVE-CUSTOMER-ID>-<AZURE-LOCATION>-<LAST-8-CHARS-OF-SUBSCRIPTION>.

8. Click Save.

Q: I’m receiving HTTP status code 429 “Too many requests” errors in Function Logs. What should I do?

The 429 HTTP status is commonly used to reject a request because a call rate limit is reached. The client will auto retry requests but may raise an exception after repeated retries. This usually happens in the timer_resources_func function by Microsoft.Storage client’s API calls, which has stricter rate limits for management operations.

Rate limiting may happen when there are large numbers of blob services and/or blob containers in a storage account. It is recommended to disable their collection by adding a feature flag as part of the function app environment variables, under App Settings.

FEATURE_FLAGS=!blob_services,!blob_containers
Disable blob Collection

Figure 4 - Function App - Add additional environment variables

If managing via Terraform, this can be set by adding the following app_settings variable to your azure.auto.tfvars file

app_settings = {
  FEATURE_FLAGS = "!blob_services,!blob_containers"
}

If managing via ARM, this can be set by adding the feature_flags parameter to the ARM deployment command as below:

az deployment sub create \
      --name observe \
      --location <THE_AZURE_LOCATION_TO_MONITOR> \
      --template-file resourcemanager-latest.json \
      --parameters \
        observe_customer=<YOUR_CUSTOMER_ID> \
        observe_token=<YOUR_DATASTREAM_TOKEN> \
        objectId=31a56... \
        applicationId=58339... \
        clientSecretValue=K3j8... \
        enterpriseAppObjectId=3aca8... \
        feature_flags='!blob_services,!blob_containers'

Note

Before enabling feature flags, please ensure you’re on the latest version of function app.

Q: Why don’t some of the messages in Function Logs link back to the Python Function Resource?

Any logging performed by functions in Python should use the logging module and not the print() function. Using the print() function generates separate events for each line and does not include the required metadata to tie it back to a function, or, in the case of multi-line stack traces, the other lines of the stack trace.