Microsoft Azure

The Observe Azure app allows you to collect data from your Azure resources within your subscription. Based on the region, the Azure app simultaneously ingests logs and metrics from standard Azure services.

The Observe Azure app works with the datasets within the application. Observe can automate many everyday data modeling tasks, ensuring an accurate picture of your infrastructure. Observe can also update your workspace with new datasets when new functionality releases for this integration.

If you currently utilize Azure data, consult with Observe to see if the Azure Integration could enhance your existing data collection strategy.

What data does Observe ingest?

Observe catalogs resources deployed at a given location (region) and collects logs and metrics from those resources.

Figure 1 - Overview of Observe Azure data collection using the Observe Function App

The Observe Function App contains three functions responsible for data collection:

• event_hub_telemetry_func - captures the data sent to the Event Hub from the Azure resources Diagnostic settings. Event Hub triggers this function and forwards the data to Observe.

• timer_resources_func - returns all resources within the location (region) and the corresponding metadata. It runs on an assigned NCRONTAB schedule and set to every 10 minutes by default.

• timer_vm_metrics_func - returns virtual machine metrics from the hypervisor. It runs on an assigned NCRONTAB schedule and set to every 5 minutes by default.

Standard Ingest Sources

The Observe Azure app ingests the following types of data from a single location (region):

• Active Directory

• Azure Kubernetes Service (AKS)

• Functions

• SQL Database

• SQL Managed Instance

• Storage Account

• Virtual Machines

Note

Microsoft Azure does not provide operating system (OS) metrics for virtual machines. Consider installing the Observe Host Monitoring app.

Using Azure Integration Data

After shaping, the incoming data populates datasets such as the following:

• AD

  • Active Directory - Active Directory Logs which includes Audit Logs, Sign-In logs and Provisioning Logs

• AKS

  • AKS Clusters - AKS Cluster Logs and Metrics

• App Services

  • App Services - Overall App Service performance, usage, and tracing

• Cognitive Services

  • Cognitive Service - Usage and Performance of Cognitive Services, such as OpenAI services

• Functions

  • Function App - Overall FunctionApp performance and usage

  • Functions - Function performance

• SQL Database

  • SQL Server - Servers SQL Databases employ

  • SQL Database - Logs audit, deadlocks, error, query, timeout activity, and metrics such as query store runtime and wait statistics.

• SQL Managed Instances

  • SQL Managed Instance - Status and basic metrics of instances

  • SQL Managed Instance Databases - Logs of errors and metrics, including query store runtime and wait statistics for databases running within SQL Managed Instances

• Storage Account

  • Storage Account - Basic info about storage account, logs about error and performance related to a Storage Account, blob and containers.

  • Blob Service - Overall info about blob service, performance metrics and associated storage account.

  • Blob Container - Overall info about blob container.

• Virtual Machines

  • Virtual Machine - Status and basic metrics of your virtual machines

  • Virtual Machine Disk - Status and usage of allocated disks for your virtual machines

Installation

Observe Azure Collection Setup

Note

This must be deployed to each region in each subscription.

Note

In order for the collection to work, you must ensure that microsoft.insights is registered as a resource provider for your subscription. Microsoft Documentation provides guidance on how to register resource providers using the Azure Portal, Azure CLI, and Azure PowerShell.

When you create an Azure data collection using Terraform or Azure Resource Manager, you need to specify an Azure location. Find the location of your Azure services by looking at the Resource groups page or each Azure service page, such as Azure Virtual Machines. Use the following Azure CLI command to find the right naming convention for your Azure services:

$ az account list-locations --output table

DisplayName        Name             Regional Display Name 
------------       ----             ---------------------
East US		      eastus            (US) East US
East US 2         eastus2          (US) East US 2
South Central US  southcentralus   (US) South Central US

For example, if you have a Virtual Machine (VM) located in the East US, use location = "eastus" in the Terraform configuration, or --location eastus in Azure Resource Manager.

Azure App

You can install the Azure app on the Apps page to install the Azure integration.

1. Click Configuration and select the Azure integrations to install using the app.

2. Select Connections.

3. Click Create connection. This creates an ingest token to use with the data stream.

4. Follow the instructions to create the ingest token.

Figure 2 - List of available Azure integrations

Terraform

Before you begin, you should, at a minimum, have a datastream available to send Azure data, including an ingest token to be employed by the Data Collection Module terraform-azure-collection.

1. Install the Azure CLI and Azure’s Functions Core Tools

2. Be sure the Azure CLI installs correctly by logging into Azure.

  az login

3. You should receive a token from your browser that looks like this:

    [
      {
        "cloudName": "AzureCloud",
        "homeTenantId": "########-####-####-####-############",
        "id": "########-####-####-####-############",
        "isDefault": true,
        "managedByTenants": [],
        "name": "Acme Inc",
        "state": "Enabled",
        "tenantId": "########-####-####-####-############",
        "user": {
          "name": "[email protected]",
          "type": "user"
        }
      }
    ]

4. If you have multiple subscription / tenant, then use below command to set correct subscription/tenant where you would like to deploy Observe Azure collection stack -

  az account set --subscription "My Demos"

OR

  az login --tenant <tenant>

For details on Azure subscription/tenant follow here.

5. Clone the Observe Terraform Collection Module repository locally.

    git clone [email protected]:observeinc/terraform-azure-collection.git

6. Assign the Application Variables.

Inside the root of the terraform-azure-collection, create a file called azure.auto.tfvars. The file contains the following content:

    observe_customer = "<OBSERVE_CUSTOMER_ID>"
    observe_token = "<DATASTREAM_INGEST_TOKEN>"
    observe_domain = "<OBSERVE_DOMAIN(i.e. observe-staging.com)>"
    timer_resources_func_schedule = "<TIMER_TRIGGER_FUNCTION_SCHEDULE>" 
    timer_vm_metrics_func_schedule = "<TIMER_TRIGGER_FUNCTION_SCHEDULE>"
    location = "<AZURE_REGIONAL_NAME>"

Note

The Azure app assigns these default values:

• timer_resources_func_schedule is the schedule used to collect data from your Azure resources.

• timer_vm_metrics_func_schedule - is the schedule used to collect metrics on your Virtual Machines.

Both values based on NCRONTAB.

The location value sets the Azure Regional Name to eastus by default.

7. Deploy the Azure collection module.

Inside the root directory of the terraform-azure-collection module, run the following commands:

      terraform init
      terraform apply -auto-approve

Data collection should begin shortly.

Note

To capture logs and metrics of most resources, you must add diagnostic settings. Resource metadata collection is done automatically although data can take up to 20 minutes to show up in Observe. Check the Microsoft Azure function logs to ensure desired results.

8. Collecting data from multiple locations - To collect data from multiple locations, re-deploy the Observe Azure Collection with the updated location you want to collect data.

Azure Resource Manager

1. Create an App Registration by searching for App Registrations in the Search bar, and then clicking New registration. Create for the Registration.

   

   Figure 1 - App Registration Overview after Creation

2. Create a Client Secret by clicking “Add a certificate or secret”.

Figure 2 - App Registration Client Secret

3. Record the following 3 values:

• Secret Value

• Application ID

• Object ID

4. Search for Enterprise Applications in the top search bar.

Figure 3 - Enterprise App Overview Page

5. Record the Enterprise App Object ID.

6. Initialize Azure Cloud Shell or another shell environment.

  az login

7. If you have multiple subscription / tenant, then use below command to set correct subscription/tenant where you would like to deploy Observe Azure collection stack -

  az account set --subscription "My Demos"

OR

  az login --tenant <tenant>

For details on Azure subscription/tenant follow here.

8. Download the Observe Azure Resource Manager template.

Using Bash

  curl -Lo resourcemanager-latest.json https://observeinc.s3.us-west-2.amazonaws.com/azure/resourcemanager-latest.json

Using PowerShell

  Invoke-WebRequest -Uri https://observeinc.s3.us-west-2.amazonaws.com/azure/resourcemanager-latest.json -OutFile resourcemanager-latest.json

9. Enter the following text in the shell. Run the command to create the resource.

    az deployment sub create \
      --name observe \
      --location <THE_AZURE_LOCATION_TO_MONITOR> \
      --template-file resourcemanager-latest.json \
      --parameters \
        observe_customer=<YOUR_CUSTOMER_ID> \
        observe_token=<YOUR_DATASTREAM_TOKEN> \
        objectId=31a56... \
        applicationId=58339... \
        clientSecretValue=K3j8... \
        enterpriseAppObjectId=3aca8...

10. After a few minutes, the command generates the subscription.

Figure 4 - Successful Deployment

Data collection should begin shortly.

Note

To capture logs and metrics of most resources, you must add diagnostic settings as shown below. Resource metadata collection is done automatically although data can take up to 20 minutes to show up in Observe.

11. Collecting data from multiple locations To collect data from multiple locations, re-deploy the Observe Azure Collection with the updated location you want to collect data.

Azure Resource Configuration

To capture logs and metrics of most resources, you must add diagnostic settings. Expand the desired Azure service name for instructions on adding diagnostic settings.

Active Directory

1. Log into the Azure portal.

2. Search for the Active Directory in the search field.

Figure 1 - Search for Active Directory

3. Go to Monitoring and Diagnostic settings. Select Diagnostic setting.

Figure 2 - Active Directory - Add Diagnostic Settings

4. Select the following options:

• Diagnostic setting name - Observe

• AuditLogs

• SignInLogs

• NonInteractiveUserSignInLogs

• ServicePrincipalSignInLogs

• ManagedIdentitySignInLogs

• ProvisioningLogs

• Stream to an event hub

5. Select the event hub created by the Observe collection function.

6. Click Save.

AKS

Note

For comprehensive Kubernetes monitoring, install the Observe Kubernetes App

1. Log into the Azure portal.

2. In the Azure search field, enter Kubernetes Services.

Figure 1 - Search for Kubernetes Services

3. Choose the service you want to monitor.

4. From the menu, select Diagnostic Settings.

Figure 2 - Kubernetes Services - Select Diagnostic Settings

5. Click Add diagnostic setting.

Figure 3 - Kubernetes Services - Add Diagnostic Setting

6. Create a name for your setting. Select all of the metrics listed, then click Stream to an Event hub. Be sure to select the Event hub namespace and Event hub name with your Observe customer ID.

Figure 4 - Kubernetes Services - Diagnostic Setting

7. Click Save.

App Services

1. Log into the Azure portal.

2. Search for the App Services in the search field and select the App Services.

Figure 1 - Search for App Services

3. Select the App Service you want to monitor.

Figure 2 - App Service - Select App Service

4. Search for Diagnostic settings and select it.

Figure 3 - App Service- Select Diagnostic Settings

5. Select Add diagnostic setting.

Figure 4 - App Services - Add Diagnostic Settings

6. Select the following options:

• Diagnostic setting name - Observe

• HTTP logs

• App Service Console Logs

• App Service Application Logs

• Access Audit Logs

• IPSecurity Audit logs

• App Service Platform logs

• All Metrics

• Stream to an event hub

7. Select the event hub created by the Observe collection function.

8. Click Save.

If you have Application Insights enabled and want to get detailed info for your App Service:

1. Search for Application Insights and select it.

Figure 5 - App Services- Search Application Insights

2. Click View Application Insights data.

3. Search for Diagnostic settings and select it.

Figure 6 - App Services - Search Diagnostic Settings

4. Select Add diagnostic setting.

5. Select the following options:

• Diagnostic setting name - Observe

• All logs

• All Metrics

• Stream to an event hub

6. Select the event hub created by the Observe collection function.

7. Click Save.

Figure 7 - App Service - Diagnostic Setting

Cognitive Services

1. Log into the Azure portal.

2. Search for the Cognitive Services in the search field and select it.

Figure 1 - Search for Cognitive Services

3. Select the Cognitive Service Type you want to monitor.

Figure 2 - Cognitive Services - Select Service Type

4. Select the Cognitive Service you want to monitor.

Figure 3 - Cognitive Services - Select Service

5. Search for Diagnostic settings and select it.

Figure 4 - Cognitive Services - Select Diagnostic Settings

6. Select Add diagnostic setting.

Figure 5 - Cognitive Services - Add Diagnostic Settings

7. Select the following options:

• Diagnostic setting name - Observe

• All Logs

• All Metrics

• Stream to an event hub

8. Select the event hub created by the Observe collection function.

9. Click Save.

Figure 6 - Cognitive Services - Diagnostic Setting

Functions

1. Log into the Azure portal.

2. Search for the Function App in the search field and select the Function App.

Figure 1 - Search for Function App

3. Select the Function App you want to monitor.

Figure 2 - Function App - Select Function App

4. Search for Diagnostic settings and select it.

Figure 3 - Function App - Select Diagnostic Settings

5. Select Add diagnostic setting.

Figure 4 - Function App - Add Diagnostic Settings

6. Select the following options:

• Diagnostic setting name - Observe

• Function Application Logs

• All Metrics

• Stream to an event hub

7. Select the event hub created by the Observe collection function.

8. Click Save.

Figure 5 - Function App - Diagnostic Setting

SQL Database

1. Log into the Azure portal.

2. In the Azure search field, enter SQL Database.

3. Select your database.

Figure 1 - SQL Database - Select SQL Database

4. From the left menu, choose Diagnostic Settings and click Add diagnostic setting.

Figure 2- SQL Database - Add Diagnostic Setting

5. Choose all Logs, Audit Logs, and all Metrics. Send to the event hub with your customer ID.

Figure 3 - SQL Database - Diagnostic Setting

6. Create a name for Diagnostic Setting and click Save.

SQL Managed Instances

1. Log into the Azure portal.

2. In the Azure search field, enter SQL Managed Instances.

3. Select your SQL Managed Instance.

Figure 1 - SQL Managed Instances - Select SQL Managed Instance

4. From the left menu, choose Diagnostic Settings and click Add diagnostic setting.

Figure 2 - SQL Managed Instances - Add Diagnostic Setting

5. Select all Logs, Audit logs, and all Metrics. Send to event hub with your customer-id.

Figure 3 - SQL Managed Instances - Diagnostic Setting

6. Create a name for the diagnostic setting and click Save.

Storage Account

1. Log into the Azure portal.

2. Search for the Storage Accounts in the search field.

Figure 1 - Search for Storage Account

3. Go to the Monitoring section and select Diagnostic setting. This enables logs and metrics at the Storage Account level.

Figure 2 - Storage Account - Diagnostic Settings

4. Select Add Diagnostics Settings

Figure 2 - Storage Account - Add Diagnostic Settings

5. Select Metrics -> Transaction and select the event hub created by the Observe collection function.

6. Click Save.

Blob

1. Go to the Monitoring and Diagnostic settings. Select Diagnostic setting. Select blob under the Storage Account.

Figure 2 -Blob - Diagnostic Settings

2. Select Add Diagnostics Settings

3. Select Metrics -> Transaction. Select Logs -> Categories -> StorageRead, StorageWrite, StorageDelete.

4. Select the event hub created by the Observe collection function.

5. Click Save.

Virtual Machines

Currently, the Azure app collects this data using the timer_resources_func and timer_vm_metrics_func functions within the Observe Function app deployed. It does not require any diagnostic settings.

Creating Diagnostic Settings at Scale

Creating Azure Policies

If you want to create Diagnostic Settings for all resources of a resource type, you can configure an Azure Policy. Create custom policy definitions for each type of resource to monitor. Microsoft provides documentation on how to accomplish this.

Once you create the policies in your Azure account, Observe recommends creating a policy initiative, assigning the relevant policies, assigning that policy to a Management Group, and then adding your subscriptions to that management group.

Note

Complete these steps before moving on to the next section.

Creating a Management Group

1. Log into the Azure portal.

2. In the Azure search field, enter Management Groups.

3. Click Create.

4. Give the policy an ID such as observediagnosticsettings.

5. Set the policy display name such as Observe Diagnostic Settings.

Figure 1 - Settings for creating a management group

6. Click Submit.

Populating the Management Group

Once you create the management group, you need to move any other management groups or subscriptions to apply under the Observe Diagnostic Settings management group. Use the following steps:

1. Log into the Azure portal.

2. In the Azure search field, enter Management Groups.

3. Click on the three dots on the right of the management group or subscription.

4. Click Move.

5. Select the Observe Diagnostic Settings management group as the destination.

6. Click Save.

Creating an Initiative

An initiative consists of a collection of one or more policies that allow for simpler policy management. To create an initiative, use the following steps:

1. Log into the Azure portal.

2. In the Azure search field, enter Policy.

3. Select Definitions from the left menu of the page.

4. Click Initiative Definition at the top of the page.

5. Choose the management group created in the previous step for Initiative location.

6. Set the name of the initiative to Send to Observe.

7. Set the category to Monitoring after selecting on Use existing.

Figure 2 - Basic settings for creating initiative

8. Click Next.

9. Select all of the policies for the diagnostic settings for the resource types to monitor.

10. Click Initiative Parameters at the top of the page.

11. Click Create initiative parameter to create a new initiative parameter.

12. Create an initiative parameter called azureRegions. Set the type to array, and add a strong type of location. Add a default value, and then click Save.

Figure 3 - Azure Regions initiative parameter

1. Create an initiative parameter called eventHubName. Set the type to string, and give it a strong type of Generic - Microsoft.EventHub/Namespaces/EventHubs.

2. Select yes for Assign Permissions, and then click Save.

Figure 4 - Azure event hub name initiative parameter

3. Create an initiative parameter called eventHubRuleId.
4. Set the type to string, and add a strong type of Generic - Microsoft.EventHub/Namespaces/AuthorizationRules. 5. Select yes for Assign Permissions, and then click Save.

Figure 5 - Azure event hub authorization rule initiative parameter

1. Create an initiative parameter called metricsEnabled.

2. Set the type to string, and then set the allowed value to [“True”, “False”].

3. Set the default value to True and then click Save.

Figure 6 - Azure metrics enabled initiative parameter

1. Create an initiative parameter called logsEnabled. Set the type to string, and set the allowed value to [“True”, “False”].

2. Set the default value to True and then click Save.

Figure 7 - Azure logs enabled initiative parameter

1. Create an initiative parameter named profileName.

2. Set the type to string, and then set the default value to SendToObserve. 3. Click Save.

Figure 8 - Azure profile name initiative parameter

1. Navigate to the Policy Parameters section.

2. Clear the Only show parameters that need input or review checkbox.

3. Set every Value Type to Use Initiative Parameter and select the corresponding initiative parameter.

Figure 9 - Azure Initiative Policy Parameter Settings

4. Click Review and then Create.

Assigning the Initiative

Assign the initiative to each region you want to monitor using the following steps:

1. Log into the Azure portal.

2. In the Azure search field, enter Policy.

3. Select Assignments.

4. Click Assign Initiative.

5. Choose the management group created in the previous step as the Scope.

6. Select the Send to Observe initiative created in the previous section.

7. Name the assignment Send to Observe - .

Figure 10 - Initiative Assignment Basic Settings

1. Click Parameters.

2. Clear the Only show parameters that need input or review checkbox.

3. Select the region to deploy for the azureRegions parameter.

4. Select the eventhub name from the region.

Figure 11 - Initiative Assignment Parameters Event Hub name

5. Select the Eventhub authorization rule for the Eventhub.

Figure 12 - Initiative Assignment Event Hub Authorization Rule

Figure 13 - Initiative Assignment Full Parameter Listing

6. Click Review + create and then Create.

Once you create the assignment, any new resources of the resource type automatically receive the diagnostic settings to send to Observe created a short period after you create the resource.

Remediation

To remediate non-compliant resources from the Azure portal, perform the following steps:

1. Log into the Azure portal.

2. In the Azure search field, enter Policy.

3. Click Remediation.

4. Find the policy for the resource type you want to remediate and click the 3 dots.

5. Click Remediate.

6. Click Remediate again on the Remediation page.

This creates the diagnostic settings for all of those resources sent to Observe.

FAQ

Troubleshooting Azure

Note

Be sure you install Diagnostic Settings for your integration (metrics and logs) before you begin to troubleshoot your instance. If none are configured, then only resource metadata will show up in Observe which can first take up to 20 minutes.

Q: I’ve installed via Azure Resource Manager or the Observe collection module terraform-observe-collection, but I don’t see any data in Observe. How can I troubleshoot the app?

• Check to see if your deployment settings are correct, whether through parameters of Azure Resource Manager or through variables in azure.auto.tfvars if using Terraform.

• It’s common to swap values for objectId and enterpriseAppObjectId, or to introduce spaces or other extra characters in these settings.

• Check Observe ingest token observe_token is valid in the Observe instance and the corresponding datastream has not been deleted.

• Enable Azure’s Application Insights and select your Observe Resource Group. Ensure that the application functions perform correctly.

Figure 4 - Enabling Application Insights On Your Resource Group

Figure 5 - Select Application Insights Inside Your Resource Group

Figure 6 - Review Observe Application Insights

Note

If you decide to remove the Observe collection module, remove Azure Application Insights from the portal first.

If the functions exist, try restarting the instance for your Observe Function App.

Figure 7 - Restart Function App Machine

Q: I see logs & metrics data in Observe but I don’t see any resource metadata in Observe. How can I troubleshoot?

• Navigate to timer_resources_func function in the function app and ensure that successful invocations are taking place as in the below image:

• If there are failing invocations, navigate to the “Monitor” tab and click on an invocation to view failure details.

• By default, Terraform or ARM will attempt to register any resource providers that it supports. Verify that the following resource providers are at least registered, as these are accessed by the timer_resources_func function. See here for more information.

  • Microsoft.Compute

  • Microsoft.Storage

  • Microsoft.Network

  • Microsoft.Insights

  • Microsoft.Sql

  • Microsoft.Web

  • Microsoft.KeyVault

  • Microsoft.ContainerService

  • Microsoft.Resources

Q: How do I send Azure collection function logs into Observe?

To send Azure collection function logs into Observe for an Observe engineer to troubleshoot, enable the diagnostic settings on the Observe function app:

1. Log into the Azure portal.

2. Search for the Function App in the search field and select the Function App.

Figure 1 - Search for Function App

3. Select the Observe collection function called observeApp-<OBSERVE-CUSTOMER-ID>-<AZURE-LOCATION>-<LAST-8-CHARS-OF-SUBSCRIPTION>.

4. Search for Diagnostic settings and select it.

Figure 2- Function App- Select Diagnostic Settings

5. Select Add diagnostic setting.

Figure 3 - Function App - Add Diagnostic Settings

6. Select the following options:

• Diagnostic setting name - observe

• Function Application Logs

• AllMetrics

• Stream to an event hub

7. Select the event hub created by the Observe collection function. This will look like observeeventhub-<OBSERVE-CUSTOMER-ID>-<AZURE-LOCATION>-<LAST-8-CHARS-OF-SUBSCRIPTION>.

8. Click Save.

Q: Why don’t some of the messages in Function Logs link back to the Python Function Resource?

Any logging performed by functions in Python should use the logging module and not the print() function. Using the print() function generates separate events for each line and does not include the required metadata to tie it back to a function, or, in the case of multi-line stack traces, the other lines of the stack trace.