documentationMicrosoft Azure¶
The Observe Azure App allows you to collect data from your Azure resources within your subscription. Based on the region, the Azure app simultaneously ingests logs and metrics from standard Azure services.
The Observe Azure App works with the datasets within the application. Observe can automate many everyday data modeling tasks, ensuring an accurate picture of your infrastructure. Observe can also update your workspace with new datasets when new functionality releases for this integration.
If you currently utilize Azure data, consult with Observe to see if the Azure Integration could enhance your existing data collection strategy.
What data does Observe ingest?¶
Observe catalogs resources deployed at a given location (region) and collects logs and metrics from those resources.
Figure 1 - Overview of Observe Azure data collection using the Observe Function App
The Observe Function App contains three functions responsible for data collection:
event_hub_telemetry_func- captures the data sent to the Event Hub from the Azure resources Diagnostic settings. Event Hub triggers this function and forwards the data to Observe.timer_resources_func- returns all resources within the location (region) and the corresponding metadata. It runs on an assigned NCRONTAB schedule and set to every 10 minutes by default.timer_vm_metrics_func- returns virtual machine metrics from the hypervisor. It runs on an assigned NCRONTAB schedule and set to every 5 minutes by default.
Standard Ingest Sources¶
The Observe Azure app automatically ingests the following types of data from a single location (region):
Azure Kubernetes Service (AKS) basic performance of your AKS Clusters
Note
Azure does not provide operating system (OS) metrics for virtual machines. Consider installing the Observe Host Monitoring app.
Using Azure Integration Data¶
After shaping, the incoming data populates datasets such as the following:
AKS
AKS Clusters - AKS Cluster Logs and Metrics
App Services
App Services - Overall App Service performance, usage, and tracing
Functions
Function App - Overall FunctionApp performance and usage
Functions - Function performance
SQL Database
SQL Server - Servers SQL Databases employ
SQL Database - Logs audit, deadlocks, error, query, timeout activity, and metrics such as query store runtime and wait statistics.
SQL Managed Instances
SQL Managed Instance - Status and basic metrics of instances
SQL Managed Instance Databases - Logs of errors and metrics, including query store runtime and wait statistics for databases running within SQL Managed Instances
Virtual Machines
Virtual Machine - Status and basic metrics of your virtual machines
Virtual Machine Disk - Status and usage of allocated disks for your virtual machines
Installation¶
Observe Azure Collection Setup¶
Note
In order for the collection to work, you must ensure that microsoft.insights is registered as a resource provider for your subscription. Microsoft Documentation provides guidance on how to register resource providers using the Azure Portal, Azure CLI, and Azure PowerShell.
When you create an Azure data collection using Terraform or Azure Resource Manager, you need to specify an Azure location. Find the location of your Azure services by looking at the Resource groups page or each Azure service page, such as Azure Virtual Machines. Use the following Azure CLI command to find the right naming convention for your Azure services:
$ az account list-locations --output table
DisplayName Name Regional Display Name
------------ ---- ---------------------
East US eastus (US) East US
East US 2 eastus2 (US) East US 2
South Central US southcentralus (US) South Central US
For example, if you have a Virtual Machine (VM) located in the East US, use location = "eastus" in the Terraform configuration, or --location eastus in Azure Resource Manager.
You can install the Azure app on the Apps page to install the Azure integration.
Click Configuration and select the Azure integrations you want to install using the app.
Select Connections.
Click Create connection. This creates an ingest token to use with the data stream.
Follow the instructions to create the ingest token.
Figure 2 - List of available Azure integrations
Before you begin, you should, at a minimum, have a datastream available to send Azure data, including an ingest token to be employed by the Data Collection Module terraform-azure-collection.
Install the Azure CLI.
Be sure the Azure CLI installs correctly by logging into Azure.
az login
3. You should receive a token from your browser that looks like this:
[
{
"cloudName": "AzureCloud",
"homeTenantId": "########-####-####-####-############",
"id": "########-####-####-####-############",
"isDefault": true,
"managedByTenants": [],
"name": "Acme Inc",
"state": "Enabled",
"tenantId": "########-####-####-####-############",
"user": {
"name": "[email protected]",
"type": "user"
}
}
]
4. Clone the Observe Terraform Collection Module repository locally.
git clone [email protected]:observeinc/terraform-azure-collection.git
5. Assign the Application Variables.
Inside the root of the terraform-azure-collection, create a file called azure.auto.tfvars. The file contains the following content:
observe_customer = "<OBSERVE_CUSTOMER_ID>"
observe_token = "<DATASTREAM_INGEST_TOKEN>"
observe_domain = "<OBSERVE_DOMAIN(i.e. observe-staging.com)>"
timer_resources_func_schedule = "<TIMER_TRIGGER_FUNCTION_SCHEDULE>"
timer_vm_metrics_func_schedule = "<TIMER_TRIGGER_FUNCTION_SCHEDULE>"
location = "<AZURE_REGIONAL_NAME>"
Note
The Azure app assigns these default values:
timer_resources_func_scheduleis the schedule used to collect data from your Azure resources.timer_vm_metrics_func_schedule- is the scheduled used to collect metrics on your Virtual Machines.
Both values based on NCRONTAB.
The location value sets the Azure Regional Name to eastus by default.
6. Deploy the Azure collection module.
Inside the root directory of the terraform-azure-collection module, run the following commands:
terraform init
terraform apply -auto-approve
Data collection should begin shortly.
Create an App Registration by searching for App Registrations in the Search bar, and then clicking New registration. Create for the Registration.
Figure 1 - App Registration Overview after Creation
2. Create a Client Secret by clicking “Add a certificate or secret”.
Figure 2 - App Registration Client Secret
3. Record the following 3 values:
Secret Value
Application ID
Object ID
4. Search for Enterprise Applications in the top search bar.
Figure 3 - Enterprise App Overview Page
5. Record the Enterprise App Object ID.
6. Initialize Azure Cloud Shell or another shell environment.
az login
7. Download the Observe Azure Resource Manager template.
Using Bash
curl -Lo resourcemanager-latest.json https://observeinc.s3.us-west-2.amazonaws.com/azure/resourcemanager-latest.json
Using PowerShell
Invoke-WebRequest -Uri https://observeinc.s3.us-west-2.amazonaws.com/azure/resourcemanager-latest.json -OutFile resourcemanager-latest.json
8. Enter the following text in the shell. Run the command to create the resource.
az deployment sub create \
--name observe \
--location <THE_AZURE_LOCATION_TO_MONITOR> \
--template-file resourcemanager-latest.json \
--parameters \
observe_customer=<YOUR_CUSTOMER_ID> \
observe_token=<YOUR_DATASTREAM_TOKEN> \
objectId=31a56... \
applicationId=58339... \
clientSecretValue=K3j8... \
enterpriseAppObjectId=3aca8...
9. After a few minutes, the command generates the subscription.
Figure 4 - Successful Deployment
Data collection should begin shortly.
Azure Resource Configuration¶
To capture logs and metrics of most resources, you must add diagnostic settings. Expand the desired Azure service name for instructions on adding diagnostic settings.
AKS
Note
For comprehensive Kubernetes monitoring, install the Observe Kubernetes App
Log into the Azure portal.
In the Azure search field, enter Kubernetes Services.
Figure 1 - Search for Kubernetes Services
3. Choose the service you want to monitor.
4. From the menu, select Diagnostic Settings.
Figure 2 - Kubernetes Services - Select Diagnostic Settings
5. Click Add diagnostic setting.
Figure 3 - Kubernetes Services - Add Diagnostic Setting
6. Create a name for your setting. Select all of the metrics listed, then click Stream to an Event hub. Be sure to select the Event hub namespace and Event hub name with your Observe customer ID.
Figure 4 - Kubernetes Services - Diagnostic Setting
7. Click Save.
App Services
Log into the Azure portal.
Search for the App Services in the search field and select the App Services.
Figure 11 - Search for App Services
3. Select the App Service you want to monitor.
Figure 12 - App Service - Select App Service
4. Search for Diagnostic settings and select it.
Figure 13 - App Service- Select Diagnostic Settings
5. Select Add diagnostic setting.
Figure 14 - App Services - Add Diagnostic Settings
6. Select the following options:
Diagnostic setting name - Observe
HTTP logs
App Service Console Logs
App Service Application Logs
Access Audit Logs
IPSecurity Audit logs
App Service Platform logs
All Metrics
Stream to an event hub
7. Select the event hub created by the Observe collection function.
8. Click Save.
If you have Application Insights enabled and would like get detailed info for your App Service:
1. Search for Application Insights and select it.
Figure 13 - App Services- Search Application Insights
2. Click View Application Insights data.
3. Search for Diagnostic settings and select it.
Figure 13 - App Services - Search Diagnostic Settings
4. Select Add diagnostic setting.
5. Select the following options:
Diagnostic setting name - Observe
All logs
All Metrics
Stream to an event hub
6. Select the event hub created by the Observe collection function.
7. Click Save.
Figure 15 - App Service - Diagnostic Setting
Functions
Log into the Azure portal.
Search for the Function App in the search field and select the Function App.
Figure 1 - Search for Function App
3. Select the Function App you want to monitor.
Figure 2 - Function App - Select Function App
4. Search for Diagnostic settings and select it.
Figure 3 - Function App - Select Diagnostic Settings
5. Select Add diagnostic setting.
Figure 4 - Function App - Add Diagnostic Settings
6. Select the following options:
Diagnostic setting name - Observe
Function Application Logs
All Metrics
Stream to an event hub
7. Select the event hub created by the Observe collection function.
8. Click Save.
Figure 5 - Function App - Diagnostic Setting
SQL Database
Log into the Azure portal.
In the Azure search field, enter SQL Database.
Select your database.
Figure 1 - SQL Database - Select SQL Database
4. From the left menu, choose Diagnostic Settings and click Add diagnostic setting.
Figure 2- SQL Database - Add Diagnostic Setting
5. Choose all Logs, Audit Logs, and all Metrics. Send to the event hub with your customer ID.
Figure 3 - SQL Database - Diagnostic Setting
6. Create a name for Diagnostic Setting and click Save.
SQL Managed Instances
Log into the Azure portal.
In the Azure search field, enter SQL Managed Instances.
Select your SQL Managed Instance.
Figure 1 - SQL Managed Instances - Select SQL Managed Instance
4. From the left menu, choose Diagnostic Settings and click Add diagnostic setting.
Figure 2 - SQL Managed Instances - Add Diagnostic Setting
5. Select all Logs, Audit logs, and all Metrics. Send to event hub with your customer-id.
Figure 3 - SQL Managed Instances - Diagnostic Setting
6. Create a name for the diagnostic setting and click Save.
Virtual Machines
Currently, the Azure app collects this data using the timer_resources_func and timer_vm_metrics_func functions within the Observe Function app deployed. It does not require any diagnostic settings.
Creating Diagnostic Settings at Scale
Creating Azure Policies¶
If you want to create Diagnostic Settings for all resources of a resource type, you can configure an Azure Policy. Create custom policy definitions for each type of resource to monitor. Microsoft provides documentation on how to accomplish this.
Once you create the policies in your Azure account, Observe recommends creating a policy initiative, assigning the relevant policies, assigning that policy to a Management Group, and then adding your subscriptions to that management group
NOTE: This steps needs to completed before moving on to the following steps.
Creating a Management Group¶
Log into the Azure portal.
In the Azure search field, enter Management Groups.
Click Create.
Give the policy an ID such as observediagnosticsettings.
Set the policy display name such as Observe Diagnostic Settings.
Figure 1 - Settings for creating a management group
6. Click Submit.
Populating the Management Group¶
Once you create the management group, you need to move any other management groups or subscriptions to apply under the Observe Diagnostic Settings management group. Use the following steps:
Log into the Azure portal.
In the Azure search field, enter Management Groups.
Click on the three dots on the right of the management group or subscription.
Click Move.
Select the Observe Diagnostic Settings management group as the destination.
Click Save.
Creating an Initiative¶
An initiative consists of a collection of one or more policies that allow for simpler policy management. To create an initiative, use the following steps:
Log into the Azure portal.
In the Azure search field, enter Policy.
Select Definitions from the left menu of the page.
Click Initiative Definition at the top of the page.
Choose the management group created in the previous step for Initiative location.
Set the name of the initiative to Send to Observe.
Set the category to Monitoring after selecting on Use existing.
Figure 2 - Basic settings for creating initiative
8. Click Next.
9. Select all of the policies for the diagnostic settings for the resource types to monitor.
10. Click Initiative Parameters at the top of the page.
11. Click Create initiative parameter to create a new initiative parameter.
12. Create an initiative parameter called azureRegions. Set the type to array, and add a strong type of location. Add a default value, and then click Save.
Figure 3 - Azure Regions initiative parameter
Create an initiative parameter called eventHubName. Set the type to string, and give it a strong type of Generic - Microsoft.EventHub/Namespaces/EventHubs.
Select yes for Assign Permissions, and then click Save.
Figure 4 - Azure event hub name initiative parameter
3. Create an initiative parameter called eventHubRuleId.
4. Set the type to string, and add a strong type of Generic - Microsoft.EventHub/Namespaces/AuthorizationRules.
5. Select yes for Assign Permissions, and then click Save.
Figure 5 - Azure event hub authorization rule initiative parameter
Create an initiative parameter called metricsEnabled.
Set the type to string, and then set the allowed value to [“True”, “False”].
Set the default value to True and then click Save.
Figure 6 - Azure metrics enabled initiative parameter
Create an initiative parameter called logsEnabled. Set the type to string, and set the allowed value to [“True”, “False”].
Set the default value to True and then click Save.
Figure 7 - Azure logs enabled initiative parameter
Create an initiative parameter called profileName.
Set the type to string, and then set the default value to SendToObserve. 3. Click Save.
Figure 8 - Azure profile name initiative parameter
Navigate to the Policy Parameters section.
Clear the Only show parameters that need input or review checkbox.
Set every Value Type to Use Initiative Parameter and select the corresponding initiative parameter.
Figure 9 - Azure Initiative Policy Parameter Settings
4. Click Review and then Create.
Assigning the Initiative¶
Assign the initiative to each region you want to monitor using the following steps:
Log into the Azure portal.
In the Azure search field, enter Policy.
Select Assignments.
Click Assign Initiative.
Choose the management group created in the previous step as the Scope.
Select the Send to Observe initiative created in the previous section.
Name the assignment Send to Observe -
.
Figure 10 - Initiative Assignment Basic Settings
Click Parameters.
Clear the Only show parameters that need input or review checkbox.
Select the region to deploy for the azureRegions parameter.
Select the eventhub name from the region.
Figure 11 - Initiative Assignment Parameters Event Hub name
5. Select the eventhub authorization rule for the chosen eventhub.
Figure 12 - Initiative Assignment Event Hub Authorization Rule
Figure 13 - Initiative Assignment Full Parameter Listing
6. Click Review + create and then Create.
Once you create the assignment, any new resources of the resource type automatically receive the diagnostic settings to send to Observe created a short period after you create the resource.
Remediation¶
To remediate non-compliant resources from the Azure portal, perform the following steps:
Log into the Azure portal.
In the Azure search field, enter Policy.
Click Remediation.
Find the policy for the resource type you want to remediate and click the 3 dots.
Click Remediate.
Click Remediate again on the Remediation page.
This creates the diagnostic settings for all of those resources sent to Observe.
Uninstalling the Azure app¶
To uninstall the Azure app from your Observe workspace, follow the instructions located at Apps page.
Remove the terraform-azure-collection module by running the following in the root directory:
terraform destroy
Note
You may encounter the following bug in the Azure provider during your removal of the app:
`Error: Deleting service principal with object ID “########-####-####-####-############”, got status 403
ServicePrincipalsClient.BaseClient.Delete(): unexpected status 403 with OData error: Authorization_RequestDenied: Insufficient privileges to complete the operation.`
If this happens, remove azuread_service_principal.observe_service_principal from the terraform state and continue the destroy process.
terraform state rm azuread_service_principal.observe_service_principal
terraform destroy
Find the service_principal object name by using the
idreturned, for example, replace it with yourid.
terraform state list -id=249783e5-bcfd-480b-b8e8-5f8aaa7452e8
2. Remove the object from the state. Make sure to wrap the object in single quotes, for example, and replace the name with that returned in the previous step.
terraform state rm 'module.collection["eastus"].azuread_service_principal.observe_service_principal'
3. Perform the terraform destroy function again.
terraform destroy
To uninstall the Observe Azure Integration using the Azure Resource Manager, follow these steps:
Using the Azure CLI, determine the Resource Group name for installed Observe deployment:
az deployment sub show --name observe --query ["properties.dependencies[].resourceGroup"]
This can also be optionally determined from Resource Groups
2. Delete the deployment at a subscription scope.
az deployment sub delete --name observe
3. Delete the Observe Resource Group for the installed Observe deployment and App Registration as shown here.
Note
Deleting the Observe Resource Group soft-deletes the Key Vault. To prevent any reinstallation errors, navigate to Key Vaults and purge the created Key Vault as shown here.
FAQ¶
Troubleshooting Azure¶
Note
Be sure you install Diagnostic Settings for your integration before you begin to troubleshoot your instance.
Q: I’ve installed the Observe collection module terraform-observe-collection, but no data flows into Observe; what do I check?
Check to see if your functions uploaded to your Observe Function app, called “observe-app-
- ”
Figure 3 - Debugging Function App
However, if you can’t find them, you most likely missed a step in the terraform-azure-collection. The most commonly seen error is not installing the missing Azure Functions Core Tools.
Enable Azure’s Application Insights and select your Observe Resource Group. Ensure functions perform correctly.
Figure 4 - Enabling Application Insights On Your Resource Group
Figure 5 - Select Application Insights Inside Your Resource Group
Figure 6 - Review Observe Application Insights
Note
If you decide to remove the Observe collection module, remove Azure Application Insights from the portal first.
If the functions exist, try restarting the instance for your Observe Function App.
Figure 7 - Restart Function App Machine
Q: Why don’t some of the messages in Function Logs link back to the Python Function Resource?
Any logging performed by functions in Python should use the logging module and not the print() function. Using the print() function generates separate events for each line and does not include the required metadata to tie it back to a function, or, in the case of multi-line stack traces, the other lines of the stack trace.
Manually Deleting the App¶
Deleting the Observe Resource Group and App Registration removes all resources created by Observe. Note that the Key Vault is soft deleted. To re-install - after deletion, you may need to navigate to Key Vaults and then click Manage deleted key vaults.
Figure 8 - Removing the Observe Resource Group
Figure 9 - Removing the Observe App Registration