Troubleshoot Azure

📘

Note

Be sure you install Diagnostic Settings for your integration (metrics and logs) before you begin to troubleshoot your instance. If none are configured, then only resource metadata will show up in Observe which can first take up to 20 minutes.

Q: I’ve installed via Azure Resource Manager or the Observe collection module terraform-observe-collection, but I don't see any data in Observe. How can I troubleshoot the app?

• Check to see if your deployment settings are correct, whether through parameters of Azure Resource Manager or through variables in azure.auto.tfvars if using Terraform.

• It’s common to swap values for objectId and enterpriseAppObjectId, or to introduce spaces or other extra characters in these settings.

• Check Observe ingest token observe_token is valid in the Observe instance and the corresponding datastream has not been deleted.

• Enable Azure's Application Insights and select your Observe Resource Group. Ensure that the application functions perform correctly.

1. Enabling Application Insights On Your Resource Group:

2. Select Application Insights inside your Resource Group:

3. Review Observe application insights:

📘

Note

If you decide to remove the Observe collection module, remove Azure Application Insights from the portal first.

If the functions exist, try restarting the instance for your Observe Function App.

Q: I see logs & metrics data in Observe but I don't see any resource metadata in Observe. How can I troubleshoot?

• Navigate to timer_resources_func function in the function app and ensure that successful invocations are taking place as in the below image:

• If there are failing invocations, navigate to the "Monitor" tab and click on an invocation to view failure details.

• By default, Terraform or ARM will automatically attempt to register any resource providers that it supports. Verify that the following resource providers are at least registered, as these are accessed by the timer_resources_func function. See Register resource provider in the Microsoft documentation for more information.

  • Microsoft.Compute
  • Microsoft.Storage
  • Microsoft.Network
  • Microsoft.Insights
  • Microsoft.Sql
  • Microsoft.Web
  • Microsoft.KeyVault
  • Microsoft.ContainerService
  • Microsoft.Resources
  • Microsoft.EventHub

Q: How do I send Azure collection function logs into Observe?

To send Azure collection function logs into Observe for an Observe engineer to troubleshoot, enable the diagnostic settings on the Observe function app:

1. Log into the Azure portal.
2. Search for the Function App in the search field and select the Function App.

3. Select the Observe collection function called observeApp-<OBSERVE-CUSTOMER-ID>-<AZURE-LOCATION>-<LAST-8-CHARS-OF-SUBSCRIPTION>.
4. Search for Diagnostic settings and select it.

5. Select Add diagnostic setting.

6. Select the following options:

• Diagnostic setting name - observe
• Function Application Logs
• AllMetrics
• Stream to an event hub

7. Select the event hub created by the Observe collection function. This will look like observeeventhub-<OBSERVE-CUSTOMER-ID>-<AZURE-LOCATION>-<LAST-8-CHARS-OF-SUBSCRIPTION>.
8. Click Save.

Q: I'm receiving HTTP status code 429 "Too many requests" errors in Function Logs. What should I do?

The 429 HTTP status is commonly used to reject a request because a call rate limit is reached. The client will auto retry requests but may raise an exception after repeated retries. This usually happens in the timer_resources_func function by Microsoft.Storage client's API calls, which has stricter rate limits for management operations.

Rate limiting may happen when there are large numbers of blob services and/or blob containers in a storage account. It is recommended to disable their collection by adding a feature flag as part of the function app environment variables, under App Settings.

FEATURE_FLAGS=!blob_services,!blob_containers

If managing via Terraform, this can be set by adding the following app_settings variable to your azure.auto.tfvars file

app_settings = {
  FEATURE_FLAGS = "!blob_services,!blob_containers"
}

If managing via ARM, this can be set by adding the feature_flags parameter to the ARM deployment command as below:

/az deployment sub create \
      --name observe \
      --location <THE_AZURE_LOCATION_TO_MONITOR> \
      --template-file resourcemanager-latest.json \
      --parameters \
        observe_customer=<YOUR_CUSTOMER_ID> \
        observe_token=<YOUR_DATASTREAM_TOKEN> \
        objectId=31a56... \
        applicationId=58339... \
        clientSecretValue=K3j8... \
        enterpriseAppObjectId=3aca8... \
        feature_flags='!blob_services,!blob_containers'

📘

Note

Before enabling feature flags, please ensure you're on the latest version of function app.

Q: Why don't some of the messages in Function Logs link back to the Python Function Resource?

Any logging performed by functions in Python should use the logging module and not the print() function. Using the print() function generates separate events for each line and does not include the required metadata to tie it back to a function, or, in the case of multi-line stack traces, the other lines of the stack trace.