This post discusses the two common root causes that block access to Azure Key Vault: Access Policy and Network. In my experience, 90% of Key Vault access issues can be resolved by addressing these two factors. Simply put, to determine if an identity (user or application) can retrieve the stored data, the Key Vault checks: (1) if related permissions have been granted to the identity; and (2) if the source network of the identity (e.g., its public IP or VNet) has been allowed/whitelisted on the Key Vault. Two use cases from the perspective of an individual user and a web app will be illustrated to explain how the authorization works. We then introduce a tool (MSI Validator) that is helpful for advanced troubleshooting.
Case 1: Manage Secrets in Azure Key Vault as a User
Access Policy
Azure Key Vault is a service that allows you to securely manage sensitive information like API keys, passwords, and certificates. It is similar to Secrets Manager in AWS or GCP. As a user, we may want to store some kind of secrets, such as API tokens, in the Key Vault through Azure Portal. When users try to access a Key Vault, the most common errors they see are permission-related ones. For instance, the screenshot below shows that the user is not authorized to view the secrets. Another example is that the user can view the secrets but cannot create or delete them.


The above error message says, “The operation is not enabled in this key vault’s access policy.” So we know the key vault uses access policy model (*) for its data plane authorization (i.e., to control who can manage the stored data, but not the Key Vault itself). We can also check this in Access Configuration:

To resolve the above issue, we simply go to the access policies and create a new policy for that specific user with read/write permissions for secrets (you will need an appropriate IAM role such as Key Vault Contributor to create policies, i.e., to manage the Key Vault itself)

(*) Azure provides two permissions models for data authorization: RBAC (Role-Based Access Control) and Access Policy. Even if RBAC has been recommended by Microsoft recently, Access Policy lets you have more fine-grained control over access, and it is still very common in legacy Key Vaults. We discuss the RBAC model for key vault in another post.
Network
Once the permissions are granted in the access policies, the user is good to go — if the network does not block her! In fact, if the Key Vault is not open to the public, the user will see another error about the firewall:

The solution is straightforward:
- Identify the user’s Public IP. It can be obtained by visiting https://httpbin.org/ip. In an enterprise setting, the IP(s) can be the outbound IP range of the organization’s VPN or NAT’ed network.
- Navigate to Networking and add the public IP(s) to the firewall. Ta-da! You can finally manage secrets in the Key Vault!


Case 2: Retrieve the Stored Secret as a Web App
Use Key Vault References as App Settings
Another common scenario is that a web app needs to retrieve sensitive configurations stored in a key vault. For example, in the above screenshot, the key vault kv-srehowto-uw contains a secret called api-token that our web app needs. In the app, we can set an environment variable referring to that secret using key vault reference. The value of the env variable must follow one of the formats:
@Microsoft.KeyVault(SecretUri=https://<vault-name>.vault.azure.net/secrets/<secret-name>)@Microsoft.KeyVault(VaultName=<vault-name>;SecretName=<secret-name>)


After setting up the key vault reference, you may see reference errors in the Source column, as shown in the above screenshot. By clicking the error you may see more error details. Again, to resolve the issue, the top two things on our checklist are permission and network.
First, we find the identity used by the web app and check if the identity has permissions in the key vault. Navigate to Identity of the web app, you will see either system-assigned or user-assigned identity of the app. In our example, this app has a system-assigned identity, i.e., the Service Principal with the same name as the app.

We then go to the Access Policies of the Key Vault and make sure the identity has related permissions.

After the permissions are confirmed, we check the network. In the app’s Networking, you will see that the outbound IP of the app is one of the following:
- If the app has VNet Integration, the outbound network is the VNet/Subnet shown in the Virtual network integration
- If not, the outbound IPs are the list shown in the Outbound addresses

In an enterprise setting, we usually recommend using VNet Integration for better security. Another reason to use VNet Integration is that the outbound addresses may change periodically as they are provided by Azure. Once having the app’s outbound network (vnet-srehowto-uw/sn-srehowto-uw in this case), we add it to the Key Vault’s Networking.

Go back to the app’s environment variables and pull reference values. This time you will see the cute green checkmark on the key vault reference!

MSI-Validator
MSI (Managed Service Identity)-Validator is a command line tool that checks the connection to Key Vault, Storage Account, or SQL Server on behalf of the App Service. It works on App Services hosted on Windows and using Managed Identity (i.e., system-assigned identity). It provides more error details than Azure Portal and is helpful for advanced troubleshooting. For instance, the steps to test the connection from our example web app to the key vaults are:
- Download the executable of MSI-Validator from here (source: Managed Identity for Azure App Services – Microsoft Community Hub). Alternatively, you can build the executable from the source code.
- Drag and drop the executable to the app’s Kudu Debug console. You can reach the Kudu Console by visiting
https://<your-app-name>.scm.azurewebsites.net, or by navigating to Advances Tools in Azure Portal (search “Kudu” in the menu). - Run
msi-validator test-connection -r "keyvault" -e "https://<keyvault-name>.vault.azure.net/secrets/<secret-key>"to retrieve the secret value

You can see from the above screenshot that the real value of the secret is returned. If anything blocks the connection between the app and the key vault, you will also get detailed error messages to troubleshoot further.

