[Azure] Key Vault Access Troubleshooting 101: Access Policy and Network

This post discusses the two common root causes that block access to Azure Key Vault: Access Policy and Network. In my experience, 90% of Key Vault access issues can be resolved by addressing these two factors. Simply put, to determine if an identity (user or application) can retrieve the stored data, the Key Vault checks: (1) if related permissions have been granted to the identity; and (2) if the source network of the identity (e.g., its public IP or VNet) has been allowed/whitelisted on the Key Vault. Two use cases from the perspective of an individual user and a web app will be illustrated to explain how the authorization works. We then introduce a tool (MSI Validator) that is helpful for advanced troubleshooting.

Case 1: Manage Secrets in Azure Key Vault as a User

Access Policy

Azure Key Vault is a service that allows you to securely manage sensitive information like API keys, passwords, and certificates. It is similar to Secrets Manager in AWS or GCP. As a user, we may want to store some kind of secrets, such as API tokens, in the Key Vault through Azure Portal. When users try to access a Key Vault, the most common errors they see are permission-related ones. For instance, the screenshot below shows that the user is not authorized to view the secrets. Another example is that the user can view the secrets but cannot create or delete them.

User is not authorized to view secrets in the Key Vault
User cannot create or delete secrets in the Key Vault

The above error message says, “The operation is not enabled in this key vault’s access policy.” So we know the key vault uses access policy model (*) for its data plane authorization (i.e., to control who can manage the stored data, but not the Key Vault itself). We can also check this in Access Configuration:

Permission model for Key Vault’s data plane

To resolve the above issue, we simply go to the access policies and create a new policy for that specific user with read/write permissions for secrets (you will need an appropriate IAM role such as Key Vault Contributor to create policies, i.e., to manage the Key Vault itself)

Grant user permissions in the access policies

(*) Azure provides two permissions models for data authorization: RBAC (Role-Based Access Control) and Access Policy. Even if RBAC has been recommended by Microsoft recently, Access Policy lets you have more fine-grained control over access, and it is still very common in legacy Key Vaults. We discuss the RBAC model for key vault in another post.

Network

Once the permissions are granted in the access policies, the user is good to go — if the network does not block her! In fact, if the Key Vault is not open to the public, the user will see another error about the firewall:

Access blocked by the firewall

The solution is straightforward:

  1. Identify the user’s Public IP. It can be obtained by visiting https://httpbin.org/ip. In an enterprise setting, the IP(s) can be the outbound IP range of the organization’s VPN or NAT’ed network.
  2. Navigate to Networking and add the public IP(s) to the firewall. Ta-da! You can finally manage secrets in the Key Vault!
Add IP addresses to the Key Vault’s firewall
Display secrets in Key Vault

Case 2: Retrieve the Stored Secret as a Web App

Use Key Vault References as App Settings

Another common scenario is that a web app needs to retrieve sensitive configurations stored in a key vault. For example, in the above screenshot, the key vault kv-srehowto-uw contains a secret called api-token that our web app needs. In the app, we can set an environment variable referring to that secret using key vault reference. The value of the env variable must follow one of the formats:

  • @Microsoft.KeyVault(SecretUri=https://<vault-name>.vault.azure.net/secrets/<secret-name>)
  • @Microsoft.KeyVault(VaultName=<vault-name>;SecretName=<secret-name>)
Add key vault reference as an environment variable
Key Vault reference error

After setting up the key vault reference, you may see reference errors in the Source column, as shown in the above screenshot. By clicking the error you may see more error details. Again, to resolve the issue, the top two things on our checklist are permission and network.

First, we find the identity used by the web app and check if the identity has permissions in the key vault. Navigate to Identity of the web app, you will see either system-assigned or user-assigned identity of the app. In our example, this app has a system-assigned identity, i.e., the Service Principal with the same name as the app.

System-assigned identity of the web app, with the same name as the app

We then go to the Access Policies of the Key Vault and make sure the identity has related permissions.

After the permissions are confirmed, we check the network. In the app’s Networking, you will see that the outbound IP of the app is one of the following:

  • If the app has VNet Integration, the outbound network is the VNet/Subnet shown in the Virtual network integration
  • If not, the outbound IPs are the list shown in the Outbound addresses
Outbound network or IPs of the web app

In an enterprise setting, we usually recommend using VNet Integration for better security. Another reason to use VNet Integration is that the outbound addresses may change periodically as they are provided by Azure. Once having the app’s outbound network (vnet-srehowto-uw/sn-srehowto-uw in this case), we add it to the Key Vault’s Networking.

Add VNet/Subnet to Key Vault’s Networking

Go back to the app’s environment variables and pull reference values. This time you will see the cute green checkmark on the key vault reference!

Validated key vault reference in the environment variables

MSI-Validator

MSI (Managed Service Identity)-Validator is a command line tool that checks the connection to Key Vault, Storage Account, or SQL Server on behalf of the App Service. It works on App Services hosted on Windows and using Managed Identity (i.e., system-assigned identity). It provides more error details than Azure Portal and is helpful for advanced troubleshooting. For instance, the steps to test the connection from our example web app to the key vaults are:

  1. Download the executable of MSI-Validator from here (source: Managed Identity for Azure App Services – Microsoft Community Hub). Alternatively, you can build the executable from the source code.
  2. Drag and drop the executable to the app’s Kudu Debug console. You can reach the Kudu Console by visiting https://<your-app-name>.scm.azurewebsites.net, or by navigating to Advances Tools in Azure Portal (search “Kudu” in the menu).
  3. Run msi-validator test-connection -r "keyvault" -e "https://<keyvault-name>.vault.azure.net/secrets/<secret-key>" to retrieve the secret value
Using MSI-Valiator from web app to get stored secret in Key Vault

You can see from the above screenshot that the real value of the secret is returned. If anything blocks the connection between the app and the key vault, you will also get detailed error messages to troubleshoot further.

Error returned by MSI-Validator

Jim
Jim

Site reliability engineer working in a global gaming and entertainment company

Articles: 18

Leave a Reply

Your email address will not be published. Required fields are marked *