What is Virtual Machine Scale Set (VMSS) and Its Usage
Azure Virtual Machine Scale Sets (VMSS) allow you to create and manage a group of load-balanced virtual machines (VMs). The number of VM instances can automatically increase or decrease on demand or by a defined schedule. While VMSS can be used in various scenarios such as automatically scaling out web servers, we primarily use it as self-hosted agent pools on Azure DevOps. The key benefits of using VMSS agents are auto-scaling and cost efficiency. That is, the number of agents (VMs) can automatically scale in and out on the demand of the pipeline jobs, avoiding the need for dedicated agents that remain idle.
What is Custom Script Extension and Its Usage
Custom Script Extension allows you to download and run scripts on the VMs for post-deployment configuration or software installation. In our case, we need to install the required security software, PowerShell modules, and common SDKs when spinning up the VMs for pipeline jobs. For example, the PowerShell script below performs several operations on a Windows VM:
- Install NuGet, the package manager for .NET
- Install PowerShell Az Modules
- Install Chocolatey, the package manager for Windows, and use it to install JDK 8 and Node.js
- Restart the Azure Pipeline agents
Install-PackageProvider -Name NuGet -Force
if (!(Get-Module -ListAvailable -Name Az.*)) {
Install-Module -Name Az -Repository PSGallery -Force
}
# https://chocolatey.org/install
if (!(Get-Command -Name choco.exe -ErrorAction SilentlyContinue)) {
Set-ExecutionPolicy Bypass -Scope Process -Force
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
Invoke-Expression ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
$env:Path += ";$($env:ChocolateyInstall)\bin"
}
choco install jdk8 -y
choco install nodejs.install -y
Restart-Service -DisplayName "Azure Pipelines Agent*"We typically store the script in a storage account, but the script location can be anywhere (such as GitHub or an internal file server) as long as the endpoint is accessible/routable from the VMSS.
Deploy Custom Script Extension
Through Azure Portal (Not Recommended)
Azure Portal provides the UI that allows you to create a Custom Scription Extension with scripts stored in storage accounts. Go to Settings -> Extensions + application and click Add. By searching “custom script extension,” you can select a script in a storage account to create the extension, as shown in the figures below. However, this method is not recommended because it generates a shared access signature (SAS) token for accessing the script file, and we don’t know when the SAS token will expire. Quote from Microsoft:
If you deploy the Custom Script Extension from the Azure portal, you don’t have control over the expiration of the SAS token for accessing the script in your storage account. The initial deployment works, but when the storage account’s SAS token expires, any subsequent scaling operation fails because the Custom Script Extension can no longer access the storage account.
A better approach is using PowerShell or Azure CLI to deploy Custom Scripting Extensions, which gives us direct control over the time allowed for script access.


With PowerShell
Using PowerShell commands, you can manage how to grant the VMSS access to the storage account: storage account key (full access), SAS, or managed identity (more limited, fine-grained access). The PowerShell script below creates the extension using the storage account key:
$vmss = Get-AzVmss -ResourceGroupName "<resource-group-name>" -VMScaleSetName "<vmss-name>"
$settings = @{
"fileUris" = @("https://<storage-account-name>.blob.core.windows.net/<container-name>/<script-name>.ps1")
"commandToExecute" = "powershell -ExecutionPolicy Unrestricted -File <script-name>.ps1>"
}
$protectedSettings = @{
"storageAccountName" = "<storage-account-name>"
"storageAccountKey" = "<storage-account-key>"
}
Add-AzVmssExtension -VirtualMachineScaleSet $vmss `
-Name "CustomScript" `
-Publisher "Microsoft.Compute" `
-Type "CustomScriptExtension" `
-TypeHandlerVersion 1.9 `
-Setting $settings `
-ProtectedSetting $protectedSettings
Update-AzVmss -ResourceGroupName "<resource-group-name>" -Name "<vmss-name>" -VirtualMachineScaleSet $vmssNote:
fileUrisis an array, so it is possible to put multiple script locations and execute them all. To that end, you need to create a lead script that calls other scripts.- If you want to use a SAS token (instead of a storage account key) to download the script file, remove
protectedSettingsand attach the SAS token to the link infileUris - To use a managed identity to download the script file, replace the content of
protectedSettingswith:
$protectedSettings = @{
"managedIdentity" = @{} # for system-assigned identity to the VMSS
# for user-assigned identity
# "managedIdentity" = @{ "clientId" = "<client-id>"}
}After the extension is created, you can see it in the Azure portal:

You can also log into one of the virtual machine instances to confirm if the required packages are installed:

With Azure CLI
The Azure CLI command az vmss extension set is used to add an extension to a VMSS or update an existing extension. The following command, modified from Tutorial: Install applications in Virtual Machine Scale Sets with the Azure CLI, adds an extension with a storage account key:
az vmss extension set \
--resource-group <resource-group-name>\
--vmss-name <vmss-name> \
--name CustomScript \
--publisher Microsoft.Azure.Extensions \
--version 2.0 \
--settings '{"fileUris": ["https://<storage-account-name>.blob.core.windows.net/<container-name>/<script-name>.ps1"]}' \
--protected-settings '{"commandToExecute": "powershell -ExecutionPolicy Unrestricted -File <script-name>.ps1", "storageAccountName": "<storage-account-name>", "storageAccountKey": "<storage-account-key>"}'Troubleshooting
During Extension Deployment
Multiple VMExtensions Not Supported
Update-AzVmss: Multiple VMExtensions per handler not supported for OS type 'Windows'. VMExtension 'CustomScript2' with handler 'Microsoft.Compute.CustomScriptExtension' already added or specified in input.This means you can only have one Custom Scription Extension on Windows VM/VMSS. Do not try to create multiple extensions for multiple scripts. Instead, create only one extension and put multiple links in fileUris, and first execute a lead script that calls other scripts.
403 Authorization Error Reported by the Storage
The troubleshooting process is similar to what we do for Databricks to access storage accounts.
- If the VMSS use a Virtual Network, make sure that the VNet/Subnet is whitelisted on the storage
- If a managed identity is used to download the scripts, make sure the identity has the proper IAM role, e.g., Storage Blob Data Reader, to read files from the storage container
During Script Execution
If the extension is deployed successfully but the script encounters errors during execution, you can go to any VM instance of the scale set, and click the extension to check its status and error message, as shown in the figure below:

If you need more details about the error, you can check the extension logs inside the VM. For example, on a Windows VM, the extension logs will typically reside in
C:\WindowsAzure\Logs\Plugins\Microsoft.Compute.CustomScriptExtension\<extension-version>\CustomScriptHandler.logYou will see logs like:
[1348+00000001] [02/02/2025 16:54:51.58] [INFO] Downloading files specified in configuration...
[1348+00000001] [02/02/2025 16:54:51.66] [INFO] DownloadFiles: fileUri = https://sasrehowtouwd.blob.core.windows.net/scripts/myscript.ps1?[REDACTED]
[1348+00000001] [02/02/2025 16:54:51.66] [INFO] DownloadFiles: Initializing CloudBlobClient with baseUri = https://sasrehowtouwd.blob.core.windows.net
[1348+00000001] [02/02/2025 16:54:51.69] [ERROR] DownloadFiles: Encountered an error while downloading files. Cancelling download tasks. Exception: System.ArgumentException: Missing mandatory parameters for valid Shared Access Signature
at Microsoft.WindowsAzure.Storage.Core.Util.NavigationHelper.ParseBlobQueryAndVerify(Uri address, StorageCredentials& parsedCredentials, Nullable`1& parsedSnapshot)
...On a Linux VM, the extension logs will typically reside in
/var/log/azure/
/var/lib/waagent/Refer to Troubleshooting Azure Windows VM extension failures for more known issues.
