AzureRM Builder: private_virtual_network_with_public_ip assigns a Basic SKU Public IP

[dan-hills-hrt]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Overview of the Issue

When attempting to build using the AzureARM builder and using the property private_virtual_network_with_public_ip, the VM resource is being deployed with a public IP assigned with a Basic SKU.

During the build process, if I attempt to associate the resource with an existing vnet which is using a NAT rule, this fails as it expects and supports only Standard SKU public ips.

The NAT gateway has been used as a way to access a separate, remote resources from the Packer VM using a trusted IP address.

Reproduction Steps

Steps to reproduce this issue

1. Assign both the private_virtual_network_with_public_ip and virtual_network_name to a resource declaration.
2. On the az vnet resource, attach a NAT gateway with a public IP (the public ip resource must be of a Standard SKU type).
3. Trigger the build. During resource deployment, the error above will occur. You will also be able to confirm the SKU type of the public ip resource after a failed build.

Packer version

1.11.2

Simplified Packer Template

source "azure-arm" "azure_windows-desktop" {

    # Azure configuration
    client_id       = var.client_id
    client_secret   = var.client_secret
    subscription_id = var.subscription_id
    tenant_id       = var.tenant_id
    azure_tags      = {
        task = "image-deployment"
    }
    build_resource_group_name = var.resource_group_name

    # Image configuration
    image_publisher = "MicrosoftWindowsDesktop"
    image_offer     = "Windows-11"
    image_sku       = "win11-24h2-ent"

    # VM configuration
    vm_size                           =  "Standard_DS4_v2"
    license_type                      = "Windows_Client"
    os_type                           = "Windows"
    os_disk_size_gb                   = 128
    managed_image_name                = "packerBuild_test"
    managed_image_resource_group_name = var.resource_group_name

    # Network configuration
    private_virtual_network_with_public_ip = true
    virtual_network_name                   = var.virtual_network_name
    virtual_network_resource_group_name    = var.resource_group_name
    virtual_network_subnet_name            = var.virtual_network_subnet_name

    # User configuration
    communicator   = "winrm"
    winrm_username = "internaladmin"
    winrm_password = var.winrm_password
    winrm_insecure = true
    winrm_use_ssl  = true
    winrm_timeout  = "5m"
}

Operating system and Environment details

Windows 11

Log Fragments and crash.log files

The trimmed log received:

==> azure-arm.azure_windows-desktop: Deploying deployment template ...
==> azure-arm.azure_windows-desktop:  -> ResourceGroupName : 'rg_packer'
==> azure-arm.azure_windows-desktop:  -> DeploymentName    : 'pkrdp29jtugfv40'
==> azure-arm.azure_windows-desktop: ERROR: -> DeploymentFailed : At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.
==> azure-arm.azure_windows-desktop: ERROR:   -> SubnetWithNatGatewayAndBasicSkuResourceNotAllowed : NAT Gateway /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/natGateways/ngw_packer cannot be deployed on subnet containing Basic SKU Public IP addresses or Basic SKU Load Balancer. NIC /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/networkInterfaces/pkrni29jtugfv40/ipConfigurations/ipconfig in subnet /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/virtualNetworks/vnet_packer/subnets/nat has reference to Basic SKU Public IP address or Load Balancer /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/publicIPAddresses/pkrip29jtugfv40.
==> azure-arm.azure_windows-desktop:
==> azure-arm.azure_windows-desktop: polling after CreateOrUpdate: polling failed: the Azure API returned the following error:
==> azure-arm.azure_windows-desktop:
==> azure-arm.azure_windows-desktop: Status: "DeploymentFailed"
==> azure-arm.azure_windows-desktop: Code: "SubnetWithNatGatewayAndBasicSkuResourceNotAllowed"
==> azure-arm.azure_windows-desktop: Message: "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\nNAT Gateway /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/natGateways/ngw_packer cannot be deployed on subnet containing Basic SKU Public IP addresses or Basic SKU Load Balancer. NIC /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/networkInterfaces/pkrni29jtugfv40/ipConfigurations/ipconfig in subnet /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/virtualNetworks/vnet_packer/subnets/nat has reference to Basic SKU Public IP address or Load Balancer /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/publicIPAddresses/pkrip29jtugfv40."
==> azure-arm.azure_windows-desktop: Activity Id: ""
==> azure-arm.azure_windows-desktop:
==> azure-arm.azure_windows-desktop: ---
==> azure-arm.azure_windows-desktop:
==> azure-arm.azure_windows-desktop: API Response:
==> azure-arm.azure_windows-desktop:
==> azure-arm.azure_windows-desktop: ----[start]----
==> azure-arm.azure_windows-desktop: {"status":"Failed","error":{"code":"DeploymentFailed","target":"/subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Resources/deployments/pkrdp29jtugfv40","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"SubnetWithNatGatewayAndBasicSkuResourceNotAllowed","message":"NAT Gateway /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/natGateways/ngw_packer cannot be deployed on subnet containing Basic SKU Public IP addresses or Basic SKU Load Balancer. NIC /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/networkInterfaces/pkrni29jtugfv40/ipConfigurations/ipconfig in subnet /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/virtualNetworks/vnet_infra_packer/subnets/nat has reference to Basic SKU Public IP address or Load Balancer /subscriptions/<subscription>/resourceGroups/rg_packer/providers/Microsoft.Network/publicIPAddresses/pkrip29jtugfv40.","details":[]}]}}
==> azure-arm.azure_windows-desktop: -----[end]-----

This error repeats a few more times but the errors seems to be consistent.

After a build attempt, I was also able to confirm that the public ip resource is of a Basic SKU type:

Name            sku   tier    
----            ---   ----    
pkrip29jtugfv40 Basic Regional

This SKU will also need to be updated to Standard as the Basic SKU for public IPs is planned to be retired and may not function in the future: https://azure.microsoft.com/en-us/updates?id=upgrade-to-standard-sku-public-ip-addresses-in-azure-by-30-september-2025-basic-sku-will-be-retired

[JenGoldstrich]

@dan-hills-hrt I am working on fixing this now and was curious if you could provide some feedback on the approach, we have until the end of March before this becomes a breaking change, so ideally I hope to fix this in the next week or so.

Standard SKU IPs require a network security group to allow ingress, currently whenever creating a Standard SKU Public IP the plugin creates a new NSG that allows ingress from any IP, or from a list of any allowed IP addresses the user specifies.

My assumption would be that if a user has created there own subnet, it would be preferable for them to manage their own Network Security Group outside of the Packer build process, and that Packer would just try and create a Public IP with Standard SKU and assign it to the pre-existing subnet. However there is another recently opened issue #468 opened by @cailyoung who has requested the ability to set the client IP addresses when setting private_virtual_network_with_public_ip.

I could see an implementation where the plugin tries to create the NSG and assign it to the existing subnet, but there are risks to this as it involves us modifying an existing resource, which may likely already have an NSG, subnets can only have one. There is also risk here if the plugin unexpectedly halts or is SIGKILLed that we'd leave an orphaned uncleaned up NSG exposing the user's vnet, which seems like an unacceptable risk to me. Do you think that expecting the users to create their own NSG here is reasonable?

[cailyoung]

@JenGoldstrich is there a reason the NSG isn't being applied to the NIC instead of the subnet?

[cailyoung]

In our case the source IP restriction will float due to where we run Packer from. So a prebuilt NSG wouldn't work for us, however I can see the value in BYO NSG (similar to BYO VNet)

[JenGoldstrich]

There isn't a reason as far as I know @cailyoung from the docs you link it seems like that's the way to go to enable us to handle the problem of wanting Packer to manage the NSG for you in a private vnet with public IP scenario. IMO the default should probably be to expect users to configure their own NSG still, and then allow users to enable the new functionality (just to make it less of a disruptive change to people's existing network infra). I'll try wiring that up to test, thanks for the suggestion!