[az containerapp env certificate upload] Container Apps Environment - InternalServerError on adding a certificate via KeyVault and Managed Identity

[omni-htg]

Describe the bug

When trying to add a new certificate to a Container Apps Environment using the KeyVault option with an assigned Managed Identity, the process fails with (InternalServerError) Internal server error occurred.

The Managed Identity was both assigned to the Env using the Settings / Identity tab in the Azure Portal, and providing it via the --identity option in the cli.
It also has both Key Vault Secrets Officer and Key Vault Certificates Officer roles, as indicated in this article.

This happens in all cases:

• Azure Portal
• Azure Preview Portal
• az CLI using containerapp extension 0.3.52

Thank you for your assistance.

Related command

az containerapp env certificate upload -g <ResourceGroup> --name <Environment> --akv-url <Certificate's Secret url> --identity <Resource ID for Managed Identity>

Errors

(InternalServerError) Internal server error occurred. correlation ID: 87b5614d-1a2f-4a5e-a3ae-2da7c67db809

Issue script & Debug output

urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "PUT /subscriptions/SUB/resourceGroups/RG/providers/Microsoft.App/managedEnvironments/CAE/certificates/<autogenerated_name>?api-version=2024-02-02-preview HTTP/1.1" 500 203
cli.azure.cli.core.util: Response status: 500
cli.azure.cli.core.util: Response headers:
cli.azure.cli.core.util: 'Cache-Control': 'no-cache'
cli.azure.cli.core.util: 'Pragma': 'no-cache'
cli.azure.cli.core.util: 'Content-Length': '203'
cli.azure.cli.core.util: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.util: 'Expires': '-1'
cli.azure.cli.core.util: 'x-ms-ratelimit-remaining-subscription-resource-requests': '99'
cli.azure.cli.core.util: 'api-supported-versions': '2022-03-01, 2022-06-01-preview, 2022-10-01, 2022-11-01-preview, 2023-04-01-preview, 2023-05-01, 2023-05-02-preview, 2023-08-01-preview, 2023-11-02-preview, 2024-02-02-preview, 2024-03-01'
cli.azure.cli.core.util: 'X-Powered-By': 'ASP.NET'
cli.azure.cli.core.util: 'x-ms-failure-cause': 'service'
cli.azure.cli.core.util: 'x-ms-request-id': '87b5614d-1a2f-4a5e-a3ae-2da7c67db809'
cli.azure.cli.core.util: 'x-ms-correlation-request-id': '87b5614d-1a2f-4a5e-a3ae-2da7c67db809'
cli.azure.cli.core.util: 'x-ms-routing-request-id': REDACTED
cli.azure.cli.core.util: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.util: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.util: 'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.util: 'X-MSEdge-Ref': REDACTED
cli.azure.cli.core.util: 'Date': 'Wed, 05 Jun 2024 13:36:12 GMT'
cli.azure.cli.core.util: Response content:
cli.azure.cli.core.util: {"error":{"code":"InternalServerError","message":"Internal server error occurred. correlation ID: 87b5614d-1a2f-4a5e-a3ae-2da7c67db809","details":null,"target":null,"additionalInfo":null,"traceId":null}}
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "C:\Users\USER.azure\cliextensions\containerapp\azext_containerapp\containerapp_env_certificate_decorator.py", line 160, in create_or_update
r = self.client.create_or_update_certificate(self.cmd, self.get_argument_resource_group_name(), self.get_argument_name(), self.cert_name, self.certificate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/containerapp/_clients.py", line 692, in create_or_update_certificate
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/util.py", line 1007, in send_raw_request
azure.cli.core.azclierror.HTTPError: Internal Server Error({"error":{"code":"InternalServerError","message":"Internal server error occurred. correlation ID: 87b5614d-1a2f-4a5e-a3ae-2da7c67db809","details":null,"target":null,"additionalInfo":null,"traceId":null}})

Expected behavior

A new certificate entity being created for the CAE that is associated to the indicated KeyVault Certificate through the Managed Identity.

Environment Summary

azure-cli 2.61.0
core 2.61.0
telemetry 1.1.0

Extensions:
containerapp 0.3.52
cosmosdb-preview 1.0.0
interactive 0.5.3

Dependencies:
msal 1.28.0
azure-mgmt-resource 23.1.1

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\USER.azure\cliextensions'

Python (Windows) 3.11.8 (tags/v3.11.8:db85d51, Feb 6 2024, 22:03:32) [MSC v.1937 64 bit (AMD64)]

Additional context

Curiously, it seems that even though the CLI is using the "latest" containerapp extension, it is using the 2024-02-02-preview API Version, when apparently a stable 2024-03-01 is already available?

Thank you!

[yonzhan]

Thank you for opening this issue, we will look into it.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @calvinsID.

[Greedygre]

Hi @omni-htg

I checked the error log, error code is ForbiddenByFirewall.

error message:
Client address is not authorized and caller is not a trusted service

You can refer this to authorize the client address IP(The managed environment's OutBoundIP) to the key vault:
https://stackoverflow.com/questions/52985252/client-address-is-not-authorized-and-caller-is-not-a-trusted-service-in-azure

[omni-htg]

Thank you for looking into this, @Greedygre .

When you mention

The managed environment's OutBoundIP

do you mean the Static Ip ?

I currently have my Container Apps (and environment) in a subnet on the same VNet as the KeyVault.
What I did not do was:

1. Add that Static Ip to the Firewall IP list.
2. Add the Azure.KeyVault service endpoint to the Container Apps subnet.

What is in place is:

1. The "Allow trusted Microsoft services to bypass this firewall" exception.
2. The Container Apps themselves seem to be able to access the KeyVault when adding a secret (or at least, they get added and don't report any issues).

I will do some trial and error on my side, but any confirmation on yours would be greatly appreaciated!
If possible, I'd love to avoid having to add the CAE IP to the Firewall.

---

On another note, is it intended that I don't recieve the error message ForbiddenByFirewall on the CLI, and that the "latest" extension is not using the latest API version?

Thanks a ton!

[Greedygre]

ForbiddenByFirewall
Hi @omni-htg

The ForbiddenByFirewall is not shown in CLI now.

The OutBoundIP, you can get it from your containerapp with command:
az containerapp show -n {} -g {} --query "properties.outboundIpAddresses"

[omni-htg]

In my environment, the CAE is an Internal one, only available to the VNet where all the other Azure services (including the KeyVault) have been set up -- I believe this is why properties.outboundIpAddresses is null.
I have tried to add the staticIp property from the CAE into the Firewall IP list in the KV, but was disallowed for it being a "private IP".
Please advise.

[chinadragon0515]

@omni-htg

Can I know what you mean “I currently have my Container Apps (and environment) in a subnet on the same VNet as the KeyVault.”？

Azure keyvault supports add vnet/private endpoint, can you add Container apps env subnet to the allow list or create private endpoint for AKV, then container apps can access AKV via private endpoint.

thanks
Vincent

[omni-htg]

I apologize, tried to explain in a quick fashion and made it more convoluted.

Azure keyvault supports add vnet/private endpoint, can you add Container apps env subnet to the allow list

This is exactly what was needed.
Initially I had the KV as part of the VNet by "allowing" a KeyVault-specific Subnet, completely missing the point.
Once I added the Container Apps subnet it worked flawlessly.

So now I can close this, thank you!
I'll leave with the suggestion that, if possible, add a more verbose message on Azure Portal and CLI when it gets blocked by the Firewall like in my case.