[SerialConsole] Cannot connect to Virtual Machine Scalesets using azure cli serial console extension

[rhkodiak]

Describe the bug

There is an issue with the azure cli serial console extension where if a user creates a Virtual Machine Scaleset and tries to connect to one of the instances an error is thrown. The error message that get's thrown is below.

(OperationNotAllowed) Operation 'VMScaleSetVMs.instanceView.GET' is not allowed on Virtual Machine Scale Set 'rhkodiak-vmss'.
Code: OperationNotAllowed
Message: Operation 'VMScaleSetVMs.instanceView.GET' is not allowed on Virtual Machine Scale Set 'rhkodiak-vmss'.

The command used to connect to the VMSS instance is also below.

az serial-console connect -n connect --name rhkodiak-vmss --resource-group rhkodiak_vmss --instance-id 1

Code updates will be needed to fix this issue with the serial console extension.

Related command

az serial-console connect -n connect --name rhkodiak-vmss --resource-group rhkodiak_vmss --instance-id 1

Errors

(OperationNotAllowed) Operation 'VMScaleSetVMs.instanceView.GET' is not allowed on Virtual Machine Scale Set 'rhkodiak-vmss'.
Code: OperationNotAllowed
Message: Operation 'VMScaleSetVMs.instanceView.GET' is not allowed on Virtual Machine Scale Set 'rhkodiak-vmss'.

Issue script & Debug output

cli.knack.cli: Command arguments: ['serial-console', 'connect', '-n', 'connect', '--name', 'rhkodiak-vmss', '--resource-group', 'rhkodiak_vmss', '--instance-id', '1', '--debug']
cli.knack.cli: init debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fb6102201f0>, <function OutputProducer.on_global_arguments at 0x7fb61013ad30>, <function CLIQuery.on_global_arguments at 0x7fb6100d3310>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'serial-console': ['azext_serialconsole']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: Total (0) 0.000 0 0
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: ai-examples 0.112 1 1 /usr/lib/python3.9/site-packages/azure-cli-extensions/ai-examples
cli.azure.cli.core: serial-console 0.002 2 6 /home/rhkodiak/.azure/cliextensions/serial-console
cli.azure.cli.core: Total (2) 0.114 3 7
cli.azure.cli.core: Loaded 3 groups, 7 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : serial-console connect
cli.azure.cli.core: Command table: serial-console connect
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7fb60f577430>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/rhkodiak/.azure/commands/2023-10-04.14-00-27.serial-console_connect.344.log'.
az_command_data_logger: command args: serial-console connect -n {} --name {} --resource-group {} --instance-id {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x7fb60f521040>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x7fb60f5080d0>, <function register_cache_arguments..add_cache_arguments at 0x7fb60f511ee0>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fb61013adc0>, <function CLIQuery.handle_query_parameter at 0x7fb6100d33a0>, <function register_ids_argument..parse_ids_arguments at 0x7fb60f511e50>]
az_command_data_logger: extension name: serial-console
az_command_data_logger: extension version: 0.1.6
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=ComputeManagementClient
urllib3.connectionpool: Starting new HTTP connection (1): localhost:50342
urllib3.connectionpool: http://localhost:50342 "POST /oauth2/token HTTP/1.1" 200 2444
msrestazure.azure_active_directory: MSI: Retrieving a token from http://localhost:50342/oauth2/token, with payload {'resource': 'https://management.core.windows.net/'}
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=StorageManagementClient
urllib3.connectionpool: Starting new HTTP connection (1): localhost:50342
urllib3.connectionpool: http://localhost:50342 "POST /oauth2/token HTTP/1.1" 200 2444
msrestazure.azure_active_directory: MSI: Retrieving a token from http://localhost:50342/oauth2/token, with payload {'resource': 'https://management.core.windows.net/'}
cli.azure.cli.core.auth.adal_authentication: MSIAuthenticationWrapper.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
urllib3.connectionpool: Starting new HTTP connection (1): localhost:50342
urllib3.connectionpool: http://localhost:50342 "POST /oauth2/token HTTP/1.1" 200 2444
msrestazure.azure_active_directory: MSI: Retrieving a token from http://localhost:50342/oauth2/token, with payload {'resource': 'https://management.core.windows.net/'}
cli.azure.cli.core.auth.adal_authentication: Normalize expires_on: '1696430020' -> 1696430020
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/xxxxxxxx/resourceGroups/rhkodiak_vmss/providers/Microsoft.Compute/virtualMachineScaleSets/rhkodiak-vmss/virtualMachines/1/instanceView?api-version=2022-11-01'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '5ed23ace-62be-11ee-bb6d-bace1cce66db'
cli.azure.cli.core.sdk.policies: 'CommandName': 'serial-console connect'
cli.azure.cli.core.sdk.policies: 'ParameterSetName': '-n --name --resource-group --instance-id --debug'
cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.53.0 (RPM) azsdk-python-azure-mgmt-compute/30.0.0 Python/3.9.14 (Linux-5.4.0-1109-azure-x86_64-with-glibc2.35) cloud-shell/1.0'
cli.azure.cli.core.sdk.policies: 'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/xxxxxxxx/resourceGroups/rhkodiak_vmss/providers/Microsoft.Compute/virtualMachineScaleSets/rhkodiak-vmss/virtualMachines/1/instanceView?api-version=2022-11-01 HTTP/1.1" 409 180
cli.azure.cli.core.sdk.policies: Response status: 409
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Content-Length': '180'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-resource': 'Microsoft.Compute/GetVMScaleSetVM3Min;999,Microsoft.Compute/GetVMScaleSetVM30Min;4998,Microsoft.Compute/VMScaleSetVMViews3Min;4999'
cli.azure.cli.core.sdk.policies: 'x-ms-request-charge': '1'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '673e7798-05dd-45b3-bea6-21eb9e3bcdb3'
cli.azure.cli.core.sdk.policies: 'Server': 'Microsoft-HTTPAPI/2.0, Microsoft-HTTPAPI/2.0'
cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '11999'
cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '4dabcc7d-3764-4802-9fc3-b942c94d31d8'
cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'SOUTHCENTRALUS:20231004T140027Z:4dabcc7d-3764-4802-9fc3-b942c94d31d8'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'Date': 'Wed, 04 Oct 2023 14:00:27 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {
"error": {
"code": "OperationNotAllowed",
"message": "Operation 'VMScaleSetVMs.instanceView.GET' is not allowed on Virtual Machine Scale Set 'rhkodiak-vmss'."
}
}
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 663, in execute
raise ex
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 697, in _run_job
result = cmd_copy(params)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 333, in call
return self.handler(*args, **kwargs)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
File "/home/rhkodiak/.azure/cliextensions/serial-console/azext_serialconsole/custom.py", line 631, in connect_serialconsole
check_resource(cmd.cli_ctx, resource_group_name,
File "/home/rhkodiak/.azure/cliextensions/serial-console/azext_serialconsole/custom.py", line 595, in check_resource
result, storage_account_region = get_region_from_storage_account(cli_ctx, resource_group_name, vm_vmss_name,
File "/home/rhkodiak/.azure/cliextensions/serial-console/azext_serialconsole/custom.py", line 687, in get_region_from_storage_account
result_data = client.virtual_machine_scale_set_vms.get_instance_view(
File "/usr/lib64/az/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
return func(*args, **kwargs)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/mgmt/compute/v2022_11_01/operations/_operations.py", line 13052, in get_instance_view
map_error(status_code=response.status_code, response=response, error_map=error_map)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/core/exceptions.py", line 107, in map_error
raise error
azure.core.exceptions.ResourceExistsError: (OperationNotAllowed) Operation 'VMScaleSetVMs.instanceView.GET' is not allowed on Virtual Machine Scale Set 'rhkodiak-vmss'.
Code: OperationNotAllowed
Message: Operation 'VMScaleSetVMs.instanceView.GET' is not allowed on Virtual Machine Scale Set 'rhkodiak-vmss'.

cli.azure.cli.core.azclierror: (OperationNotAllowed) Operation 'VMScaleSetVMs.instanceView.GET' is not allowed on Virtual Machine Scale Set 'rhkodiak-vmss'.
Code: OperationNotAllowed
Message: Operation 'VMScaleSetVMs.instanceView.GET' is not allowed on Virtual Machine Scale Set 'rhkodiak-vmss'.
az_command_data_logger: (OperationNotAllowed) Operation 'VMScaleSetVMs.instanceView.GET' is not allowed on Virtual Machine Scale Set 'rhkodiak-vmss'.
Code: OperationNotAllowed
Message: Operation 'VMScaleSetVMs.instanceView.GET' is not allowed on Virtual Machine Scale Set 'rhkodiak-vmss'.
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7fb60f577670>]
az_command_data_logger: exit code: 1
cli.main: Command ran in 0.714 seconds (init: 0.141, invoke: 0.573)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3851 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/usr/bin/python3.9 /usr/lib/az/lib/python3.9/site-packages/azure/cli/telemetry/init.py /home/rhkodiak/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

The expected behavior is that the user would be able to connect to the specified instance of the virtual machine scaleset to use serial console for troubleshooting purposes.

Environment Summary

azure-cli 2.53.0

core 2.53.0
telemetry 1.1.0

Extensions:
ai-examples 0.2.5
ml 2.20.0
serial-console 0.1.6
ssh 2.0.1

Dependencies:
msal 1.24.0b2
azure-mgmt-resource 23.1.0b2

Python location '/usr/bin/python3.9'
Extensions directory '/home/rhkodiak/.azure/cliextensions'
Extensions system directory '/usr/lib/python3.9/site-packages/azure-cli-extensions'

Python (Linux) 3.9.14 (main, Feb 8 2023, 03:39:52)
[GCC 11.2.0]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[rhkodiak]

Hi @yonzhan ,

A customer reported this issue to our Azure Serial Console team and I was able reproduce the issue. I'm investigating the code to see if there is a fix that needs to occur or if this was the expected behavior.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @CraigWiand.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @CraigWiand.