Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression: az ml job create fails while uploading code @ v2.17.0 #6294

Closed
dionhaefner opened this issue May 17, 2023 · 13 comments
Closed

Regression: az ml job create fails while uploading code @ v2.17.0 #6294

dionhaefner opened this issue May 17, 2023 · 13 comments
Assignees
@navba-MSFT
Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported Issues that are reported by GitHub users external to the Azure organization. CXP Attention This issue is handled by CXP team. extension/ml Machine Learning

Comments

@dionhaefner
Copy link
Contributor

dionhaefner commented May 17, 2023 •
edited
Loading

Describe the bug

Command Name
az ml job create
Extension Name: ml. Version: 2.17.0.

Errors:

Code: UserError
Message: User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

  • Compute cluster behind a VNet
  • Cache miss on the job code to be uploaded
  • az ml job create -f {} --set {} --workspace {} --resource-group {} --debug

Expected Behavior

Job gets scheduled successfully, as it does with az extension add -n ml --version 2.16.0.

Since it works fine with the older version I assume that I actually do have all the necessary permissions.

Environment Summary

macOS-13.3.1-arm64-arm-64bit, Darwin 22.4.0
Python 3.10.11
Installer: HOMEBREW

azure-cli 2.48.1

Extensions:
ml 2.17.0
ssh 1.1.6

Dependencies:
msal 1.20.0
azure-mgmt-resource 22.0.0

Additional Context

👇

Full debug output 
DEBUG: cli.knack.cli: Command arguments: ['ml', 'job', 'create', '-f', '00-testjob.yml', '--query', 'name', '-o', 'tsv', '--debug', '--set', 'inputs.job_name=fast']
DEBUG: cli.knack.cli: __init__ debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [, , ]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'ml': ['azext_mlv2']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands
DEBUG: cli.azure.cli.core: Total (0)                 0.000         0         0
DEBUG: cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
DEBUG: cli.azure.cli.core: Loading extensions:
DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
DEBUG: azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
DEBUG: azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
DEBUG: azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
DEBUG: cli.azure.cli.core: ml                        1.402        20       138  /opt/az/azcliextensions/ml
DEBUG: cli.azure.cli.core: Total (1)                 1.402        20       138  
DEBUG: cli.azure.cli.core: Loaded 20 groups, 138 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command  : ml job create
DEBUG: cli.azure.cli.core: Command table: ml job create
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate []
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/runner/.azure/commands/2023-05-17.20-04-46.ml_job_create.3236.log'.
INFO: az_command_data_logger: command args: ml job create -f {} --query {} -o {} --debug --set {}
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [.add_subscription_parameter at 0x7f17e76df6d0>]
INFO: cli.knack.commands: Configured default 'rg-pasteur-shared-eastus-001' for arg resource_group_name
INFO: cli.knack.commands: Configured default 'ml-pasteur-shared-eastus-002' for arg workspace_name
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [.add_ids_arguments at 0x7f17e7721630>, .add_cache_arguments at 0x7f17e7721750>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [, , .parse_ids_arguments at 0x7f17e77216c0>]
INFO: az_command_data_logger: extension name: ml
INFO: az_command_data_logger: extension version: 2.17.0
DEBUG: cli.azure.cli.core.commands.***_factory: Getting management service *** ***_type=AzureMachineLearningWorkspaces
DEBUG: cli.azure.cli.core.auth.persistence: build_persistence: location='/home/runner/.azure/service_principal_entries.json', encrypt=False
DEBUG: cli.azure.cli.core.auth.persistence: build_persistence: location='/home/runner/.azure/msal_token_cache.json', encrypt=False
DEBUG: cli.azure.cli.core.auth.binary_cache: load: /home/runner/.azure/msal_http_cache.bin
DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG: msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['***_secret_post', 'private_key_jwt', '***_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/***/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/***/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/***/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net/'}
DEBUG: msal.application: Broker enabled? False
DEBUG: msal.application: Region to be used: None
DEBUG: cli.azure.cli.core.commands.***_factory: Getting management service *** ***_type=_ml_***_cli
DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG: msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['***_secret_post', 'private_key_jwt', '***_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/***/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/***/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/***/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net/'}
DEBUG: msal.application: Broker enabled? False
DEBUG: msal.application: Region to be used: None
Class AutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class AutoDeleteConditionSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseAutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class IntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class ProtectionLevelSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseIntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
DEBUG: attr_dict: getting schema_ignored
DEBUG: attr_dict: setting schema_ignored to {}
DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.azure.com/.default',), kwargs={}
DEBUG: cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.azure.com/.default',), kwargs={}
DEBUG: msal.application: Cache hit an AT
DEBUG: msal.telemetry: Generate or reuse correlation_id: fedd27d3-4d2b-45ed-9445-01b7ecdfac54
DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions/***/resourceGroups/rg-pasteur-shared-eastus-001/providers/Microsoft.MachineLearningServices/workspaces/ml-pasteur-shared-eastus-002/codes/bbb13029-1b47-4630-9539-3aa3596f3e92/versions?api-version=2023-04-01&hash=2f7028b6b3d386f99c243d833e81258022e29fbb7b472544f8b6a6b57025c27c&hashVersion=202208'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-***-request-id': '113dbb33-f4ee-11ed-9c71-db7919d1ca5a'
    'User-Agent': 'azureml-cli-v2/2.17.0 azure-ai-ml/1.7.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.10.10 (Linux-5.15.0-1037-azure-x86_64-with-glibc2.35) '
    'Authorization': '***'
Request body:
This request has no body
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
DEBUG: urllib3.connectionpool: [https://management.azure.com:443](https://management.azure.com/) "GET /subscriptions/***/resourceGroups/rg-pasteur-shared-eastus-001/providers/Microsoft.MachineLearningServices/workspaces/ml-pasteur-shared-eastus-002/codes/bbb13029-1b47-4630-9539-3aa3596f3e92/versions?api-version=2023-04-01&hash=2f7028b6b3d386f99c243d833e81258022e29fbb7b472544f8b6a6b57025c27c&hashVersion=202208 HTTP/1.1" 200 None
DEBUG: azure.core.pipeline.policies._universal: Response status: '200'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'gzip'
    'Expires': '-1'
    'Vary': 'Accept-Encoding'
    'x-ms-ratelimit-remaining-subscription-reads': '11997'
    'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d'
    'x-ms-response-type': 'standard'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-eastus-01'
    'x-request-time': '0.038'
    'Server-Timing': 'traceparent;desc="00-743009c3963e29541c290a11a2d7027d-49219efc0fa2f032-01"'
    'x-ms-request-id': '474bd4bb-205d-4e03-8d0d-8503166f8062'
    'x-ms-correlation-request-id': '474bd4bb-205d-4e03-8d0d-8503166f8062'
    'x-ms-routing-request-id': 'EASTUS:20230517T200446Z:474bd4bb-205d-4e03-8d0d-8503166f8062'
    'Date': 'Wed, 17 May 2023 20:04:46 GMT'
Response content:
{
  "value": []
}
DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions/***/resourceGroups/rg-pasteur-shared-eastus-001/providers/Microsoft.MachineLearningServices/workspaces/ml-pasteur-shared-eastus-002/codes/bbb13029-1b47-4630-9539-3aa3596f3e92/versions/1/startPendingUpload?api-version=2023-04-01'
Request method: 'POST'
Request headers:
    'Content-Type': 'application/json'
    'Content-Length': '47'
    'Accept': 'application/json'
    'x-ms-***-request-id': '113dbb33-f4ee-11ed-9c71-db7919d1ca5a'
    'User-Agent': 'azureml-cli-v2/2.17.0 azure-ai-ml/1.7.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.10.10 (Linux-5.15.0-1037-azure-x86_64-with-glibc2.35) '
    'Authorization': '***'
Request body:
{"pendingUploadType": "TemporaryBlobReference"}
DEBUG: urllib3.connectionpool: [https://management.azure.com:443](https://management.azure.com/) "POST /subscriptions/***/resourceGroups/rg-pasteur-shared-eastus-001/providers/Microsoft.MachineLearningServices/workspaces/ml-pasteur-shared-eastus-002/codes/bbb13029-1b47-4630-9539-3aa3596f3e92/versions/1/startPendingUpload?api-version=2023-04-01 HTTP/1.1" 401 1174
DEBUG: azure.core.pipeline.policies._universal: Response status: '401'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Content-Length': '1174'
    'Content-Type': 'application/json; charset=utf-8'
    'Expires': '-1'
    'x-ms-ratelimit-remaining-subscription-writes': '1199'
    'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d'
    'x-ms-response-type': 'error'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-eastus-01'
    'x-request-time': '0.079'
    'x-ms-request-id': '909e24bd-a6c4-4034-94bd-a61a96f61d86'
    'x-ms-correlation-request-id': '909e24bd-a6c4-4034-94bd-a61a96f61d86'
    'x-ms-routing-request-id': 'EASTUS:20230517T200446Z:909e24bd-a6c4-4034-94bd-a61a96f61d86'
    'Date': 'Wed, 17 May 2023 20:04:46 GMT'
    'Connection': 'close'
Response content:
{
  "error": {
    "code": "UserError",
    "message": "User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..",
    "details": [],
    "additionalInfo": [
      {
        "type": "ComponentName",
        "info": {
          "value": "managementfrontend"
        }
      },
      {
        "type": "Correlation",
        "info": {
          "value": {
            "operation": "b363aec2026e45b83dcdf8cc6590b562",
            "request": "9158cff4005704cb"
          }
        }
      },
      {
        "type": "Environment",
        "info": {
          "value": "eastus"
        }
      },
      {
        "type": "Location",
        "info": {
          "value": "eastus"
        }
      },
      {
        "type": "Time",
        "info": {
          "value": "2023-05-17T20:04:46.9600341+00:00"
        }
      },
      {
        "type": "InnerError",
        "info": {
          "value": {
            "code": "Auth",
            "innerError": {
              "code": "Authorization",
              "innerError": null
            }
          }
        }
      }
    ]
  }
}
Traceback (most recent call last):
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/job.py", line 90, in ml_job_create
    job = ml_***.jobs.create_or_update(job=job, skip_validation=skip_validation)
  File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 337, in wrapper
    return_value = f(*args, **kwargs)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 609, in create_or_update
    raise ex
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 541, in create_or_update
    self._resolve_arm_id_or_upload_dependencies(job)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 890, in _resolve_arm_id_or_upload_dependencies
    self._resolve_arm_id_or_azureml_id(job, self._orchestrators.get_asset_arm_id)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1103, in _resolve_arm_id_or_azureml_id
    job = self._resolve_arm_id_for_command_job(job, resolver)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1140, in _resolve_arm_id_for_command_job
    job.code = resolver(
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 232, in get_asset_arm_id
    result = self._get_code_asset_arm_id(asset, register_asset=register_asset)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 298, in _get_code_asset_arm_id
    raise e
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 273, in _get_code_asset_arm_id
    code_asset = self._code_assets.create_or_update(code_asset)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 263, in wrapper
    return f(*args, **kwargs)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 180, in create_or_update
    raise ex
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 118, in create_or_update
    sas_info = get_storage_info_for_non_registry_asset(
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 1036, in get_storage_info_for_non_registry_asset
    response = service_***.code_versions.create_or_get_start_pending_upload(
  File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_rest***/v2023_04_01/operations/_code_versions_operations.py", line 678, in create_or_get_start_pending_upload
    map_error(status_code=response.status_code, response=response, error_map=error_map)
  File "/opt/az/lib/python3.10/site-packages/azure/core/exceptions.py", line 107, in map_error
    raise error
azure.core.exceptions.ClientAuthenticationError: (UserError) User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..
Code: UserError
Message: User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..
ERROR: cli: None
DEBUG: cli: Received HttpResponseError: Traceback (most recent call last):
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/job.py", line 90, in ml_job_create
    job = ml_***.jobs.create_or_update(job=job, skip_validation=skip_validation)
  File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 337, in wrapper
    return_value = f(*args, **kwargs)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 609, in create_or_update
    raise ex
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 541, in create_or_update
    self._resolve_arm_id_or_upload_dependencies(job)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 890, in _resolve_arm_id_or_upload_dependencies
    self._resolve_arm_id_or_azureml_id(job, self._orchestrators.get_asset_arm_id)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1103, in _resolve_arm_id_or_azureml_id
    job = self._resolve_arm_id_for_command_job(job, resolver)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1140, in _resolve_arm_id_for_command_job
    job.code = resolver(
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 232, in get_asset_arm_id
    result = self._get_code_asset_arm_id(asset, register_asset=register_asset)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 298, in _get_code_asset_arm_id
    raise e
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 273, in _get_code_asset_arm_id
    code_asset = self._code_assets.create_or_update(code_asset)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 263, in wrapper
    return f(*args, **kwargs)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 180, in create_or_update
    raise ex
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 118, in create_or_update
    sas_info = get_storage_info_for_non_registry_asset(
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 1036, in get_storage_info_for_non_registry_asset
    response = service_***.code_versions.create_or_get_start_pending_upload(
  File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_rest***/v2023_04_01/operations/_code_versions_operations.py", line 678, in create_or_get_start_pending_upload
    map_error(status_code=response.status_code, response=response, error_map=error_map)
  File "/opt/az/lib/python3.10/site-packages/azure/core/exceptions.py", line 107, in map_error
    raise error
azure.core.exceptions.ClientAuthenticationError: (UserError) User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..
Code: UserError
Message: User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..

DEBUG: cli.azure.cli.core.azclierror: Traceback (most recent call last):

File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke

cmd_result = self.invocation.execute(args)

File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 663, in execute

raise ex

File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 726, in _run_jobs_serially

results.append(self._run_job(expanded_arg, cmd_copy))

File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 697, in run_job

result = cmd_copy(params)

File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 333, in call

return self.handler(*args, **kwargs)

File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler

return op(**command_args)

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/job.py", line 104, in ml_job_create

log_and_raise_error(err, debug, yaml_operation=yaml_operation)

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/raise_error.py", line 181, in log_and_raise_error

raise cli_error

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/job.py", line 90, in ml_job_create

job = ml***.jobs.create_or_update(job=job, skip_validation=skip_validation)

File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer

return func(*args, **kwargs)

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 337, in wrapper

return_value = f(*args, **kwargs)

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 609, in create_or_update

raise ex

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 541, in create_or_update

self._resolve_arm_id_or_upload_dependencies(job)

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 890, in _resolve_arm_id_or_upload_dependencies

self._resolve_arm_id_or_azureml_id(job, self._orchestrators.get_asset_arm_id)

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1103, in _resolve_arm_id_or_azureml_id

job = self._resolve_arm_id_for_command_job(job, resolver)

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1140, in _resolve_arm_id_for_command_job

job.code = resolver(

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 232, in get_asset_arm_id

result = self._get_code_asset_arm_id(asset, register_asset=register_asset)

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 298, in _get_code_asset_arm_id

raise e

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 273, in _get_code_asset_arm_id

code_asset = self._code_assets.create_or_update(code_asset)

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 263, in wrapper

return f(args, kwargs)

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 180, in create_or_update

raise ex

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 118, in create_or_update

sas_info = get_storage_info_for_non_registry_asset(

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/asset_utils.py", line 1036, in get_storage_info_for_non_registry_asset

response = service.code_versions.create_or_get_start_pending_upload(

File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer

return func(args, kwargs)

File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_rest/v2023_04_01/operations/_code_versions_operations.py", line 678, in create_or_get_start_pending_upload

map_error(status_code=response.status_code, response=response, error_map=error_map)

File "/opt/az/lib/python3.10/site-packages/azure/core/exceptions.py", line 107, in map_error

raise error

azure.core.exceptions.ClientAuthenticationError: (UserError) User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..

Code: UserError

Message: User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..


ERROR: cli.azure.cli.core.azclierror: (UserError) User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..

Code: UserError

Message: User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..

ERROR: az_command_data_logger: (UserError) User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..

Code: UserError

Message: User is not authorized to access provided resources due to none of the workspaces/registries are accessible from outside of the designated private link..

DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f17e76c6dd0>]

INFO: az_command_data_logger: exit code: 1

INFO: cli.main: Command ran in 2.412 seconds (init: 0.176, invoke: 2.235)

INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1

INFO: telemetry.***: Accumulated 0 events. Flush the ***s.

INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1

INFO: telemetry.save: Save telemetry record of length 4270 in cache

WARNING: telemetry.check: Negative: The /home/runner/.azure/telemetry.txt was modified at 2023-05-17 20:04:04.441504, which in less than 600.000000 s

@ghost ghost added question The issue doesn't require a change to the product in order to be resolved. Most issues start as that customer-reported Issues that are reported by GitHub users external to the Azure organization. Auto-Assign Auto assign by bot labels May 17, 2023
@yonzhan
Copy link
Collaborator

yonzhan commented May 17, 2023

Thank you for opening this issue, we will look into it.

@ghost ghost added CXP Attention This issue is handled by CXP team. Machine Learning extension/ml labels May 17, 2023
@navba-MSFT navba-MSFT self-assigned this Jul 4, 2023
@navba-MSFT
Copy link
Contributor

@dionhaefner Apologies for the late reply. Thanks for reaching out to us and reporting this issue. We are looking into this issue and we will provide an update.

This error occurs when the workspace or registry you are trying to access is not accessible from outside of the designated private link.

To resolve this issue, you can try the following steps:

Check if your workspace or registry is behind a private endpoint. If it is, you need to create a private endpoint connection to access it. You can follow the steps in the Create a private endpoint connection to Azure Machine Learning article to create a private endpoint connection.

If your workspace or registry is not behind a private endpoint, check if you have the necessary permissions to access it. You can check your permissions by going to the Access control (IAM) tab in the Azure portal and checking if you have the necessary roles assigned.

If you are using a service principal to authenticate, make sure that it has the necessary permissions to access the workspace or registry. You can check the permissions of your service principal by going to the Access control (IAM) tab in the Azure portal and checking if it has the necessary roles assigned.

@navba-MSFT navba-MSFT added the needs-author-feedback More information is needed from author to address the issue. label Jul 4, 2023
@dionhaefner
Copy link
Contributor Author

We have double and triple checked all permissions, and everything works fine with v2.16.0.

Are there additional permissions required since v2.17.0?

@ghost ghost added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. labels Jul 4, 2023
@navba-MSFT
Copy link
Contributor

@dionhaefner I am waiting to hear back from the Product team on this. I will keep you posted. We appreciate your patience.

@navba-MSFT
Copy link
Contributor

@dionhaefner Could you please run the same above CLI command with --debug parameter and share the detailed debug output here ? Awaiting your reply.

@navba-MSFT navba-MSFT added needs-author-feedback More information is needed from author to address the issue. and removed needs-team-attention This issue needs attention from Azure service team or SDK team labels Jul 13, 2023
@dionhaefner
Copy link
Contributor Author

@navba-MSFT It's in the original post ("Additional Context").

@microsoft-github-policy-service microsoft-github-policy-service bot added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. labels Jul 13, 2023
@navba-MSFT
Copy link
Contributor

@dionhaefner My bad. I missed that completely. I will check this further and get back to you.

@navba-MSFT navba-MSFT added bug This issue requires a change to an existing behavior in the product in order to be resolved. and removed question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jul 20, 2023
@navba-MSFT
Copy link
Contributor

@dionhaefner Could you please share the subscription, resource group, workspace name, and timestamps (in UTC) when this happens over email navba [@] microsoft . (dot) com ?

@navba-MSFT navba-MSFT added needs-author-feedback More information is needed from author to address the issue. and removed needs-team-attention This issue needs attention from Azure service team or SDK team labels Jul 20, 2023
@navba-MSFT
Copy link
Contributor

@dionhaefner I wanted to do quick follow-up to check if you had a chance to look at my above comment. Please let us know if you had any updates on this. Awaiting your reply.

@dionhaefner
Copy link
Contributor Author

I've been on vacation, will send you the info ASAP.

@microsoft-github-policy-service microsoft-github-policy-service bot added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. labels Jul 25, 2023
@navba-MSFT
Copy link
Contributor

I've been on vacation, will send you the info ASAP.

@dionhaefner Any update on this ?

@navba-MSFT navba-MSFT added needs-author-feedback More information is needed from author to address the issue. and removed needs-team-attention This issue needs attention from Azure service team or SDK team labels Jul 27, 2023
@dionhaefner
Copy link
Contributor Author

It seems that I can't reproduce the issue anymore. Everything works as it should now, both with the most recent az ml extension and v2.17.0. Hard to tell whether this is because our setup changed in the meantime or because the bug has been fixed.

I will upgrade our production environment to the most recent version of az ml and see what happens. I'll report back with the requested information if the issue re-surfaces.

Thanks for the support @navba-MSFT!

@microsoft-github-policy-service microsoft-github-policy-service bot added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. labels Jul 27, 2023
@navba-MSFT
Copy link
Contributor

@dionhaefner Thanks for getting back. We will now proceed with closure of this GitHub issue. If you need any further assistance on this issue in future, please feel free to reopen this thread. We would be happy to help.

@navba-MSFT navba-MSFT removed the needs-team-attention This issue needs attention from Azure service team or SDK team label Aug 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@navba-MSFT navba-MSFT

Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported Issues that are reported by GitHub users external to the Azure organization. CXP Attention This issue is handled by CXP team. extension/ml Machine Learning
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dionhaefner @yonzhan @navba-MSFT