Error when running az ml data import

[matsuobasho]

Describe the bug

Get an error when trying to upload a dataset to a blobstorage.

Related command

1. Create local_file.yaml file:

name: test-data
description: Data asset created from local file.
type: uri_file
path: ../test_data/file_name.pkl

2. Run

az ml data import -f absolute_path/cloud/local_file.yaml --workspace-name <ws_name> --resource-group <rg> --datastore <ds_name>

Note that the file for upload is in a directory that's parallel to the cloud subdirectory, hence I specify a relative path.

Errors

Class WorkspaceHubOperations: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Method import_data: This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
←[91mMet error <class 'AttributeError'>:'Data' object has no attribute 'source'
Please check log by running the command with '--debug' for more details.←[0m

Issue script & Debug output

Enable color in terminal.←[0m
←[36mcli.knack.cli: Event: Cli.PreExecute []←[0m
←[36mcli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x00000221049153F0>, <function OutputProducer.on_global_arguments at 0x0000022104A900D0>, <function CLIQuery.on_global_arguments at 
0x0000022104CAD3F0>]←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []←[0m
←[36mcli.azure.cli.core: Modules found from index for 'ml': ['azext_mlv2']←[0m
←[36mcli.azure.cli.core: Loading command modules:←[0m
←[36mcli.azure.cli.core: Name                  Load Time    Groups  Commands←[0m
←[36mcli.azure.cli.core: Total (0)                 0.000         0         0←[0m
←[36mcli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']←[0m
←[36mcli.azure.cli.core: Loading extensions:←[0m
←[36mcli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory←[0m
←[36mazext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.←[0m
←[36mazext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.←[0m
←[36mazext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.←[0m
←[36mcli.azure.cli.core: ml                        1.659        21       145  C:\Users\bob\.azure\cliextensions\ml←[0m
←[36mcli.azure.cli.core: Total (1)                 1.659        21       145  ←[0m
←[36mcli.azure.cli.core: Loaded 21 groups, 145 commands.←[0m
←[36mcli.azure.cli.core: Found a match in the command table.←[0m
←[36mcli.azure.cli.core: Raw command  : ml data import←[0m
←[36mcli.azure.cli.core: Command table: ml data import←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x00000221073F0550>]←[0m
←[36mcli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 'C:\Users\bob\.azure\commands\2023-11-14.19-11-27.ml_data_import.15740.log'.←[0m
←[32maz_command_data_logger: command args: ml data import -f {} --workspace-name {} --resource-group {} --datastore {} --debug←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x00000221073F3130>]←[0m
←[32mcli.knack.commands: Configured default 'bob-rg' for arg resource_group_name←[0m
←[32mcli.knack.commands: Configured default 'aus-east' for arg workspace_name←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x000002210743D1B0>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x000002210743D2D0>]←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPreParseArgs []←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x0000022104A90160>, <function CLIQuery.handle_query_parameter at 0x0000022104CAD480>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x000002210743D240>]←[0m
←[32maz_command_data_logger: extension name: ml←[0m
←[32maz_command_data_logger: extension version: 2.20.0←[0m
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
←[36mcli.azure.cli.core.commands.client_factory: Getting management service client client_type=AzureMachineLearningWorkspaces←[0m
←[36mcli.azure.cli.core.auth.persistence: build_persistence: location='C:\\Users\\bob\\.azure\\msal_token_cache.bin', encrypt=True←[0m
←[36mcli.azure.cli.core.auth.binary_cache: load: C:\Users\bob\.azure\msal_http_cache.bin←[0m
←[36murllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)←[0m
←[36mmsal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/kerberos', 'tenant_region_scope': 'NA', 
'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}←[0m
←[36mmsal.application: Broker enabled? False←[0m
←[36mcli.azure.cli.core.commands.client_factory: Getting management service client client_type=_ml_client_cli←[0m
←[36murllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)←[0m
←[36mmsal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/kerberos', 'tenant_region_scope': 'NA', 
'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}←[0m
←[36mmsal.application: Broker enabled? False←[0m
Class WorkspaceHubOperations: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Method import_data: This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Traceback (most recent call last):
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\custom\data.py", line 152, in ml_data_import
    job = ml_client.data.import_data(data_import, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_telemetry\activity.py", line 275, in wrapper
    return f(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_utils\_experimental.py", line 100, in wrapped
    return func(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\operations\_data_operations.py", line 388, in import_data
    data_import.type = AssetTypes.MLTABLE if isinstance(data_import.source, Database) else AssetTypes.URI_FOLDER
AttributeError: 'Data' object has no attribute 'source'
←[91mcli: None←[0m
←[36mcli: Traceback (most recent call last):
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\custom\data.py", line 152, in ml_data_import
    job = ml_client.data.import_data(data_import, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_telemetry\activity.py", line 275, in wrapper
    return f(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_utils\_experimental.py", line 100, in wrapped
    return func(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\operations\_data_operations.py", line 388, in import_data
    data_import.type = AssetTypes.MLTABLE if isinstance(data_import.source, Database) else AssetTypes.URI_FOLDER
AttributeError: 'Data' object has no attribute 'source'
←[0m
←[36mcli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\custom\data.py", line 152, in ml_data_import
    job = ml_client.data.import_data(data_import, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_telemetry\activity.py", line 275, in wrapper
    return f(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_utils\_experimental.py", line 100, in wrapped
    return func(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\operations\_data_operations.py", line 388, in import_data
    data_import.type = AssetTypes.MLTABLE if isinstance(data_import.source, Database) else AssetTypes.URI_FOLDER
AttributeError: 'Data' object has no attribute 'source'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 663, in execute
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 697, in _run_job
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 333, in __call__
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\custom\data.py", line 156, in ml_data_import
    log_and_raise_error(err, debug, yaml_operation=yaml_operation)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\custom\raise_error.py", line 181, in log_and_raise_error
    raise cli_error
knack.util.CLIError: Met error <class 'AttributeError'>:'Data' object has no attribute 'source'
Please check log by running the command with '--debug' for more details.
←[0m
←[91mcli.azure.cli.core.azclierror: Met error <class 'AttributeError'>:'Data' object has no attribute 'source'
Please check log by running the command with '--debug' for more details.←[0m
←[91maz_command_data_logger: Met error <class 'AttributeError'>:'Data' object has no attribute 'source'
Please check log by running the command with '--debug' for more details.←[0m
←[36mcli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x00000221073F0790>]←[0m
←[32maz_command_data_logger: exit code: 1←[0m
←[32mcli.__main__: Command ran in 2.344 seconds (init: 0.245, invoke: 2.099)←[0m
←[32mtelemetry.main: Begin splitting cli events and extra events, total events: 1←[0m
←[32mtelemetry.client: Accumulated 0 events. Flush the clients.←[0m
←[32mtelemetry.main: Finish splitting cli events and extra events, cli events: 1←[0m
←[32mtelemetry.save: Save telemetry record of length 3868 in cache←[0m
←[32mtelemetry.main: Begin creating telemetry upload process.←[0m
←[32mtelemetry.process: Creating upload process: "C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry\__init__.pyc C:\Users\bob\.azure"←[0m
←[32mtelemetry.process: Return from creating process←[0m
←[32mtelemetry.main: Finish creating telemetry upl

Expected behavior

Dataset is added to the blobstore.

Environment Summary

azure-cli: 2.53.0
azure-cli-core: 2.53.0
azure-cli-telemetry: 1.1.0
extensions:
  ml: 2.20.0

Additional context

Issue perhaps related to the blob storage that I created. When I try to add the data manually to it in the GUI, I see this message:

I created this using the following CLI command:
az ml datastore create -f path_to_file/blob.yaml --workspace-name <ws_name> --resource-group <rg>

[yonzhan]

Thank you for opening this issue, we will look into it.

[matsuobasho]

I realized my mistake, I was running az ml data import instead of az ml data create. The former is intended for import from database or filesystem, while I'm importing from my local machine.
https://learn.microsoft.com/en-us/cli/azure/ml/data?view=azure-cli-latest#az-ml-data-create