Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

az ml online-endpoint create raises Resource Not Found Error #7107

Open
kmr0877 opened this issue Dec 20, 2023 · 5 comments
Open

az ml online-endpoint create raises Resource Not Found Error #7107

kmr0877 opened this issue Dec 20, 2023 · 5 comments
Labels
Auto-Assign Auto assign by bot customer-reported Issues that are reported by GitHub users external to the Azure organization. extension/ml Machine Learning question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention This issue is responsible by Azure service team.

Comments

@kmr0877
Copy link

kmr0877 commented Dec 20, 2023 •
edited
Loading

Describe the bug

When trying to create an endpoint using AML CLI raises an error shown below

Related command

az ml online-endpoint create \
            -f kubernetes.yml \
            --subscription ${SUBSCRIPTION} \
            --resource-group ${RESOURCE_GROUP} \
            --workspace-name ${WORKSPACE_NAME} \
            --auth-mode key \
            --debug

Errors

2023-12-20T04:48:32.2079609Z Response content:
2023-12-20T04:48:32.2080089Z {
2023-12-20T04:48:32.2080533Z   "error": {
2023-12-20T04:48:32.2081040Z     "code": "NotFound",
2023-12-20T04:48:32.2081707Z     "message": "The specified resource was not found.",
2023-12-20T04:48:32.2082278Z     "details": [
2023-12-20T04:48:32.2083075Z       {
2023-12-20T04:48:32.2083645Z         "code": "InferencingClientCallFailed",
2023-12-20T04:48:32.2084259Z         "message": "{{\"title\":\"Not Found\"}}",
2023-12-20T04:48:32.2084924Z         "details": [],
2023-12-20T04:48:32.2085381Z         "additionalInfo": []
2023-12-20T04:48:32.2085817Z       }
2023-12-20T04:48:32.2086197Z     ],
2023-12-20T04:48:32.2086617Z     "additionalInfo": [
2023-12-20T04:48:32.2087061Z       {
2023-12-20T04:48:32.2087492Z         "type": "ComponentName",
2023-12-20T04:48:32.2087944Z         "info": {
2023-12-20T04:48:32.2088403Z           "value": "managementfrontend"
2023-12-20T04:48:32.2088860Z         }
2023-12-20T04:48:32.2089255Z       },
2023-12-20T04:48:32.2089653Z       {
2023-12-20T04:48:32.2090072Z         "type": "Correlation",
2023-12-20T04:48:32.2090510Z         "info": {
2023-12-20T04:48:32.2090947Z           "value": {
2023-12-20T04:48:32.2091491Z             "operation": "22f61e***f8789bf3db42462ee64ee7b48",
2023-12-20T04:48:32.2092040Z             "request": "d***633f8acdb3ae57"
2023-12-20T04:48:32.2092499Z           }
2023-12-20T04:48:32.2092904Z         }
2023-12-20T04:48:32.2093289Z       },
2023-12-20T04:48:32.2093689Z       {
2023-12-20T04:48:32.2094124Z         "type": "Environment",
2023-12-20T04:48:32.2094588Z         "info": {
2023-12-20T04:48:32.2095037Z           "value": "australiaeast"
2023-12-20T04:48:32.2095474Z         }
2023-12-20T04:48:32.2095862Z       },
2023-12-20T04:48:32.2096273Z       {
2023-12-20T04:48:32.2096696Z         "type": "Location",
2023-12-20T04:48:32.2097140Z         "info": {
2023-12-20T04:48:32.2097610Z           "value": "australiaeast"
2023-12-20T04:48:32.2098055Z         }
2023-12-20T04:48:32.2098461Z       },
2023-12-20T04:48:32.2098856Z       {
2023-12-20T04:48:32.2099288Z         "type": "Time",
2023-12-20T04:48:32.2099728Z         "info": {
2023-12-20T04:48:32.2100400Z           "value": "2***23-12-2***T***4:48:31.7776546+***:***"
2023-12-20T04:48:32.2100888Z         }
2023-12-20T04:48:32.2101295Z       }
2023-12-20T04:48:32.2101696Z     ]
2023-12-20T04:48:32.2102083Z   }
2023-12-20T04:48:32.2102710Z }
2023-12-20T04:48:32.2900402Z Traceback (most recent call last):
2023-12-20T04:48:32.2901430Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/online_endpoint.py", line 122, in ml_online_endpoint_create
2023-12-20T04:48:32.2902400Z     endpoint = ml_client.begin_create_or_update(endpoint, local=local)
2023-12-20T04:48:32.2902968Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2903740Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_ml_client.py", line 1***87, in begin_create_or_update
2023-12-20T04:48:32.2904772Z     return _begin_create_or_update(entity, self._operation_container.all_operations, **kwargs)
2023-12-20T04:48:32.2905368Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2907895Z   File "/opt/az/lib/python3.11/functools.py", line 9***9, in wrapper
2023-12-20T04:48:32.2908539Z     return dispatch(args[***].__class__)(*args, **kw)
2023-12-20T04:48:32.2909077Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2909700Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_ml_client.py", line 1175, in _
2023-12-20T04:48:32.2910450Z     return operations[AzureMLResourceType.ONLINE_ENDPOINT].begin_create_or_update(entity, **kwargs)
2023-12-20T04:48:32.2911042Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2911999Z   File "/opt/az/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
2023-12-20T04:48:32.2912623Z     return func(*args, **kwargs)
2023-12-20T04:48:32.2913110Z            ^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2913743Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
2023-12-20T04:48:32.2914365Z     return f(*args, **kwargs)
2023-12-20T04:48:32.2914837Z            ^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2915538Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_endpoint_operations.py", line 256, in begin_create_or_update
2023-12-20T04:48:32.2916208Z     raise ex
2023-12-20T04:48:32.2916878Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_endpoint_operations.py", line 251, in begin_create_or_update
2023-12-20T04:48:32.2917548Z     raise ex
2023-12-20T04:48:32.2918232Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_endpoint_operations.py", line 24***, in begin_create_or_update
2023-12-20T04:48:32.2920295Z     poller = self._online_operation.begin_create_or_update(
2023-12-20T04:48:32.2920844Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2921604Z   File "/opt/az/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
2023-12-20T04:48:32.2922234Z     return func(*args, **kwargs)
2023-12-20T04:48:32.2922695Z            ^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2923798Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_restclient/v2***22_***2_***1_preview/operations/_online_endpoints_operations.py", line 921, in begin_create_or_update
2023-12-20T04:48:32.2924590Z     raw_result = self._create_or_update_initial(
2023-12-20T04:48:32.2925110Z                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2925896Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_restclient/v2***22_***2_***1_preview/operations/_online_endpoints_operations.py", line 855, in _create_or_update_initial
2023-12-20T04:48:32.2926764Z     map_error(status_code=response.status_code, response=response, error_map=error_map)
2023-12-20T04:48:32.2927595Z   File "/opt/az/lib/python3.11/site-packages/azure/core/exceptions.py", line 1***7, in map_error
2023-12-20T04:48:32.2928150Z     raise error
2023-12-20T04:48:32.2928927Z azure.core.exceptions.ResourceNotFoundError: (NotFound) The specified resource was not found.
2023-12-20T04:48:32.2929473Z Code: NotFound
2023-12-20T04:48:32.2929979Z Message: The specified resource was not found.
2023-12-20T04:48:32.2930618Z Exception Details:	(InferencingClientCallFailed) {{"title":"Not Found"}}
2023-12-20T04:48:32.2931217Z 	Code: InferencingClientCallFailed
2023-12-20T04:48:32.2931715Z 	Message: {{"title":"Not Found"}}
2023-12-20T04:48:32.2932185Z ERROR: cli: None
2023-12-20T04:48:32.2932892Z DEBUG: cli: Received HttpResponseError: Traceback (most recent call last):
2023-12-20T04:48:32.2933631Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/online_endpoint.py", line 122, in ml_online_endpoint_create
2023-12-20T04:48:32.2934340Z     endpoint = ml_client.begin_create_or_update(endpoint, local=local)
2023-12-20T04:48:32.2934893Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2935587Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_ml_client.py", line 1***87, in begin_create_or_update
2023-12-20T04:48:32.2936359Z     return _begin_create_or_update(entity, self._operation_container.all_operations, **kwargs)
2023-12-20T04:48:32.2936969Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2937517Z   File "/opt/az/lib/python3.11/functools.py", line 9***9, in wrapper
2023-12-20T04:48:32.2938126Z     return dispatch(args[***].__class__)(*args, **kw)
2023-12-20T04:48:32.2938639Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2939266Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_ml_client.py", line 1175, in _
2023-12-20T04:48:32.2939993Z     return operations[AzureMLResourceType.ONLINE_ENDPOINT].begin_create_or_update(entity, **kwargs)
2023-12-20T04:48:32.2940602Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2941362Z   File "/opt/az/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
2023-12-20T04:48:32.2941962Z     return func(*args, **kwargs)
2023-12-20T04:48:32.2942426Z            ^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2943071Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
2023-12-20T04:48:32.2943699Z     return f(*args, **kwargs)
2023-12-20T04:48:32.2944164Z            ^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2944871Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_endpoint_operations.py", line 256, in begin_create_or_update
2023-12-20T04:48:32.2945535Z     raise ex
2023-12-20T04:48:32.2946199Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_endpoint_operations.py", line 251, in begin_create_or_update
2023-12-20T04:48:32.2946843Z     raise ex
2023-12-20T04:48:32.2947508Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_endpoint_operations.py", line 24***, in begin_create_or_update
2023-12-20T04:48:32.2948270Z     poller = self._online_operation.begin_create_or_update(
2023-12-20T04:48:32.2948794Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2949525Z   File "/opt/az/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
2023-12-20T04:48:32.2950123Z     return func(*args, **kwargs)
2023-12-20T04:48:32.2950598Z            ^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2951380Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_restclient/v2***22_***2_***1_preview/operations/_online_endpoints_operations.py", line 921, in begin_create_or_update
2023-12-20T04:48:32.2952161Z     raw_result = self._create_or_update_initial(
2023-12-20T04:48:32.2952672Z                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.2953554Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_restclient/v2***22_***2_***1_preview/operations/_online_endpoints_operations.py", line 855, in _create_or_update_initial
2023-12-20T04:48:32.2954430Z     map_error(status_code=response.status_code, response=response, error_map=error_map)
2023-12-20T04:48:32.2955255Z   File "/opt/az/lib/python3.11/site-packages/azure/core/exceptions.py", line 1***7, in map_error
2023-12-20T04:48:32.2955808Z     raise error
2023-12-20T04:48:32.2956475Z azure.core.exceptions.ResourceNotFoundError: (NotFound) The specified resource was not found.
2023-12-20T04:48:32.2957035Z Code: NotFound
2023-12-20T04:48:32.2957545Z Message: The specified resource was not found.
2023-12-20T04:48:32.2958147Z Exception Details:	(InferencingClientCallFailed) {{"title":"Not Found"}}
2023-12-20T04:48:32.2958739Z 	Code: InferencingClientCallFailed
2023-12-20T04:48:32.2959233Z 	Message: {{"title":"Not Found"}}
2023-12-20T04:48:32.2959434Z 
2023-12-20T04:48:32.3408677Z DEBUG: cli.azure.cli.core.azclierror: Traceback (most recent call last):
2023-12-20T04:48:32.3409844Z   File "/opt/az/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
2023-12-20T04:48:32.3410700Z     cmd_result = self.invocation.execute(args)
2023-12-20T04:48:32.3411274Z                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3412104Z   File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
2023-12-20T04:48:32.3412737Z     raise ex
2023-12-20T04:48:32.3413596Z   File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
2023-12-20T04:48:32.3414368Z     results.append(self._run_job(expanded_arg, cmd_copy))
2023-12-20T04:48:32.3414983Z                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3415788Z   File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
2023-12-20T04:48:32.3416564Z     result = cmd_copy(params)
2023-12-20T04:48:32.3417032Z              ^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3417732Z   File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
2023-12-20T04:48:32.3418355Z     return self.handler(*args, **kwargs)
2023-12-20T04:48:32.3418848Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3419559Z   File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
2023-12-20T04:48:32.3420179Z     return op(**command_args)
2023-12-20T04:48:32.3420641Z            ^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3421267Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/online_endpoint.py", line 136, in ml_online_endpoint_create
2023-12-20T04:48:32.3421986Z     log_and_raise_error(err, debug, yaml_operation=yaml_operation)
2023-12-20T04:48:32.3422652Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/raise_error.py", line 181, in log_and_raise_error
2023-12-20T04:48:32.3423246Z     raise cli_error
2023-12-20T04:48:32.3423876Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/online_endpoint.py", line 122, in ml_online_endpoint_create
2023-12-20T04:48:32.3424601Z     endpoint = ml_client.begin_create_or_update(endpoint, local=local)
2023-12-20T04:48:32.3425145Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3425903Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_ml_client.py", line 1***87, in begin_create_or_update
2023-12-20T04:48:32.3426737Z     return _begin_create_or_update(entity, self._operation_container.all_operations, **kwargs)
2023-12-20T04:48:32.3427313Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3427886Z   File "/opt/az/lib/python3.11/functools.py", line 9***9, in wrapper
2023-12-20T04:48:32.3428474Z     return dispatch(args[***].__class__)(*args, **kw)
2023-12-20T04:48:32.3429008Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3429878Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_ml_client.py", line 1175, in _
2023-12-20T04:48:32.3430645Z     return operations[AzureMLResourceType.ONLINE_ENDPOINT].begin_create_or_update(entity, **kwargs)
2023-12-20T04:48:32.3431242Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3432004Z   File "/opt/az/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
2023-12-20T04:48:32.3432688Z     return func(*args, **kwargs)
2023-12-20T04:48:32.3433174Z            ^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3433808Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
2023-12-20T04:48:32.3434441Z     return f(*args, **kwargs)
2023-12-20T04:48:32.3434890Z            ^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3435606Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_endpoint_operations.py", line 256, in begin_create_or_update
2023-12-20T04:48:32.3436273Z     raise ex
2023-12-20T04:48:32.3436961Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_endpoint_operations.py", line 251, in begin_create_or_update
2023-12-20T04:48:32.3437589Z     raise ex
2023-12-20T04:48:32.3438281Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_endpoint_operations.py", line 24***, in begin_create_or_update
2023-12-20T04:48:32.3439079Z     poller = self._online_operation.begin_create_or_update(
2023-12-20T04:48:32.3439607Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3440329Z   File "/opt/az/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
2023-12-20T04:48:32.3440944Z     return func(*args, **kwargs)
2023-12-20T04:48:32.3441408Z            ^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3442206Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_restclient/v2***22_***2_***1_preview/operations/_online_endpoints_operations.py", line 921, in begin_create_or_update
2023-12-20T04:48:32.3443166Z     raw_result = self._create_or_update_initial(
2023-12-20T04:48:32.3443675Z                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-20T04:48:32.3444446Z   File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_restclient/v2***22_***2_***1_preview/operations/_online_endpoints_operations.py", line 855, in _create_or_update_initial
2023-12-20T04:48:32.3445326Z     map_error(status_code=response.status_code, response=response, error_map=error_map)
2023-12-20T04:48:32.3446136Z   File "/opt/az/lib/python3.11/site-packages/azure/core/exceptions.py", line 1***7, in map_error
2023-12-20T04:48:32.3446695Z     raise error
2023-12-20T04:48:32.3447271Z azure.core.exceptions.ResourceNotFoundError: (NotFound) The specified resource was not found.
2023-12-20T04:48:32.3447840Z Code: NotFound
2023-12-20T04:48:32.3448329Z Message: The specified resource was not found.
2023-12-20T04:48:32.3448935Z Exception Details:	(InferencingClientCallFailed) {{"title":"Not Found"}}
2023-12-20T04:48:32.3449549Z 	Code: InferencingClientCallFailed
2023-12-20T04:48:32.3450058Z 	Message: {{"title":"Not Found"}}
2023-12-20T04:48:32.3450262Z 
2023-12-20T04:48:32.3450790Z ERROR: cli.azure.cli.core.azclierror: (NotFound) The specified resource was not found.
2023-12-20T04:48:32.3451328Z Code: NotFound
2023-12-20T04:48:32.3451817Z Message: The specified resource was not found.
2023-12-20T04:48:32.3452435Z Exception Details:	(InferencingClientCallFailed) {{"title":"Not Found"}}
2023-12-20T04:48:32.3453007Z 	Code: InferencingClientCallFailed
2023-12-20T04:48:32.3453515Z 	Message: {{"title":"Not Found"}}
2023-12-20T04:48:32.3454075Z ERROR: az_command_data_logger: (NotFound) The specified resource was not found.
2023-12-20T04:48:32.3454756Z Code: NotFound
2023-12-20T04:48:32.3455259Z Message: The specified resource was not found.
2023-12-20T04:48:32.3455856Z Exception Details:	(InferencingClientCallFailed) {{"title":"Not Found"}}
2023-12-20T04:48:32.3456444Z 	Code: InferencingClientCallFailed
2023-12-20T04:48:32.3456944Z 	Message: {{"title":"Not Found"}}
2023-12-20T04:48:32.3457625Z DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at ***x7fc3611e***ea***>]
2023-12-20T04:48:32.3458372Z INFO: az_command_data_logger: exit code: 3
2023-12-20T04:48:32.3459096Z INFO: cli.__main__: Command ran in 13.948 seconds (init: ***.18***, invoke: 13.769)
2023-12-20T04:48:32.4143302Z INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
2023-12-20T04:48:32.4144241Z INFO: telemetry.client: Accumulated *** events. Flush the clients.
2023-12-20T04:48:32.4144906Z INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
2023-12-20T04:48:32.4149925Z INFO: telemetry.save: Save telemetry record of length 41***9 in cache
2023-12-20T04:48:32.4151054Z INFO: telemetry.main: Begin creating telemetry upload process.
2023-12-20T04:48:32.4155359Z INFO: telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /home/vsts/work/_temp/.azclitask"
2023-12-20T04:48:32.4157249Z INFO: telemetry.process: Return from creating process
2023-12-20T04:48:32.4158320Z INFO: telemetry.main: Finish creating telemetry upload process.
2023-12-20T04:48:32.9890779Z ##[error]Script failed with exit code: 3
2023-12-20T04:48:32.9910024Z [command]/usr/bin/az account clear
2023-12-20T04:48:33.6781580Z ##[section]Finishing: az_ml_endpoint_create_or_update

Issue script & Debug output

See above for detailed error message

kubernetes.yml used in above command:

$schema: https://azuremlschemas.azureedge.net/latest/kubernetesOnlineEndpoint.schema.json
name: nml-hydrocyclone-roping-qa
compute: azureml:****
auth_mode: key

Expected behavior

Create an endpoint in provided AML workspace

Environment Summary

$ az --version
azure-cli                         2.55.0

core                              2.55.0
telemetry                          1.1.0

Extensions:
ml                                2.22.0

Dependencies:
msal                            1.24.0b2
azure-mgmt-resource             23.1.0b2

Additional context

No response
@kmr0877 kmr0877 added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Dec 20, 2023
@yonzhan
Copy link
Collaborator

yonzhan commented Dec 20, 2023

Thank you for opening this issue, we will look into it.

@microsoft-github-policy-service microsoft-github-policy-service bot added question The issue doesn't require a change to the product in order to be resolved. Most issues start as that customer-reported Issues that are reported by GitHub users external to the Azure organization. Auto-Assign Auto assign by bot Service Attention This issue is responsible by Azure service team. Machine Learning extension/ml Subscription labels Dec 20, 2023
@yonzhan yonzhan removed bug This issue requires a change to an existing behavior in the product in order to be resolved. Subscription labels Dec 20, 2023
@wangzelin007 wangzelin007 removed the Service Attention This issue is responsible by Azure service team. label Dec 26, 2023
@microsoft-github-policy-service microsoft-github-policy-service bot added the needs-team-triage This issue needs the team to triage. label Dec 26, 2023
@wangzelin007 wangzelin007 added Service Attention This issue is responsible by Azure service team. and removed needs-team-triage This issue needs the team to triage. labels Dec 26, 2023
@cassiebreviu
Copy link

I had this issue and it ended up being a soft delete issue on the endpoint name. I updated the endpoint name and then it worked. I think the error message needs to be updated.

@MakarandBatchu
Copy link

MakarandBatchu commented Jan 31, 2024 •
edited
Loading

I am also facing the same issue. I am repeatedly getting 'No Such Endpoint' error even after updating the endpoint name.

error - 'There is no endpoint with name: platform-azureml-ad-te in mlw-platform-azureml (Code: NoSuchEndpoint)'

Please note that there is no existing endpoint present and I am trying to create a new endpoint here.

@anshgandhi
Copy link

@cassiebreviu what helped you work around this issue?
I created an endpoint with the name 'endpoint-1', deleted it using the AI Studio UI. Then, trying to re-create the endpoint using the cli and and I am getting this error :/

@MakarandBatchu
Copy link

Hi @anshgandhi. The only thing that seemed to work for me is to change the name of the endpoint and recreate it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Auto-Assign Auto assign by bot customer-reported Issues that are reported by GitHub users external to the Azure organization. extension/ml Machine Learning question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention This issue is responsible by Azure service team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@anshgandhi @zhoxing-ms @wangzelin007 @kmr0877 @cassiebreviu @yonzhan @MakarandBatchu