[WinError 2] The system cannot find the file specified on connectedk8s\azext_connectedk8s\custom.py

[mahomedalid]

Describe the bug

While following the tutorial https://azurearcjumpstart.io/azure_arc_jumpstart/azure_arc_k8s/aks_hybrid/aks_edge_essentials_single/ we got the error " [WinError 2] The system cannot find the file specified " File "C:\Users\akseeadmin.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 2434

After some troubleshooting we found out the extension was looking for the kubectl client in a folder inside .azure/cliextensions/connectedk8s/kubectl_client but kubectl was not there, but in other location installed by chocolatey. The helm binary on the other hand was there, so my wild guess is that kubectl was not downloaded because it was already detecte, but when trying to use it, the extension accessed the wrong location.

Related command

.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py

Errors

"[WinError 2] The system cannot find the file specified " File "C:\Users\akseeadmin.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 2434

Issue script & Debug output

The required pre-checks for onboarding have succeeded.
The command failed with an unexpected error. Here is the traceback:
[WinError 2] The system cannot find the file specified
Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line
663, in execute
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line
726, in _run_jobs_serially
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line
697, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line
333, in call
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.p
y", line 121, in handler
File "C:\Users\akseeadmin.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 336, in create_connect
edk8s
crd_cleanup_force_delete(kubectl_client_location, kube_config, kube_context)
File "C:\Users\akseeadmin.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 2434, in crd_cleanup_f
orce_delete
response_helm_delete = Popen(cmd_helm_delete, stdout=PIPE, stderr=PIPE)
File "subprocess.py", line 971, in init
File "subprocess.py", line 1440, in _execute_child
FileNotFoundError: [WinError 2] The system cannot find the file specified
To check existing issues, please visit: https://github.com/Azure/azure-cli/issues

Expected behavior

No errors

Environment Summary

azure-cli 2.50.0

core 2.50.0
telemetry 1.0.8

Dependencies:
msal 1.22.0
azure-mgmt-resource 23.1.0b2

WindowsServer sku= 2022-datacenter-g2
Python (Windows) 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:05:00) [MSC v.1929 32 bit (Intel)]
AksEdge version : 1.2.414.0

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[ghost]

Thank you for your feedback. This has been routed to the support team for assistance.

[navba-MSFT]

@mahomedalid Thanks for reaching out to us and reporting this issue. You should set the environment variables for the kubectl.exe and your kube config file.

If you are on Windows box, then add kubectl.exe folder location in path variable - “Advanced System Settings -> Advanced -> Environment Variables -> PATH”. Example: if you have saved file to C:/kubectl then add this path to the PATH env variable.

Also by default, kubectl looks for a file named config in the $HOME/.kube directory. You can specify other kubeconfig files by setting the KUBECONFIG environment variable. This should address the above exception.

[lanicolas]

Hello @navba-MSFT

I am experiencing the same issue and I followed what you've shared and it does not fix the exception.

I installed kubectl via chocolatey so the path is:

Get-Command kubectl

CommandType Name Version Source

---

Application kubectl.exe 1.0.0.0 C:\ProgramData\chocolatey\bin\kubectl.exe

I have added that to my path environment variable:

$env:Path
C:\ProgramData\chocolatey\bin\kubectl.exe;C:\ProgramData\chocolatey\bin;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\dotnet;C:\ProgramData\chocolatey\bin;C:\Program Files\AksEdge\kubectl;C:\Users\arcdemo\AppData\Local\Microsoft\WindowsApps;

As well as I have set the KUBECONFIG environment variable:

$env:KUBECONFIG
C:\Users\arcdemo.kube\config

I still hit the issue:

This operation might take a while...

Downloading kubectl client for first time. This can take few minutes...

An exception has occured while saving the cluster diagnostic checks pod description in the local machine. Exception: [WinError 2] The system cannot find the file specified

The required pre-checks for onboarding have succeeded.
The command failed with an unexpected error. Here is the traceback:
[WinError 2] The system cannot find the file specified
Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 663, in execute
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 697, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 333, in call
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "C:\Users\arcdemo.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 336, in create_connectedk8s
crd_cleanup_force_delete(kubectl_client_location, kube_config, kube_context)
File "C:\Users\arcdemo.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 2434, in crd_cleanup_force_delete
response_helm_delete = Popen(cmd_helm_delete, stdout=PIPE, stderr=PIPE)
File "subprocess.py", line 971, in init
File "subprocess.py", line 1440, in _execute_child
FileNotFoundError: [WinError 2] The system cannot find the file specified

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @akashkeshari.

Issue Details

---

Describe the bug

While following the tutorial https://azurearcjumpstart.io/azure_arc_jumpstart/azure_arc_k8s/aks_hybrid/aks_edge_essentials_single/ we got the error " [WinError 2] The system cannot find the file specified " File "C:\Users\akseeadmin.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 2434

After some troubleshooting we found out the extension was looking for the kubectl client in a folder inside .azure/cliextensions/connectedk8s/kubectl_client but kubectl was not there, but in other location installed by chocolatey. The helm binary on the other hand was there, so my wild guess is that kubectl was not downloaded because it was already detecte, but when trying to use it, the extension accessed the wrong location.

Related command

.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py

Errors

"[WinError 2] The system cannot find the file specified " File "C:\Users\akseeadmin.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 2434

Issue script & Debug output

The required pre-checks for onboarding have succeeded.
The command failed with an unexpected error. Here is the traceback:
[WinError 2] The system cannot find the file specified
Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line
663, in execute
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line
726, in _run_jobs_serially
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line
697, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line
333, in call
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.p
y", line 121, in handler
File "C:\Users\akseeadmin.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 336, in create_connect
edk8s
crd_cleanup_force_delete(kubectl_client_location, kube_config, kube_context)
File "C:\Users\akseeadmin.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 2434, in crd_cleanup_f
orce_delete
response_helm_delete = Popen(cmd_helm_delete, stdout=PIPE, stderr=PIPE)
File "subprocess.py", line 971, in init
File "subprocess.py", line 1440, in _execute_child
FileNotFoundError: [WinError 2] The system cannot find the file specified
To check existing issues, please visit: https://github.com/Azure/azure-cli/issues

Expected behavior

No errors

Environment Summary

azure-cli 2.50.0

core 2.50.0
telemetry 1.0.8

Dependencies:
msal 1.22.0
azure-mgmt-resource 23.1.0b2

WindowsServer sku= 2022-datacenter-g2
Python (Windows) 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:05:00) [MSC v.1929 32 bit (Intel)]
AksEdge version : 1.2.414.0

Additional context

No response

Author:	mahomedalid
Assignees:	-
Labels:	bug, Service Attention, needs-team-attention, Connected Kubernetes
Milestone:	-

[navba-MSFT]

Adding Service team to look into this.

@akashkeshari Please look into this and assist once you get a chance. Thanks in advance.

[YRollHid]

Hello @navba-MSFT,

Indeed, I can confirm @lanicolas post cause I'm having the same issue, testing azure_arc_jumpstart (AKS Edge Essentials) scenario:

An exception has occurred while saving the cluster diagnostic checks pod description in the local machine. Exception: [WinError 2] The system cannot find the file specified

The required pre-checks for onboarding have succeeded.
The command failed with an unexpected error. Here is the traceback:
[WinError 2] The system cannot find the file specified
Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 663, in execute
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 697, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 333, in call
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "C:\Users\arcdemo.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 336, in create_connectedk8s
crd_cleanup_force_delete(kubectl_client_location, kube_config, kube_context)
File "C:\Users\arcdemo.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 2434, in crd_cleanup_force_delete
response_helm_delete = Popen(cmd_helm_delete, stdout=PIPE, stderr=PIPE)
File "subprocess.py", line 971, in init
File "subprocess.py", line 1440, in _execute_child
FileNotFoundError: [WinError 2] The system cannot find the file specified
To check existing issues, please visit: https://github.com/Azure/azure-cli/issues

[mahomedalid]

@lanicolas I don't have access to that VM anymore but I think it was /.azure/kubectl-client

@navba-MSFT @akashkeshari I do see a difference in the custom.py between install_kubectl_client and install_helm_client.

install_helm_client checks the env variable os.getenv('HELM_CLIENT_PATH') and if exists it just returns that. If not, it downloads the binary and puts that in the .azure/... location.

On the other hand install_kubectl_client does not check the env variable where kubectl could be installed, instead it expects kubectl to be in the .azure/kubectl-client location, which should not be a problem to have two different kubectl binaries. However, to install the client it does not directly download the binary, but relies on the aks install-cli --install-location command. I am not sure, but my guess is that aks install-cli does check if kubectl is installed, and because it is already installed by chocolatey in other location it simply does not do anything. install_kubectl_client returns the "optimistic" location (without checking if the binary is there, and that is where it could fail.

[BillmanH]

Exact same issue. Setting environment variable doesn't seem to help.

az connectedk8s connect -n $ClusterName -l $Location -g $ResourceGroupName --subscription $SubscriptionId --tags $tags | Out-Null

Fallowing the docs from here:
Quickstart: Deploy Azure IoT Operations to an Arc-enabled Kubernetes cluster

[vyta]

Similar issue here: Azure/azure-cli-extensions#6637

The following resolved it for me:

Invoke-WebRequest -Uri https://secure.globalsign.net/cacert/Root-R1.crt -OutFile c:\globalsignR1.crt'
Import-Certificate -FilePath c:\globalsignR1.crt -CertStoreLocation Cert:\LocalMachine\Root'
# insert az command here

[BillmanH]

For my solution, the end result was just to remove k8s altogether and then allow the installer.ps1 file to install it where it expects to have it. It seems to need to install Kubernetes every time regardless so just let it be in charge.

When you consider that this is an edge iot solution that shouldn't have a lot of other applications on it I think this is a fair way to go.