AZ CLI Connectedk8s Error crd_cleanup_force_delete(kubectl_client_location, kube_config, kube_context)

[msuleman92]

Describe the bug

Hi,

While trying to connect my Kubernetes cluster to azure arc is shows the errors related to custom.py.

Related command

az connectedk8s connect -g ARC-AKS-MicroK8 -n Arc-Data-Microk8s --kube-config "C:\Users\user.kube\config" --kube-context $Env:KUBECONTEXT --correlation-id "d009f5dd-dba8-4ac7-bac9-b54ef3a6671a"

Errors

The command failed with an unexpected error. Here is the traceback:
[WinError 2] The system cannot find the file specified
Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 663, in execute
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 697, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 333, in call
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "C:\Users\arcdemo.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 334, in create_connectedk8s
crd_cleanup_force_delete(kubectl_client_location, kube_config, kube_context)
File "C:\Users\arcdemo.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 2410, in crd_cleanup_force_delete
response_helm_delete = Popen(cmd_helm_delete, stdout=PIPE, stderr=PIPE)
File "subprocess.py", line 971, in init
File "subprocess.py", line 1440, in _execute_child
FileNotFoundError: [WinError 2] The system cannot find the file specified

Issue script & Debug output

Debug AZ CLI Connectedk8s.txt

Expected behavior

Should be Successful.

Environment Summary

"azure-cli": "2.51.0",
"azure-cli-core": "2.51.0",
"azure-cli-telemetry": "1.1.0",
"extensions": {
"arcdata": "1.5.4",
"connectedk8s": "1.4.0",
"k8s-extension": "1.4.2"

Microk8s Running on Ubuntu machine and managed with Windows VM.

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[msuleman92]

Any help on this?

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @akashkeshari.

[jsntcy]

@rohan-dassani, @TheOnlyWei, could you please help take a look at this issue?

[rohan-dassani]

@msuleman92 , seems like there is some issue with the kubectl installation

can you please cd to the directory - $HOME/.azure/kubectl-client and use the kubectl there and run the command
./kubectl.exe config get-contexts ---> (Windows machine) or
./kubectl config get-contexts ---> (Linux machine)

Can you delete the kubectl file in this location and try onboarding again

Also if the kubectl download fails again , can you run this command for manual kubectl installation
az aks install-cli --install-location
should be like below , change the $HOME according to you
C:\Users\rohandassani.azure\kubectl-client\kubectl.exe ---> (Windows)
C:\Users\rohandassani.azure\kubectl-client\kubectl ---> (Linux)

[msuleman92]

I don't see anything inside the folder.

[rohan-dassani]

@msuleman92 , the connectedk8s extension internally installs kubectl client and places it in this folder, seems like that installation has failed.

Can you try installing kubctl client manually to unblock you

az aks install-cli --install-location
should be like below, change it according to you,
C:\Users\rohandassani.azure\kubectl-client\kubectl.exe ---> (Windows)
C:\Users\rohandassani.azure\kubectl-client\kubectl ---> (Linux)

[msuleman92]

I See this error when i try to run the install command.

[israelvaldez]

Here's a text version of the above that we're getting as well, this is a pain in new VMs where we try to use to run, on old VMs it works fine.

2023-09-10T04:48:10.1180538Z WARNING: The detected architecture of current device is "amd64", and the binary for "amd64" will be downloaded. If the detectiton is wrong, please download and install the binary corresponding to the appropriate architecture.
2023-09-10T04:48:10.1182445Z WARNING: No version specified, will get the latest version of kubectl from "https://storage.googleapis.com/kubernetes-release/release/stable.txt"
2023-09-10T04:48:11.0962576Z ERROR: The command failed with an unexpected error. Here is the traceback:
2023-09-10T04:48:11.0970617Z ERROR: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
2023-09-10T04:48:11.0971010Z Traceback (most recent call last):
2023-09-10T04:48:11.0971301Z   File "urllib\request.py", line 1348, in do_open
2023-09-10T04:48:11.0971602Z   File "http\client.py", line 1282, in request
2023-09-10T04:48:11.0971878Z   File "http\client.py", line 1328, in _send_request
2023-09-10T04:48:11.0972153Z   File "http\client.py", line 1277, in endheaders
2023-09-10T04:48:11.0972412Z   File "http\client.py", line 1037, in _send_output
2023-09-10T04:48:11.0972670Z   File "http\client.py", line 975, in send
2023-09-10T04:48:11.0974077Z   File "http\client.py", line 1454, in connect
2023-09-10T04:48:11.0974320Z   File "ssl.py", line 513, in wrap_socket
2023-09-10T04:48:11.0974561Z   File "ssl.py", line 1071, in _create
2023-09-10T04:48:11.0974796Z   File "ssl.py", line 1342, in do_handshake
2023-09-10T04:48:11.0975149Z ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
2023-09-10T04:48:11.0975353Z 
2023-09-10T04:48:11.0975627Z During handling of the above exception, another exception occurred:
2023-09-10T04:48:11.0976094Z 
2023-09-10T04:48:11.0976302Z Traceback (most recent call last):
2023-09-10T04:48:11.0976950Z   File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
2023-09-10T04:48:11.0977509Z   File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 663, in execute
2023-09-10T04:48:11.0978090Z   File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
2023-09-10T04:48:11.0978695Z   File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 697, in _run_job
2023-09-10T04:48:11.0979248Z   File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 333, in __call__
2023-09-10T04:48:11.0979759Z   File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
2023-09-10T04:48:11.0980264Z   File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/acs/custom.py", line 1475, in k8s_install_cli
2023-09-10T04:48:11.0980714Z   File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/acs/custom.py", line 1619, in k8s_install_kubectl
2023-09-10T04:48:11.0980904Z   File "urllib\request.py", line 216, in urlopen
2023-09-10T04:48:11.0981027Z   File "urllib\request.py", line 519, in open
2023-09-10T04:48:11.0981156Z   File "urllib\request.py", line 536, in _open
2023-09-10T04:48:11.0981286Z   File "urllib\request.py", line 496, in _call_chain
2023-09-10T04:48:11.0981423Z   File "urllib\request.py", line 1391, in https_open
2023-09-10T04:48:11.0981554Z   File "urllib\request.py", line 1351, in do_open
2023-09-10T04:48:11.0981751Z urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
2023-09-10T04:48:11.0985454Z To check existing issues, please visit: https://github.com/Azure/azure-cli/issues
2023-09-10T04:48:11.9529397Z 
2023-09-10T04:48:11.9531604Z Success       : True
2023-09-10T04:48:11.9532415Z RestartNeeded : No
2023-09-10T04:48:11.9532705Z FeatureResult : {}
2023-09-10T04:48:11.9533188Z ExitCode      : NoChangeNeeded

[rohan-dassani]

Seems like a proxy issue in your environment.
This is because the proxy intercepts Azure CLI's HTTPS traffic, decrypts and re-encrypts it with its own certificate, but the proxy can't return all intermediate and root CA certificates, leading to the failure.

Please refer to this older github issue for help - Azure/azure-cli#19571

@yonzhan to help here with this issue since CLI team has resolved very similar issues in the past.

[vyta]

This might be related to Azure/azure-cli#27863

As per the last comment in that issue, the following resolved the issue for me:

Invoke-WebRequest -Uri https://secure.globalsign.net/cacert/Root-R1.crt -OutFile c:\globalsignR1.crt'
Import-Certificate -FilePath c:\globalsignR1.crt -CertStoreLocation Cert:\LocalMachine\Root'
# insert az command here