use MSI identity to login for az cli when useManagedIdentityExtension is set to true and aadClientSecret is not provided

[chaudhryfaisal]

RKE version:
1.2.3

Docker version: (docker version,docker info preferred)

Client: Docker Engine - Community
 Version:           19.03.14
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        5eb3275d40
 Built:             Tue Dec  1 19:20:42 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.14
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       5eb3275d40
  Built:            Tue Dec  1 19:19:17 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
Azure
cluster.yml file:

cloud_provider:
  azureCloudProvider:
    subscriptionId: subscriptionId
    tenantId: tenantId
    vmType: vmss
    useManagedIdentityExtension: true
  name: azure
cluster_name: cluster_name
ignore_docker_version: true
ingress:
  provider: nginx
kubernetes_version: v1.18.12-rancher1-1
network:
  plugin: calico
nodes:
- address: IP_ADDRESS
  hostname_override: vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000
  role:
  - controlplane
  - etcd
  - worker
  user: azureuser

Steps to Reproduce:

rke up

Results:

rke-tools seems to always try to login using aadClientId and aadClientSecret however when aadClientSecret is not provided and useManagedIdentityExtension is set to true then it should use MSI identity to login for az cli

rke up logs

time="2021-01-13T15:18:43Z" level=info msg="[sync] Syncing nodes Labels and Taints"
time="2021-01-13T15:18:43Z" level=debug msg="worker [9] starting sync for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000]"
time="2021-01-13T15:18:43Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #1"
time="2021-01-13T15:18:48Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #2"
time="2021-01-13T15:18:53Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #3"
time="2021-01-13T15:18:58Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #4"
time="2021-01-13T15:19:03Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #5"
time="2021-01-13T15:19:08Z" level=debug msg="[hosts] Can't find node by name [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], error:  \"vmss-k8s-cluster-demo-01-single-node-dev-
eastus-fic1000000\" not found"
time="2021-01-13T15:19:10Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #1"
time="2021-01-13T15:19:15Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #2"
time="2021-01-13T15:19:20Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #3"
time="2021-01-13T15:19:25Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #4"
time="2021-01-13T15:19:30Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #5"
time="2021-01-13T15:19:35Z" level=debug msg="[hosts] Can't find node by name [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], error:  \"vmss-k8s-cluster-demo-01-single-node-dev-
eastus-fic1000000\" not found"
time="2021-01-13T15:19:37Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #1"
time="2021-01-13T15:19:42Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #2"
time="2021-01-13T15:19:47Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #3"
time="2021-01-13T15:19:52Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #4"
time="2021-01-13T15:19:57Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #5"
time="2021-01-13T15:20:02Z" level=debug msg="[hosts] Can't find node by name [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], error:  \"vmss-k8s-cluster-demo-01-single-node-dev-
eastus-fic1000000\" not found"
time="2021-01-13T15:20:04Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #1"
time="2021-01-13T15:20:09Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #2"
time="2021-01-13T15:20:14Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #3"
time="2021-01-13T15:20:19Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #4"
time="2021-01-13T15:20:24Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #5"
time="2021-01-13T15:20:29Z" level=debug msg="[hosts] Can't find node by name [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], error:  \"vmss-k8s-cluster-demo-01-single-node-dev-
eastus-fic1000000\" not found"
time="2021-01-13T15:20:31Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #1"
time="2021-01-13T15:20:36Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #2"
time="2021-01-13T15:20:41Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #3"
time="2021-01-13T15:20:46Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #4"
time="2021-01-13T15:20:51Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #5"
time="2021-01-13T15:20:56Z" level=debug msg="[hosts] Can't find node by name [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], error:  \"vmss-k8s-cluster-demo-01-single-node-dev-
eastus-fic1000000\" not found"
time="2021-01-13T15:20:58Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #1"
time="2021-01-13T15:21:03Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #2"
time="2021-01-13T15:21:08Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #3"
time="2021-01-13T15:21:13Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #4"
time="2021-01-13T15:21:18Z" level=debug msg="Checking node list for node [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], try #5"
time="2021-01-13T15:21:23Z" level=debug msg="[hosts] Can't find node by name [vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000], error:  \"vmss-k8s-cluster-demo-01-single-node-dev-
eastus-fic1000000\" not found"
time="2021-01-13T15:21:25Z" level=fatal msg="[ \"vmss-k8s-cluster-demo-01-single-node-dev-eastus-fic1000000\" not found]"

suspected root cause from kubelet, kube-controller-manager and kube-apiserver container logs

 kubelet.log:Some variables were not populated correctly, using the passed config!
 kubelet.log:Some variables were not populated correctly, using the passed config!
 kubelet.log:Some variables were not populated correctly, using the passed config!
 kubelet.log:Some variables were not populated correctly, using the passed config!
 kubelet.log:Some variables were not populated correctly, using the passed config!
 kubelet.log:Some variables were not populated correctly, using the passed config!
 kubelet.log:Some variables were not populated correctly, using the passed config!
 kubelet.log:Some variables were not populated correctly, using the passed config!
 kubelet.log:Some variables were not populated correctly, using the passed config!
 kubelet.log:Some variables were not populated correctly, using the passed config!
 kube-controller-manager.log:Some variables were not populated correctly, using the passed config!
 kube-apiserver.log:Some variables were not populated correctly, using the passed config!

proposed fix
rancher/rke-tools#116

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.