When using podman, the configuration obtained using get kubeconfig cannot be used

[kebe7jun]

What happened:

root@kebe-home:~# kind get kubeconfig --name my-cluster-installer > /tmp/x
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
root@kebe-home:~# kubectl --kubeconfig /tmp/x get no
E1106 01:55:36.092466 2212989 memcache.go:265] couldn't get current server API group list: Get "https://:16443/api?timeout=32s": tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config
E1106 01:55:36.093429 2212989 memcache.go:265] couldn't get current server API group list: Get "https://:16443/api?timeout=32s": tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config
E1106 01:55:36.095009 2212989 memcache.go:265] couldn't get current server API group list: Get "https://:16443/api?timeout=32s": tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config
E1106 01:55:36.095988 2212989 memcache.go:265] couldn't get current server API group list: Get "https://:16443/api?timeout=32s": tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config
E1106 01:55:36.097525 2212989 memcache.go:265] couldn't get current server API group list: Get "https://:16443/api?timeout=32s": tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config
Unable to connect to the server: tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config
root@kebe-home:~# cat /tmp/x
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRU...
    server: https://:16443
  name: kind-my-cluster-installer
contexts:
- context:
    cluster: kind-my-cluster-installer
    user: kind-my-cluster-installer
  name: kind-my-cluster-installer
current-context: kind-my-cluster-installer
...

What you expected to happen:

kubectl command does not report an error.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• kind version: (use kind version): kind v0.19.0 go1.20.4 linux/amd64
• Runtime info: (use docker info, podman info or nerdctl info):

host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/local/lib/podman/conmon
    version: 'conmon version 2.1.10, commit: 2dcd736e46ded79a53339462bc251694b150f870'
  cpuUtilization:
    idlePercent: 95.69
    systemPercent: 1.18
    userPercent: 3.13
  cpus: 224
  databaseBackend: sqlite
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: file
  freeLocks: 2046
  hostname: cuke-metax-demo-001
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.0-112-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 2075498274816
  memTotal: 2164076400640
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
  ociRuntime:
    name: /opt/dce5/bin/runc
    package: Unknown
    path: /opt/dce5/bin/runc
    version: |-
      runc version 1.1.12
      commit: v1.1.12-0-g51d5e946
      spec: 1.0.2-dev
      go: go1.20.13
      libseccomp: 2.5.4
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/local/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 16h 25m 47.00s (Approximately 0.67 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "true"
    overlay.mount_program:
      Executable: /usr/local/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fuse-overlayfs: version 1.13-dev
        fusermount3 version: 3.10.5
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 475519213568
  graphRootUsed: 158235852800
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /var/run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.9.4
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.20.14
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4

• OS (e.g. from /etc/os-release): ubuntu
• Kubernetes version: (use kubectl version):
• Any proxies or other special environment settings?:

[aojea]

kind v0.19.0

is this happening with the latest kind version? that is an old one

[kebe7jun]

kind v0.19.0

is this happening with the latest kind version? that is an old one

The latest version also has this problem because podman does not return the correct HostIP.

[aojea]

The latest version also has this problem because podman does not return the correct HostIP.

we parse the information here

kind/pkg/cluster/internal/providers/podman/provider.go

Lines 192 to 232 in 1a8f047

	// podman inspect was broken between 2.2.0 and 3.0.0
	// https://github.com/containers/podman/issues/8444
	if v.AtLeast(version.MustParseSemantic("2.2.0")) &&
	v.LessThan(version.MustParseSemantic("3.0.0")) {
	p.logger.Warnf("WARNING: podman version %s not fully supported, please use versions 3.0.0+")
	cmd := exec.Command(
	"podman", "inspect",
	"--format",
	"{{range .NetworkSettings.Ports }}{{range .}}{{.HostIP}}/{{.HostPort}}{{end}}{{end}}",
	n.String(),
	)
	lines, err := exec.OutputLines(cmd)
	if err != nil {
	return "", errors.Wrap(err, "failed to get api server port")
	}
	if len(lines) != 1 {
	return "", errors.Errorf("network details should only be one line, got %d lines", len(lines))
	}
	// output is in the format IP/Port
	parts := strings.Split(strings.TrimSpace(lines[0]), "/")
	if len(parts) != 2 {
	return "", errors.Errorf("network details should be in the format IP/Port, received: %s", parts)
	}
	host := parts[0]
	port, err := strconv.Atoi(parts[1])
	if err != nil {
	return "", errors.Errorf("network port not an integer: %v", err)
	}
	return net.JoinHostPort(host, strconv.Itoa(port)), nil
	}
	cmd := exec.Command(
	"podman", "inspect",
	"--format",
	"{{ json .NetworkSettings.Ports }}",
	n.String(),
	)
	lines, err := exec.OutputLines(cmd)

as you can see it was broken between 2.2.0 and 3.0.0

What podman version are you using?
Can you please try manually the podman inspect commands to fix the root cause? left a comment in the PR about it https://github.com/kubernetes-sigs/kind/pull/3778/files#r1830927118

[stmcginnis]

Also, you skipped over adding the podman info output from the issue template. If you could add that, it may be helpful to see what podman is reporting.

[BenTheElder]

+1 to: can we fix the podman inspect command? We may have to change it, but there should be some way to get the correct info.

can you share something like podman inspect kind-control-plane so we can figure out what the new format is? we also need the info like podman info as mentioned above.

[kebe7jun]

The output of podman info has been updated to the description.

The result of podman inspect my-cluster-installer-control-plane

root@kebe-home:~# podman inspect  my-cluster-installer-control-plane
[
     {
....
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "",
               "IPAddress": "",
               "IPPrefixLen": 0,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {
                    "6443/tcp": [
                         {
                              "HostIp": "",
                              "HostPort": "16443"
                         }
                    ]
               },
               "SandboxKey": "/run/netns/netns-0d88ad30-3910-0ae2-5d20-151c04c25749",
               "Networks": {
                    "kind": {
                         "EndpointID": "",
                         "Gateway": "10.89.0.1",
                         "IPAddress": "10.89.0.14",
                         "IPPrefixLen": 24,
                         "IPv6Gateway": "fc00:f853:ccd:e793::1",
                         "GlobalIPv6Address": "fc00:f853:ccd:e793::e",
                         "GlobalIPv6PrefixLen": 64,
                         "MacAddress": "7a:ae:9b:9d:ff:62",
                         "NetworkID": "kind",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "3f0b231be90e",
                              "my-cluster-installer-control-plane"
                         ]
                    }
               }
          },
       ...
     }
]

[aojea]

         "Ports": {
                    "6443/tcp": [
                         {
                              "HostIp": "",
                              "HostPort": "16443"
                         }
                    ]
               },

Ok, so it seems we need to default it and your patch is correct

[stmcginnis]

It may be worth opening an issue with podman to point out this difference from docker.

[BenTheElder]

I agree, we should open an issue and ask if there's an alternate way to get this information. 127.0.0.1 is just an assumption, as much as we assumed it would continue returning this field.