k3d With Rootless Podman

[hadrabap]

Hello friends,

I would like to give a try to k3s, but first I need to do my own homework for KVM/libvirt. In the meantime I found k3d, which should allow me to try k3s quite smoothly as my harness around containers is up-and-running (I did my homework there already).

The only issue are my limitations given by my Linux distribution. I'm using Oracle Linux 8 (RHEL 8) with Podman. All software is stock. The only exception is, that I've switched from CGroups v1 to CGroups v2 which works great with other containers; no issue here. The system also uses SELinux in enforcing mode. There are no denials or violations reported, no AVCs etc. No problem here as well.

The problem is that k3d is unable to spin-up the cluster. I already followed the documentation so I've created a k3d-podman wrapper:

#!/bin/bash

DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock \
DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock        \
k3d "$@"

Unfortunately the k3d-podman create cluster freezes with the last message

Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap... 

When I look into logs of the k3d-k3s-default-server-0 container, I see lots of

The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

at the end.

It looks like a firewall issue?

I also tried to find the process of 6443 port with no luck:

[opc@sws ~]$ podman exec -ti --user 0 k3d-k3s-default-server-0 sh
/ # netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
/ # lsof
1	/run/podman-init	0	/dev/null
1	/run/podman-init	1	pipe:[645278]
1	/run/podman-init	2	pipe:[645279]
1	/run/podman-init	3	anon_inode:[signalfd]
8	/bin/busybox	0	/dev/null
8	/bin/busybox	1	pipe:[645278]
8	/bin/busybox	2	pipe:[645279]
8	/bin/busybox	10	/bin/k3d-entrypoint.sh
2242	/bin/busybox	0	/dev/pts/0
2242	/bin/busybox	1	/dev/pts/0
2242	/bin/busybox	2	/dev/pts/0
2242	/bin/busybox	10	/dev/tty
2302	/bin/coreutils	0	/dev/null
2302	/bin/coreutils	1	pipe:[645278]
2302	/bin/coreutils	2	pipe:[645279]
/ # 

So, I'm really lost here. Could you please point me to the right direction?

Thanks in advance.

---

k3d create cluster --trace logs:

[opc@sws ~]$ k3d-podman cluster create --trace
DEBU[0000] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/run/user/1000/podman/podman.sock Version:4.2.0 OSType:linux OS:"ol" Arch:amd64 CgroupVersion:2 CgroupDriver:systemd Filesystem:xfs InfoName:sws.<REDACTED>} 
DEBU[0000] Additional CLI Configuration:
cli:
  api-port: ""
  env: []
  k3s-node-labels: []
  k3sargs: []
  ports: []
  registries:
    create: ""
  runtime-labels: []
  volumes: []
hostaliases: [] 
DEBU[0000] Configuration:
agents: 0
image: docker.io/rancher/k3s:v1.25.7-k3s1
network: ""
options:
  k3d:
    disableimagevolume: false
    disableloadbalancer: false
    disablerollback: false
    loadbalancer:
      configoverrides: []
    timeout: 0s
    wait: true
  kubeconfig:
    switchcurrentcontext: true
    updatedefaultkubeconfig: true
  runtime:
    agentsmemory: ""
    gpurequest: ""
    hostpidmode: false
    serversmemory: ""
registries:
  config: ""
  use: []
servers: 1
subnet: ""
token: "" 
TRAC[0000] Trying to read config apiVersion='k3d.io/v1alpha4', kind='simple' 
DEBU[0000] ========== Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha4} ObjectMeta:{Name:} Servers:1 Agents:0 ExposeAPI:{Host: HostIP: HostPort:} Image:docker.io/rancher/k3s:v1.25.7-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
========================== 
TRAC[0000] VolumeFilterMap: map[]                       
TRAC[0000] PortFilterMap: map[]                         
TRAC[0000] K3sNodeLabelFilterMap: map[]                 
TRAC[0000] RuntimeLabelFilterMap: map[]                 
TRAC[0000] EnvFilterMap: map[]                          
DEBU[0000] ========== Merged Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha4} ObjectMeta:{Name:} Servers:1 Agents:0 ExposeAPI:{Host: HostIP: HostPort:46789} Image:docker.io/rancher/k3s:v1.25.7-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory: HostPidMode:false Labels:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
========================== 
DEBU[0000] generated loadbalancer config:
ports:
  6443.tcp:
  - k3d-k3s-default-server-0
settings:
  workerConnections: 1024 
DEBU[0000] ===== Merged Cluster Config =====
&{TypeMeta:{Kind: APIVersion:} Cluster:{Name:k3s-default Network:{Name:k3d-k3s-default ID: External:false IPAM:{IPPrefix:zero IPPrefix IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0xc0004a76c0 0xc0004a7d40] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0xc000395b80 ServerLoadBalancer:0xc00084e9a0 ImageVolume: Volumes:[]} ClusterCreateOpts:{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d] GlobalEnv:[] HostAliases:[] Registries:{Create:<nil> Use:[] Config:<nil>}} KubeconfigOpts:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true}}
===== ===== ===== 
DEBU[0000] '--kubeconfig-update-default set: enabling wait-for-server 
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-k3s-default'            
INFO[0000] Created image volume k3d-k3s-default-images  
TRAC[0000] Using Registries: []                         
TRAC[0000] 
===== Creating Cluster =====

Runtime:
{}

Cluster:
&{Name:k3s-default Network:{Name:k3d-k3s-default ID:89a5dde53e7c97671dfc4c2ede2d906feeac60b2bad51490f5683f379b649776 External:false IPAM:{IPPrefix:10.89.0.0/24 IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0xc0004a76c0 0xc0004a7d40] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0xc000395b80 ServerLoadBalancer:0xc00084e9a0 ImageVolume:k3d-k3s-default-images Volumes:[k3d-k3s-default-images]}

ClusterCreatOpts:
&{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory: NodeHooks:[] GlobalLabels:map[app:k3d k3d.cluster.imageVolume:k3d-k3s-default-images k3d.cluster.network:k3d-k3s-default k3d.cluster.network.external:false k3d.cluster.network.id:89a5dde53e7c97671dfc4c2ede2d906feeac60b2bad51490f5683f379b649776 k3d.cluster.network.iprange:10.89.0.0/24] GlobalEnv:[] HostAliases:[] Registries:{Create:<nil> Use:[] Config:<nil>}}

============================
	 
TRAC[0000] Docker Machine not specified via DOCKER_MACHINE_NAME env var 
TRAC[0000] [Docker] Not using docker-machine            
DEBU[0000] [Docker] DockerHost: '' (unix:///run/user/1000/podman/podman.sock) 
INFO[0000] Starting new tools node...                   
DEBU[0000] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
DEBU[0000] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
DEBU[0000] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
DEBU[0000] Detected CgroupV2, enabling custom entrypoint (disable by setting K3D_FIX_CGROUPV2=false) 
TRAC[0000] Creating node from spec
&{Name:k3d-k3s-default-tools Role:noRole Image:ghcr.io/k3d-io/k3d-tools:5.4.9 Volumes:[k3d-k3s-default-images:/k3d/images /run/user/1000/podman/podman.sock:/run/user/1000/podman/podman.sock] Env:[] Cmd:[] Args:[noop] Ports:map[] Restart:false Created: HostPidMode:false RuntimeLabels:map[app:k3d k3d.cluster:k3s-default k3d.version:v5.4.9] K3sNodeLabels:map[] Networks:[k3d-k3s-default] ExtraHosts:[host.k3d.internal:host-gateway] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:zero IP Static:false} HookActions:[]} 
TRAC[0000] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-k3s-default-tools Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[noop] Healthcheck:<nil> ArgsEscaped:false Image:ghcr.io/k3d-io/k3d-tools:5.4.9 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:k3s-default k3d.role:noRole k3d.version:v5.4.9] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-k3s-default-images:/k3d/images /run/user/1000/podman/podman.sock:/run/user/1000/podman/podman.sock] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:bridge PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[host.k3d.internal:host-gateway] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc00089053f} NetworkingConfig:{EndpointsConfig:map[k3d-k3s-default:0xc00068a480]}} 
DEBU[0000] Created container k3d-k3s-default-tools (ID: 9f4f3cfdf9bc95a9b7314708ebf98df3132c053505cb777b4dd1973e2417aacc) 
DEBU[0000] Node k3d-k3s-default-tools Start Time: 2023-05-17 07:24:35.099081141 +0200 CEST m=+0.207055737 
TRAC[0000] Starting node 'k3d-k3s-default-tools'        
INFO[0000] Starting Node 'k3d-k3s-default-tools'        
DEBU[0000] Truncated 2023-05-17 05:24:35.454599176 +0000 UTC to 2023-05-17 05:24:35 +0000 UTC 
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
TRAC[0001] Creating node from spec
&{Name:k3d-k3s-default-server-0 Role:server Image:docker.io/rancher/k3s:v1.25.7-k3s1 Volumes:[k3d-k3s-default-images:/k3d/images] Env:[K3S_TOKEN=CAzHcSfRpJYxcKnLktBX] Cmd:[] Args:[] Ports:map[] Restart:true Created: HostPidMode:false RuntimeLabels:map[app:k3d k3d.cluster:k3s-default k3d.cluster.imageVolume:k3d-k3s-default-images k3d.cluster.network:k3d-k3s-default k3d.cluster.network.external:false k3d.cluster.network.id:89a5dde53e7c97671dfc4c2ede2d906feeac60b2bad51490f5683f379b649776 k3d.cluster.network.iprange:10.89.0.0/24 k3d.cluster.token:CAzHcSfRpJYxcKnLktBX k3d.cluster.url:https://k3d-k3s-default-server-0:6443 k3d.server.loadbalancer:k3d-k3s-default-serverlb] K3sNodeLabels:map[] Networks:[k3d-k3s-default] ExtraHosts:[] ServerOpts:{IsInit:false KubeAPI:0xc000395b80} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:zero IP Static:false} HookActions:[]} 
TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-k3s-default-server-0 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=CAzHcSfRpJYxcKnLktBX K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san 0.0.0.0 --tls-san k3d-k3s-default-serverlb] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.25.7-k3s1 Volumes:map[] WorkingDir: Entrypoint:[/bin/k3d-entrypoint.sh] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:k3s-default k3d.cluster.imageVolume:k3d-k3s-default-images k3d.cluster.network:k3d-k3s-default k3d.cluster.network.external:false k3d.cluster.network.id:89a5dde53e7c97671dfc4c2ede2d906feeac60b2bad51490f5683f379b649776 k3d.cluster.network.iprange:10.89.0.0/24 k3d.cluster.token:CAzHcSfRpJYxcKnLktBX k3d.cluster.url:https://k3d-k3s-default-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:46789 k3d.server.loadbalancer:k3d-k3s-default-serverlb k3d.version:v5.4.9] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-k3s-default-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:bridge PortBindings:map[] RestartPolicy:{Name:unless-stopped MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc000302fca} NetworkingConfig:{EndpointsConfig:map[k3d-k3s-default:0xc00092a180]}} 
DEBU[0001] Created container k3d-k3s-default-server-0 (ID: 7d14f0cf582f248e9be27d01f532f0af3714c7b4172ea3b3d3c6ea688383a743) 
DEBU[0001] Created node 'k3d-k3s-default-server-0'      
INFO[0001] Creating LoadBalancer 'k3d-k3s-default-serverlb' 
TRAC[0001] Creating node from spec
&{Name:k3d-k3s-default-serverlb Role:loadbalancer Image:ghcr.io/k3d-io/k3d-proxy:5.4.9 Volumes:[k3d-k3s-default-images:/k3d/images] Env:[] Cmd:[] Args:[] Ports:map[6443:[{HostIP:0.0.0.0 HostPort:46789}]] Restart:true Created: HostPidMode:false RuntimeLabels:map[app:k3d k3d.cluster:k3s-default k3d.cluster.imageVolume:k3d-k3s-default-images k3d.cluster.network:k3d-k3s-default k3d.cluster.network.external:false k3d.cluster.network.id:89a5dde53e7c97671dfc4c2ede2d906feeac60b2bad51490f5683f379b649776 k3d.cluster.network.iprange:10.89.0.0/24 k3d.cluster.token:CAzHcSfRpJYxcKnLktBX k3d.cluster.url:https://k3d-k3s-default-server-0:6443 k3d.role:loadbalancer k3d.server.loadbalancer:k3d-k3s-default-serverlb k3d.version:v5.4.9] K3sNodeLabels:map[] Networks:[k3d-k3s-default] ExtraHosts:[] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:zero IP Static:false} HookActions:[{Stage:preStart Action:{Runtime:{} Content:[112 111 114 116 115 58 10 32 32 54 52 52 51 46 116 99 112 58 10 32 32 45 32 107 51 100 45 107 51 115 45 100 101 102 97 117 108 116 45 115 101 114 118 101 114 45 48 10 115 101 116 116 105 110 103 115 58 10 32 32 119 111 114 107 101 114 67 111 110 110 101 99 116 105 111 110 115 58 32 49 48 50 52 10] Dest:/etc/confd/values.yaml Mode:-rwxr--r-- Description:Write Loadbalancer Configuration}}]} 
TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-k3s-default-serverlb Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[6443:{}] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[] Healthcheck:<nil> ArgsEscaped:false Image:ghcr.io/k3d-io/k3d-proxy:5.4.9 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:k3s-default k3d.cluster.imageVolume:k3d-k3s-default-images k3d.cluster.network:k3d-k3s-default k3d.cluster.network.external:false k3d.cluster.network.id:89a5dde53e7c97671dfc4c2ede2d906feeac60b2bad51490f5683f379b649776 k3d.cluster.network.iprange:10.89.0.0/24 k3d.cluster.token:CAzHcSfRpJYxcKnLktBX k3d.cluster.url:https://k3d-k3s-default-server-0:6443 k3d.role:loadbalancer k3d.server.loadbalancer:k3d-k3s-default-serverlb k3d.version:v5.4.9] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-k3s-default-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:bridge PortBindings:map[6443:[{HostIP:0.0.0.0 HostPort:46789}]] RestartPolicy:{Name:unless-stopped MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc00029d68a} NetworkingConfig:{EndpointsConfig:map[k3d-k3s-default:0xc000000180]}} 
DEBU[0001] Created container k3d-k3s-default-serverlb (ID: d714edecd149a5a402cdab8f4f27a0e80f77a62d4111d25f9023e3d421557c80) 
DEBU[0001] Created loadbalancer 'k3d-k3s-default-serverlb' 
DEBU[0001] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
INFO[0001] Using the k3d-tools node to gather environment information 
TRAC[0001] TranslateContainerDetailsToNode: Checking for default object label app=k3d on container /k3d-k3s-default-tools 
DEBU[0001] no netlabel present on container /k3d-k3s-default-tools 
DEBU[0001] failed to get IP for container /k3d-k3s-default-tools as we couldn't find the cluster network 
DEBU[0001] Deleting node k3d-k3s-default-tools ...      
TRAC[0001] [Docker] Deleted Container k3d-k3s-default-tools 
DEBU[0001] DOCKER_SOCK=/run/user/1000/podman/podman.sock 
TRAC[0001] GOOS: linux / Runtime OS: linux ("ol")       
INFO[0001] HostIP: using network gateway 10.89.0.1 address 
INFO[0001] Starting cluster 'k3s-default'               
INFO[0001] Starting servers...                          
DEBU[0001] >>> enabling cgroupsv2 magic                 
DEBU[0001] Node k3d-k3s-default-server-0 Start Time: 2023-05-17 07:24:36.409165969 +0200 CEST m=+1.517140570 
TRAC[0001] Node k3d-k3s-default-server-0: Executing preStartAction 'WriteFileAction': [WriteFileAction] Writing 904 bytes to /bin/k3d-entrypoint.sh (mode -rwxr--r--): Write custom k3d entrypoint script (that powers the magic fixes) 
TRAC[0001] Node k3d-k3s-default-server-0: Executing preStartAction 'WriteFileAction': [WriteFileAction] Writing 1325 bytes to /bin/k3d-entrypoint-cgroupv2.sh (mode -rwxr--r--): Write entrypoint script for CGroupV2 fix 
TRAC[0001] Starting node 'k3d-k3s-default-server-0'     
INFO[0001] Starting Node 'k3d-k3s-default-server-0'     
DEBU[0001] Truncated 2023-05-17 05:24:36.820308446 +0000 UTC to 2023-05-17 05:24:36 +0000 UTC 
DEBU[0001] Waiting for node k3d-k3s-default-server-0 to get ready (Log: 'k3s is up and running') 
TRAC[0001] NodeWaitForLogMessage: Node 'k3d-k3s-default-server-0' waiting for log message 'k3s is up and running' since '2023-05-17 05:24:36 +0000 UTC' 
TRAC[0005] Found target message `k3s is up and running` in log line `Htime="2023-05-17T07:24:40+02:00" level=info msg="k3s is up and running"` 
DEBU[0005] Finished waiting for log message 'k3s is up and running' from node 'k3d-k3s-default-server-0' 
INFO[0005] All agents already running.                  
INFO[0005] Starting helpers...                          
DEBU[0005] Node k3d-k3s-default-serverlb Start Time: 2023-05-17 07:24:40.096123785 +0200 CEST m=+5.204098383 
TRAC[0005] Node k3d-k3s-default-serverlb: Executing preStartAction 'WriteFileAction': [WriteFileAction] Writing 84 bytes to /etc/confd/values.yaml (mode -rwxr--r--): Write Loadbalancer Configuration 
TRAC[0005] Starting node 'k3d-k3s-default-serverlb'     
INFO[0005] Starting Node 'k3d-k3s-default-serverlb'     
DEBU[0005] Truncated 2023-05-17 05:24:40.483664967 +0000 UTC to 2023-05-17 05:24:40 +0000 UTC 
DEBU[0005] Waiting for node k3d-k3s-default-serverlb to get ready (Log: 'start worker processes') 
TRAC[0005] NodeWaitForLogMessage: Node 'k3d-k3s-default-serverlb' waiting for log message 'start worker processes' since '2023-05-17 05:24:40 +0000 UTC' 
TRAC[0011] Found target message `start worker processes` in log line `;2023/05/17 07:24:46 [notice] 31#31: start worker processes` 
DEBU[0011] Finished waiting for log message 'start worker processes' from node 'k3d-k3s-default-serverlb' 
TRAC[0011] Reading path /etc/hosts from node k3d-k3s-default-server-0... 
DEBU[0011] Found network {Name:k3d-k3s-default ID:89a5dde53e7c97671dfc4c2ede2d906feeac60b2bad51490f5683f379b649776 Created:2023-05-17 07:24:35.019007224 +0200 CEST Scope:local Driver:bridge EnableIPv6:false IPAM:{Driver:default Options:map[driver:host-local] Config:[{Subnet:10.89.0.0/24 IPRange: Gateway:10.89.0.1 AuxAddress:map[]}]} Internal:false Attachable:false Ingress:false ConfigFrom:{Network:} ConfigOnly:false Containers:map[7d14f0cf582f248e9be27d01f532f0af3714c7b4172ea3b3d3c6ea688383a743:{Name:k3d-k3s-default-server-0 EndpointID: MacAddress:8a:93:9d:7a:de:f5 IPv4Address:10.89.0.2/24 IPv6Address:} d714edecd149a5a402cdab8f4f27a0e80f77a62d4111d25f9023e3d421557c80:{Name:k3d-k3s-default-serverlb EndpointID: MacAddress:72:89:a5:39:42:97 IPv4Address:10.89.0.3/24 IPv6Address:}] Options:map[isolate:true] Labels:map[app:k3d] Peers:[] Services:map[]} 
INFO[0011] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap... 
TRAC[0011] NodeWaitForLogMessage: Node 'k3d-k3s-default-server-0' waiting for log message 'Cluster dns configmap' since '2023-05-17 05:24:36 +0000 UTC' 
TRAC[0012] Rewritten:
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.89.0.1 host.k3d.internal
10.89.0.2 k3d-k3s-default-server-0
45.33.61.104 mirror.ghettoforge.org
69.195.83.87 elrepo.org
89.187.169.3 download.vscodium.com
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
127.0.1.1 api.snapcraft.io
138.1.117.31 oss.oracle.com
142.251.36.78 packages.cloud.google.com
147.72.252.242 www.cert.org
184.30.25.79 yum.oracle.com
193.28.235.65 download1.rpmfusion.org 
DEBU[0012] Executing command '[sh -c cat /tmp/-etc-hosts-WaCIxotIKcBeotVlncob > /etc/hosts]' in node 'k3d-k3s-default-server-0' 
TRAC[0013] Exec process '[sh -c cat /tmp/-etc-hosts-WaCIxotIKcBeotVlncob > /etc/hosts]' still running in node 'k3d-k3s-default-server-0'.. sleeping for 1 second... 
DEBU[0014] Exec process in node 'k3d-k3s-default-server-0' exited with '0' 

podman logs k3d-k3s-default-server-0 output:

[opc@sws ~]$ podman logs k3d-k3s-default-server-0 
time="2023-05-17T07:24:36+02:00" level=info msg="Starting k3s v1.25.7+k3s1 (f7c20e23)"
time="2023-05-17T07:24:36+02:00" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2023-05-17T07:24:36+02:00" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2023-05-17T07:24:36+02:00" level=info msg="Database tables and indexes are up to date"
time="2023-05-17T07:24:36+02:00" level=info msg="Kine available at unix://kine.sock"
time="2023-05-17T07:24:36+02:00" level=info msg="generated self-signed CA certificate CN=k3s-client-ca@1684301076: notBefore=2023-05-17 05:24:36.940207407 +0000 UTC notAfter=2033-05-14 05:24:36.940207407 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=system:apiserver,O=system:masters signed by CN=k3s-client-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=k3s-cloud-controller-manager signed by CN=k3s-client-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="generated self-signed CA certificate CN=k3s-server-ca@1684301076: notBefore=2023-05-17 05:24:36.944537478 +0000 UTC notAfter=2033-05-14 05:24:36.944537478 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-server-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="generated self-signed CA certificate CN=k3s-request-header-ca@1684301076: notBefore=2023-05-17 05:24:36.945312872 +0000 UTC notAfter=2033-05-14 05:24:36.945312872 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="generated self-signed CA certificate CN=etcd-server-ca@1684301076: notBefore=2023-05-17 05:24:36.945971248 +0000 UTC notAfter=2033-05-14 05:24:36.945971248 +0000 UTC"
The connection to the server localhost:8080 was refused - did you specify the right host or port?
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="generated self-signed CA certificate CN=etcd-peer-ca@1684301076: notBefore=2023-05-17 05:24:36.946976837 +0000 UTC notAfter=2033-05-14 05:24:36.946976837 +0000 UTC"
time="2023-05-17T07:24:36+02:00" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:36 +0000 UTC"
time="2023-05-17T07:24:37+02:00" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:37 +0000 UTC"
time="2023-05-17T07:24:37+02:00" level=warning msg="dynamiclistener [::]:6443: no cached certificate available for preload - deferring certificate load until storage initialization or first client request"
time="2023-05-17T07:24:37+02:00" level=info msg="Active TLS secret / (ver=) (count 12): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-10.89.0.2:10.89.0.2 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-k3d-k3s-default-server-0:k3d-k3s-default-server-0 listener.cattle.io/cn-k3d-k3s-default-serverlb:k3d-k3s-default-serverlb listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=5DE3A9BB7E4CA8ABFCB279853CECFA4EE11EF150]"
time="2023-05-17T07:24:37+02:00" level=info msg="Tunnel server egress proxy mode: agent"
time="2023-05-17T07:24:37+02:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
time="2023-05-17T07:24:37+02:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --etcd-servers=unix://kine.sock --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
time="2023-05-17T07:24:37+02:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259"
time="2023-05-17T07:24:37+02:00" level=info msg="Waiting for API server to become available"
time="2023-05-17T07:24:37+02:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"
time="2023-05-17T07:24:37+02:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"
I0517 07:24:37.089722      24 server.go:581] external host was not specified, using 10.89.0.2
I0517 07:24:37.090034      24 server.go:171] Version: v1.25.7+k3s1
I0517 07:24:37.090047      24 server.go:173] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
time="2023-05-17T07:24:37+02:00" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"
time="2023-05-17T07:24:37+02:00" level=info msg="To join server node to cluster: k3s server -s https://10.89.0.2:6443 -t ${SERVER_NODE_TOKEN}"
time="2023-05-17T07:24:37+02:00" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"
time="2023-05-17T07:24:37+02:00" level=info msg="To join agent node to cluster: k3s agent -s https://10.89.0.2:6443 -t ${AGENT_NODE_TOKEN}"
time="2023-05-17T07:24:37+02:00" level=info msg="Wrote kubeconfig /output/kubeconfig.yaml"
time="2023-05-17T07:24:37+02:00" level=info msg="Run: k3s kubectl"
time="2023-05-17T07:24:37+02:00" level=info msg="certificate CN=k3d-k3s-default-server-0 signed by CN=k3s-server-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:37 +0000 UTC"
time="2023-05-17T07:24:37+02:00" level=info msg="certificate CN=system:node:k3d-k3s-default-server-0,O=system:nodes signed by CN=k3s-client-ca@1684301076: notBefore=2023-05-17 05:24:36 +0000 UTC notAfter=2024-05-16 05:24:37 +0000 UTC"
time="2023-05-17T07:24:37+02:00" level=info msg="Module overlay was already loaded"
time="2023-05-17T07:24:37+02:00" level=info msg="Module nf_conntrack was already loaded"
time="2023-05-17T07:24:37+02:00" level=warning msg="Failed to load kernel module br_netfilter with modprobe"
time="2023-05-17T07:24:37+02:00" level=warning msg="Failed to load kernel module iptable_nat with modprobe"
time="2023-05-17T07:24:37+02:00" level=warning msg="Failed to load kernel module iptable_filter with modprobe"
time="2023-05-17T07:24:37+02:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 4194304"
time="2023-05-17T07:24:37+02:00" level=error msg="Failed to set sysctl: open /proc/sys/net/netfilter/nf_conntrack_max: permission denied"
time="2023-05-17T07:24:37+02:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
time="2023-05-17T07:24:37+02:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
time="2023-05-17T07:24:37+02:00" level=info msg="Set sysctl 'net/bridge/bridge-nf-call-iptables' to 1"
time="2023-05-17T07:24:37+02:00" level=error msg="Failed to set sysctl: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory"
time="2023-05-17T07:24:37+02:00" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
time="2023-05-17T07:24:37+02:00" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
I0517 07:24:37.343129      24 shared_informer.go:255] Waiting for caches to sync for node_authorizer
I0517 07:24:37.344104      24 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0517 07:24:37.344126      24 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0517 07:24:37.345210      24 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0517 07:24:37.345221      24 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0517 07:24:37.361782      24 genericapiserver.go:656] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I0517 07:24:37.362475      24 instance.go:261] Using reconciler: lease
I0517 07:24:37.450240      24 instance.go:574] API group "internal.apiserver.k8s.io" is not enabled, skipping.
W0517 07:24:37.571211      24 genericapiserver.go:656] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
W0517 07:24:37.572546      24 genericapiserver.go:656] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
W0517 07:24:37.575426      24 genericapiserver.go:656] Skipping API autoscaling/v2beta1 because it has no resources.
W0517 07:24:37.582723      24 genericapiserver.go:656] Skipping API batch/v1beta1 because it has no resources.
W0517 07:24:37.584330      24 genericapiserver.go:656] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
W0517 07:24:37.586269      24 genericapiserver.go:656] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
W0517 07:24:37.586365      24 genericapiserver.go:656] Skipping API discovery.k8s.io/v1beta1 because it has no resources.
W0517 07:24:37.591284      24 genericapiserver.go:656] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W0517 07:24:37.591296      24 genericapiserver.go:656] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
W0517 07:24:37.593806      24 genericapiserver.go:656] Skipping API node.k8s.io/v1beta1 because it has no resources.
W0517 07:24:37.593817      24 genericapiserver.go:656] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0517 07:24:37.593957      24 genericapiserver.go:656] Skipping API policy/v1beta1 because it has no resources.
W0517 07:24:37.604706      24 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W0517 07:24:37.604718      24 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0517 07:24:37.606328      24 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
W0517 07:24:37.606336      24 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0517 07:24:37.616442      24 genericapiserver.go:656] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0517 07:24:37.629156      24 genericapiserver.go:656] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0517 07:24:37.633347      24 genericapiserver.go:656] Skipping API apps/v1beta2 because it has no resources.
W0517 07:24:37.633358      24 genericapiserver.go:656] Skipping API apps/v1beta1 because it has no resources.
W0517 07:24:37.635317      24 genericapiserver.go:656] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
W0517 07:24:37.636987      24 genericapiserver.go:656] Skipping API events.k8s.io/v1beta1 because it has no resources.
I0517 07:24:37.637797      24 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0517 07:24:37.637810      24 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0517 07:24:37.647913      24 genericapiserver.go:656] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
time="2023-05-17T07:24:38+02:00" level=info msg="containerd is now running"
time="2023-05-17T07:24:38+02:00" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
time="2023-05-17T07:24:38+02:00" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --feature-gates=DevicePlugins=false --healthz-bind-address=127.0.0.1 --hostname-override=k3d-k3s-default-server-0 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --kubelet-cgroups=/k3s --node-labels= --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --runtime-cgroups=/k3s --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
time="2023-05-17T07:24:38+02:00" level=info msg="Handling backend connection request [k3d-k3s-default-server-0]"
time="2023-05-17T07:24:38+02:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error"
I0517 07:24:38.512959      24 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt"
I0517 07:24:38.512994      24 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt"
I0517 07:24:38.513094      24 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
I0517 07:24:38.513235      24 secure_serving.go:210] Serving securely on 127.0.0.1:6444
I0517 07:24:38.513279      24 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0517 07:24:38.513332      24 apf_controller.go:300] Starting API Priority and Fairness config controller
I0517 07:24:38.513341      24 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0517 07:24:38.513349      24 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0517 07:24:38.513365      24 available_controller.go:491] Starting AvailableConditionController
I0517 07:24:38.513370      24 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0517 07:24:38.513611      24 controller.go:83] Starting OpenAPI AggregationController
I0517 07:24:38.513638      24 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key"
I0517 07:24:38.513672      24 autoregister_controller.go:141] Starting autoregister controller
I0517 07:24:38.513685      24 cache.go:32] Waiting for caches to sync for autoregister controller
I0517 07:24:38.513712      24 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0517 07:24:38.513720      24 shared_informer.go:255] Waiting for caches to sync for cluster_authentication_trust_controller
I0517 07:24:38.513736      24 crdregistration_controller.go:111] Starting crd-autoregister controller
I0517 07:24:38.513742      24 shared_informer.go:255] Waiting for caches to sync for crd-autoregister
I0517 07:24:38.513760      24 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt"
I0517 07:24:38.514318      24 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt"
I0517 07:24:38.514448      24 controller.go:80] Starting OpenAPI V3 AggregationController
I0517 07:24:38.514740      24 controller.go:85] Starting OpenAPI controller
I0517 07:24:38.514814      24 customresource_discovery_controller.go:209] Starting DiscoveryController
I0517 07:24:38.514918      24 naming_controller.go:291] Starting NamingConditionController
I0517 07:24:38.514936      24 controller.go:85] Starting OpenAPI V3 controller
I0517 07:24:38.514976      24 establishing_controller.go:76] Starting EstablishingController
I0517 07:24:38.515183      24 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0517 07:24:38.515237      24 crd_finalizer.go:266] Starting CRDFinalizer
I0517 07:24:38.515375      24 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0517 07:24:38.520888      24 controller.go:616] quota admission added evaluator for: namespaces
E0517 07:24:38.526100      24 controller.go:156] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.43.0.1"}: failed to allocate IP 10.43.0.1: cannot allocate resources of type serviceipallocations at this time
I0517 07:24:38.543228      24 shared_informer.go:262] Caches are synced for node_authorizer
I0517 07:24:38.614234      24 cache.go:39] Caches are synced for autoregister controller
I0517 07:24:38.614241      24 shared_informer.go:262] Caches are synced for crd-autoregister
I0517 07:24:38.614426      24 apf_controller.go:305] Running API Priority and Fairness config worker
I0517 07:24:38.614452      24 shared_informer.go:262] Caches are synced for cluster_authentication_trust_controller
I0517 07:24:38.614702      24 cache.go:39] Caches are synced for AvailableConditionController controller
I0517 07:24:38.614732      24 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0517 07:24:39.346705      24 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0517 07:24:39.517018      24 storage_scheduling.go:95] created PriorityClass system-node-critical with value 2000001000
I0517 07:24:39.519249      24 storage_scheduling.go:95] created PriorityClass system-cluster-critical with value 2000000000
I0517 07:24:39.519257      24 storage_scheduling.go:111] all system priority classes are created successfully or already exist.
I0517 07:24:39.734502      24 controller.go:616] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0517 07:24:39.753554      24 controller.go:616] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
I0517 07:24:39.836715      24 alloc.go:327] "allocated clusterIPs" service="default/kubernetes" clusterIPs=map[IPv4:10.43.0.1]
W0517 07:24:39.839617      24 lease.go:251] Resetting endpoints for master service "kubernetes" to [10.89.0.2]
I0517 07:24:39.840182      24 controller.go:616] quota admission added evaluator for: endpoints
I0517 07:24:39.842407      24 controller.go:616] quota admission added evaluator for: endpointslices.discovery.k8s.io
Error from server (NotFound): nodes "k3d-k3s-default-server-0" not found
time="2023-05-17T07:24:40+02:00" level=info msg="Waiting for cloud-controller-manager privileges to become available"
time="2023-05-17T07:24:40+02:00" level=info msg="Kube API server is now running"
time="2023-05-17T07:24:40+02:00" level=info msg="ETCD server is now running"
time="2023-05-17T07:24:40+02:00" level=info msg="k3s is up and running"
time="2023-05-17T07:24:40+02:00" level=info msg="Applying CRD addons.k3s.cattle.io"
time="2023-05-17T07:24:40+02:00" level=info msg="Applying CRD helmcharts.helm.cattle.io"
time="2023-05-17T07:24:40+02:00" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io"
time="2023-05-17T07:24:40+02:00" level=info msg="Waiting for CRD helmchartconfigs.helm.cattle.io to become available"
Flag --cloud-provider has been deprecated, will be removed in 1.25 or later, in favor of removing cloud provider code from Kubelet.
Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
I0517 07:24:40.164323      24 server.go:199] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
I0517 07:24:40.167597      24 server.go:408] "Kubelet version" kubeletVersion="v1.25.7+k3s1"
I0517 07:24:40.167608      24 server.go:410] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0517 07:24:40.168366      24 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt"
W0517 07:24:40.168598      24 manager.go:159] Cannot detect current cgroup on cgroup v2
E0517 07:24:40.205614      24 info.go:114] Failed to get system UUID: open /etc/machine-id: no such file or directory
W0517 07:24:40.213754      24 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
I0517 07:24:40.215626      24 server.go:655] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
I0517 07:24:40.215856      24 container_manager_linux.go:262] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
I0517 07:24:40.215911      24 container_manager_linux.go:267] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName:/k3s SystemCgroupsName: KubeletCgroupsName:/k3s KubeletOOMScoreAdj:-999 ContainerRuntime: CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerPolicyOptions:map[] ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
I0517 07:24:40.216020      24 topology_manager.go:134] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
I0517 07:24:40.216031      24 container_manager_linux.go:302] "Creating device plugin manager" devicePluginEnabled=false
I0517 07:24:40.216063      24 state_mem.go:36] "Initialized new in-memory state store"
I0517 07:24:40.331864      24 serving.go:355] Generated self-signed cert in-memory
I0517 07:24:40.416893      24 server.go:759] "Failed to ApplyOOMScoreAdj" err="write /proc/self/oom_score_adj: permission denied"
I0517 07:24:40.418846      24 kubelet.go:381] "Attempting to sync node with API server"
I0517 07:24:40.418864      24 kubelet.go:270] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
I0517 07:24:40.418947      24 kubelet.go:281] "Adding apiserver pod source"
I0517 07:24:40.419028      24 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
I0517 07:24:40.419631      24 kuberuntime_manager.go:240] "Container runtime initialized" containerRuntime="containerd" version="v1.6.15-k3s1" apiVersion="v1"
W0517 07:24:40.419826      24 probe.go:268] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
E0517 07:24:40.421024      24 server.go:1159] "Failed to set rlimit on max file handles" err="operation not permitted"
I0517 07:24:40.421036      24 server.go:1170] "Started kubelet"
I0517 07:24:40.421117      24 server.go:155] "Starting to listen" address="0.0.0.0" port=10250
E0517 07:24:40.421388      24 cri_stats_provider.go:452] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs"
E0517 07:24:40.421407      24 kubelet.go:1317] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
I0517 07:24:40.422650      24 server.go:438] "Adding debug handlers to kubelet server"
I0517 07:24:40.425874      24 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
I0517 07:24:40.425909      24 volume_manager.go:293] "Starting Kubelet Volume Manager"
I0517 07:24:40.425940      24 desired_state_of_world_populator.go:149] "Desired state populator starts to run"
E0517 07:24:40.430351      24 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"k3d-k3s-default-server-0\" not found" node="k3d-k3s-default-server-0"
I0517 07:24:40.432818      24 cpu_manager.go:213] "Starting CPU manager" policy="none"
I0517 07:24:40.432830      24 cpu_manager.go:214] "Reconciling" reconcilePeriod="10s"
I0517 07:24:40.432843      24 state_mem.go:36] "Initialized new in-memory state store"
I0517 07:24:40.433952      24 policy_none.go:49] "None policy: Start"
I0517 07:24:40.434236      24 memory_manager.go:168] "Starting memorymanager" policy="None"
I0517 07:24:40.434264      24 state_mem.go:35] "Initializing new in-memory state store"
E0517 07:24:40.437207      24 container_manager_linux.go:435] "Updating kernel flag failed (Hint: enable KubeletInUserNamespace feature flag to ignore the error)" err="open /proc/sys/vm/overcommit_memory: permission denied" flag="vm/overcommit_memory"
E0517 07:24:40.437267      24 container_manager_linux.go:435] "Updating kernel flag failed (Hint: enable KubeletInUserNamespace feature flag to ignore the error)" err="open /proc/sys/kernel/panic: permission denied" flag="kernel/panic"
E0517 07:24:40.437321      24 kubelet.go:1397] "Failed to start ContainerManager" err="[open /proc/sys/vm/overcommit_memory: permission denied, open /proc/sys/kernel/panic: permission denied]"
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

Environment:

[opc@sws ~]$ podman info
host:
  arch: amd64
  buildahVersion: 1.27.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.4-1.module+el8.7.0+20930+90b24198.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: 3922bff22a9c3ddaae27e66d280941f60a8b2554'
  cpuUtilization:
    idlePercent: 99.72
    systemPercent: 0.09
    userPercent: 0.19
  cpus: 128
  distribution:
    distribution: '"ol"'
    variant: server
    version: "8.7"
  eventLogger: file
  hostname: sws.<REDACTED>
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.0-101.103.2.1.el8uek.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 494475345920
  memTotal: 539922157568
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.1.4-1.module+el8.7.0+20930+90b24198.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.4
      spec: 1.0.2-dev
      go: go1.18.9
      libseccomp: 2.5.2
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.module+el8.7.0+20930+90b24198.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 2h 6m 27.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - container-registry.oracle.com
  - docker.io
store:
  configFile: /home/opc/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/opc/.local/share/containers/storage
  graphRootAllocated: 23040677117952
  graphRootUsed: 13010838323200
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 20
  runRoot: /run/user/1000/containers
  volumePath: /home/opc/.local/share/containers/storage/volumes
version:
  APIVersion: 4.2.0
  Built: 1677014962
  BuiltTime: Tue Feb 21 22:29:22 2023
  GitCommit: ""
  GoVersion: go1.18.9
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.0

[opc@sws ~]$ cat /etc/os-release 
NAME="Oracle Linux Server"
VERSION="8.7"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.7"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:8:7:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8"
ORACLE_BUGZILLA_PRODUCT_VERSION=8.7
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=8.7
[opc@sws ~]$ uname -a
Linux sws.<REDACTED> 5.15.0-101.103.2.1.el8uek.x86_64 #2 SMP Mon May 1 20:11:30 PDT 2023 x86_64 x86_64 x86_64 GNU/Linux

[zeelot]

hey @hadrabap , i'm running into the same issue on Ubuntu 22.04 and Podman 4.6.2 where the cluster creation step hangs at INFO[0014] Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap... . Curious if you found a solution to this or any hints :(

[hadrabap]

Hello! I believe this issue might be related to #1120. I traced that the root origin of the issue is in runc (opencontainers/runc#3805).

I've been able to reproduce the same exact problem in KinD and verified that the fix in runc fixes it.

I didn't get enough time to dive deep into K3D to replace the runc with the new one. It seems to be quite involved: lot of images, dependency on K3S… I'm more familiar with KinD.

Anyway, the issue in runc is fixed, but unfortunately scheduled for the up-coming 1.2 release. Unfortunately I don't know the release date. :-(

P.S.: The only K8S distribution out of KinD, K3D, and minikube working with podman rootless is the patched KinD at the moment. Minikube works for me only with KVM driver.