Windows: networking cannot route inside the cluster (so no DNS)

[mjj29]

Environmental Info:
rke2 version v1.24.13+rke2r1 (05a2e96)

(control plane)
Linux 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux
(worker)
Windows 2022 Standard

Cluster Configuration:
One linux to hold the control plane, one windows worker node

Describe the bug:
When start a container on the windows node it can access external IP addresses, but it cannot access other IPs in the cluster. This means DNS doesn't work, so I can't access external things except by IP.

Steps To Reproduce:
I installed RKE2 using the deployment from Rancher which gave the instructions to deploy the various node types on the control plane node and the windows worker node. For the windows node this is by running:

curl.exe -fL https://rancher.apama.com/wins-agent-install.ps1 -o install.ps1; Set-ExecutionPolicy Bypass -Scope Process -Force; ./install.ps1 -Server https://rancher.apama.com -Label 'cattle.io/os=windows' -Token (redacted) -Worker -CaChecksum (redacted)

I then deployed a pod through the rancher UI using mcr.microsoft.com/windows/servercore:ltsc2022 as the image and got a shell to run the commands below

Expected behavior:
DNS lookups to work and networking to be able to route between the pods

Actual behavior:
(another container on the same windows node)
C:>ping 10.42.215.77
Pinging 10.42.215.77 with 32 bytes of data:
Reply from 10.42.215.77: bytes=32 time<1ms TTL=128

(an external IP address)
C:>ping 8.8.8.8
Pinging 8.8.8.8 with 32 bytes of data:
Reply from 8.8.8.8: bytes=32 time=1ms TTL=114

(a container on the linux node in the cluster)
C:>ping 10.42.212.149
Pinging 10.42.212.149 with 32 bytes of data:
Request timed out.

Doing a DNS lookup:
C:>nslookup teamcity.apama.com 10.43.0.10
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 10.43.0.10

Additional context / logs:
The control plane has a lot of pods running in various -system namespaces, but the worker doesn't have any. I was expecting at least one to handle the networking stuff.

[rbrtbnfgl]

Hi what pod are you try to contact? The ping is done on the windows node or inside the pod shell?
The windows networking is managed by the Calico-node process that runs inside the RKE2 service alongside with kube-proxy.

[mjj29]

Hi there. I'm running ping inside the pod shell. It cannot contact anything on the linux node in the cluster, including kube-dns, so it cannot do name resolution, but it can contact other pods on the same node. I've disabled the firewall on both nodes.

[rbrtbnfgl]

How did you configure RKE2? I presume that you are using calico as CNI.

[rbrtbnfgl]

Are yo sure that you are executing those commands inside a pod? Could you check the status of the deployed pod on windows?

[mjj29]

Yes, it's configured with calico for CNI.
Yes I am sure that I'm executing those commands inside a pod (using the shell access through Rancher). The status is fine of the deployed pod on windows. It's all marked as running correctly, it just can't do DNS

[manuelbuil]

I have just tested and things seem to work on my env.

Could you verify the output of kubectl get nodes -o yaml | grep projectcalico? All IPv4Addresses should be in the same range. IPv4VXLANTunnelAddr should be in their same range too.

Could you also verify that from a linux node you can ping a windows node (or the other way around)?

[rbrtbnfgl]

My concern is related to how rancher exposes the windows cli and if the pod is rightly created.

[rbrtbnfgl]

I tested a cluster deployed with rancher with a master linux node and a windows worker node.
Are you sure that the pods with mcr.microsoft.com/windows/servercore:ltsc2022 image is working? I am asking it because I wasn't able to get it on ready state.

[nadenf]

I have the same issue. Running a Github runner on Windows node with 2 node cluster.

It can ping 8.8.8.8 but not my Linux node which in turns means DNS resolution doesn't work.

Also running WS 2022, RKE2 (v1.25.10), Calico.

[nadenf]

@manuelbuil

      projectcalico.org/IPv4Address: 100.92.24.103/32
      projectcalico.org/IPv4VXLANTunnelAddr: 10.42.69.192
      projectcalico.org/IPv4Address: 192.168.1.106/24
      projectcalico.org/IPv4VXLANTunnelAddr: 10.42.61.193
      projectcalico.org/VXLANTunnelMACAddr: 00:15:5d:fd:06:d3

And yes can ping Windows node from my Linux node.

But can't ping the podIP of the Windows pod from a Linux pod.

[nadenf]

Not sure if it's related but I do have two pods:

rke2-coredns-rke2-coredns-6b9548f79f-4x4pf
rke2-coredns-rke2-coredns-6b9548f79f-9bcrv

First is running on the Linux node. Second is pending because it is set to only run on Linux nodes but has anti-affinity enabled. Is this supposed to run on each node including Windows ?

[rbrtbnfgl]

It shouldn't try to run the coreDNS pod on windows. On your setup the PODs on windows can go to the internet but they can't ping other pods on the linux node?

[nadenf]

@rbrtbnfgl .. Correct. Same issue as OP.

And the Windows machine is a fresh install of WS 2022 and RKE.

[rbrtbnfgl]

I checked the coreDNS pod will be pending on the windows node so what happens to you is right the DNS query should be forwarded to the Linux control plane node.

[nadenf]

Seeing these logs repeating for calico-node:

│ calico-node 2023-06-29 03:40:52.380 [INFO][85] felix/vxlan_mgr.go 685: VXLAN device MTU needs to be updated device="vxlan.calico" ipVersion=0x4 new=9000 old=1230                                                                                           │
│ calico-node 2023-06-29 03:40:52.380 [WARNING][85] felix/vxlan_mgr.go 687: Failed to set vxlan tunnel device MTU error=invalid argument ipVersion=0x4                                                                                                        │
│ calico-node 2023-06-29 03:40:59.909 [WARNING][85] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.42.61.192                                                                            │
│ calico-node 2023-06-29 03:40:59.909 [WARNING][85] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.42.61.193                                                                            │
│ calico-node 2023-06-29 03:40:59.909 [WARNING][85] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.42.61.194                                                                            │
│ calico-node 2023-06-29 03:40:59.909 [WARNING][85] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.42.61.255                                                                            │
│ calico-node 2023-06-29 03:41:00.700 [WARNING][85] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.42.61.192                                                                            │
│ calico-node 2023-06-29 03:41:00.700 [WARNING][85] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.42.61.193                                                                            │
│ calico-node 2023-06-29 03:41:00.700 [WARNING][85] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.42.61.194                                                                            │
│ calico-node 2023-06-29 03:41:00.700 [WARNING][85] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.42.61.255                                                                            │

[rbrtbnfgl]

Thanks these could be useful I'll check if they could be the reason of the issue.
Edit:
Are these logs from the calico-node pod on linux? The 10.42.61.x network is the one located for the windows node?

[brandond]

VXLAN device MTU needs to be updated device="vxlan.calico" ipVersion=0x4 new=9000 old=1230

Are you by any chance using jumbo frames on the Windows nodes, but not on the Linux nodes? Or vice versa?

[nadenf]

@rbrtbnfgl .. Yes the logs are from calico-node on Linux. Not sure what 10.42.61.x is.

My internal host network is 192.168.1.x.

[nadenf]

@brandond ..

Linux:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
2: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
3: wlo1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
5: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 500
6: cali2d43d5e9d02@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default
7: cali081726231f3@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default
8: cali828073b2320@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default
9: cali2ddce6e4bfc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default
10: calia77cadd9592@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default
11: cali3fb13decb07@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default
12: cali1fc3a87687d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default
13: cali9138fb1f152@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default
14: cali88ab826f222@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default
15: cali1d049815b72@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default
16: cali4d613998696@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default
19: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1230 qdisc noqueue state UNKNOWN mode DEFAULT group default
20: calid2bf05f2095@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default

Windows:

[nadenf]

I am not that familiar with Windows but as you can see above my NIC isn't connected but the Hyper-V one is.

It isn't using Jumbo Frames either though. I tried enabling Jumbo Frames on both interfaces but it didn't make any difference.

Looks like it maybe a known issue with Calico and Hyper-V.

[nadenf]

I ran Get-HNsEndpoint and it shows a bunch of Ethernet items and this is the Pod I am testing with. It's a Github Action runner that fails to access api.github.com.

ID                 : 8295a192-fd79-4ae6-a482-07758bcd123e
Name               : 1cae7f3dfbf44919a30ae139d18e1bf5e98afc4038b370e205fb23276d70b8f0_Calico
Version            : 55834574851
AdditionalParams   :
Resources          : @{AdditionalParams=; AllocationOrder=14; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=DBB2BF93-3D14-462A-B4D3-1E419CECA152; PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0; parentId=D929BB8C-8C72-47AB-B581-43EFFBBF0DD7}
State              : 3
VirtualNetwork     : 3fee3f14-521e-4e79-bcb2-61549e262ee0
VirtualNetworkName : Calico
Policies           : {@{ExceptionList=System.Object[]; Type=OutBoundNAT}, @{DestinationPrefix=10.43.0.0/16; NeedEncap=True; Type=ROUTE}, @{PA=192.168.1.106; Type=PA}, @{Action=Allow; Direction=In; Id=allow-host-to-endpoint; InternalPort=0; LocalAddresses=; LocalPort=0; Priority=900; Protocol=256; RemoteAddresses=192.168.1.106/32; RemotePort=0; RuleType=Switch; Scope=0;
                     ServiceName=; Type=ACL}...}
MacAddress         : 0E-2A-0a-2a-3d-e2
IPAddress          : 10.42.61.226
PrefixLength       : 26
GatewayAddress     : 10.42.61.193
IPSubnetId         : f7951b78-7584-489f-8999-06c2379d4482
DNSServerList      : 10.43.0.10
DNSSuffix          : actions-runner-system.svc.cluster.local,svc.cluster.local,cluster.local
Namespace          : @{ID=6aa8f011-4058-49e2-8fae-632c98a415fc}
EncapOverhead      : 50
SharedContainers   : {1cae7f3dfbf44919a30ae139d18e1bf5e98afc4038b370e205fb23276d70b8f0}

[rbrtbnfgl]

Hi @nadenf the logs seem to be unrelated to your issue because I got the same logs but network works fine to me. How do you deploy the pod is it a deployment or it's directly github that is accessing your cluster? Which is the container image used?

[rbrtbnfgl]

I'll check the Hyper-v issue if it could be related somehow.

[nadenf]

cat <<EOF | kubectl apply -f -
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runners-windows
  namespace: actions-runner-system
spec:
  template:
    spec:
      image: haranaoss/actions-runner-image:ltsc2022
      dockerdWithinRunnerContainer: true
      nodeSelector:
        kubernetes.io/os: windows
        kubernetes.io/arch: amd64
      organization: harana
      labels:
        - Windows
        - X64
EOF

This is the image.

And this is the logs from the Windows Pod:

[rbrtbnfgl]

I tried your pod and the networks id working fine.
Reading from your comments you are using windows server 2022 without any additional configuration. You enabled the container function and the start RKE2 using the default config on both Linux and Windows. RKE2 was installed using the install script or through the rancher UI?

[nadenf]

I disabled the firewall and related security features.

But otherwise I just followed the instructions on the Quick Start page.

[brandond]

Are the Windows nodes on the same network as the other nodes, or are they hosted remotely? What sort of connectivity do the different node types have?

[nadenf]

• Ubuntu Linux 22.04 and WS 2022 nodes
• On-premise directly connected to a 10GB ethernet switch
• External traffic accessible outside of Rancher e.g. Edge browser works
• Connectivity between Linux and Windows works because it is able to schedule pods

[brandond]

Connectivity between Linux and Windows works because it is able to schedule pods

Connectivity between the kubelet and apiserver is not the same as connectivity between pods. Kubelet-apiserver traffic is a simple HTTPS request. CNI traffic uses the vxlan overlay network, which is much more fragile in terms of fragmentation and potential for blocking by intermediate network security devices. It is not uncommon for traffic to the apiserver to work fine, while vxlan traffic gets blocked, dropped, or mangled by intermediate network devices.

[nadenf]

Added a ConnectX-6 card to each node which supports VXLAN, configured for Ethernet mode, directly connected them via QSPF i.e. no switch and reinstalled RKE2.

Same issue.

[rbrtbnfgl]

You could try to capture the traffic on the windows physical interface to check if the VXLAN traffic is present or it's dropped inside the windows node.
From your description you have two physical server directly connected and RKE2 runs on them without any VM is it correct?

[nadenf]

Yes directly connected, no VMs other than Hyper-V on the Windows side.

Will look into sniffing the traffic.

[manuelbuil]

@manuelbuil

      projectcalico.org/IPv4Address: 100.92.24.103/32
      projectcalico.org/IPv4VXLANTunnelAddr: 10.42.69.192
      projectcalico.org/IPv4Address: 192.168.1.106/24
      projectcalico.org/IPv4VXLANTunnelAddr: 10.42.61.193
      projectcalico.org/VXLANTunnelMACAddr: 00:15:5d:fd:06:d3

And yes can ping Windows node from my Linux node.

But can't ping the podIP of the Windows pod from a Linux pod.

Sorry for the delay, I have just seen this reply. One node is using 100.92.24.103/32 and the other node is using 192.168.1.106/24 for inter-node communication, I don't think those IP addresses are in the same network, so I'd say it makes sense that pods in different nodes cannot communicate with each other. I'd say that the problem you are having is that in one node, calico is using the wrong interface for communication.

This is how the interface is picked in rke2-windows https://github.com/rancher/rke2/blob/master/pkg/windows/calico.go#L277-L281

[rbrtbnfgl]

I think that linux is using the wrong interface not windows. You should use the node-ip to forse to use the right one. RKE2 is selecting tailscale0 that it's not the right one.

[nadenf]

@manuelbuil .. Great. Looks like that is the issue which is actually on my Linux node that is running the rke2-server.

Looking at the rke2-calico chart there is no way for me to configure the Calico IP Detection Behaviour.

Also would be good to document this as having multiple interfaces is pretty common and right now it's just picking the first one it finds. Or even better is if the install script can detect multiple interfaces and ask the user which one it should use.

Should I raise a separate issue or I can submit a PR to update the chart to add support for it ?

[rbrtbnfgl]

are you using rancher UI or you are running the binary directly? The chart used is the same from Calico with patches you should be able to configure the chart on the same way the upstream calico chart is configured. You could add your configuration on the path /var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml

[nadenf]

I followed the Quick Start guide. And then I created the following file at rke2-calico-config.yaml to force it to use the right interface:

---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-calico
  namespace: kube-system
spec:
  valuesContent: |-
    installation:
      calicoNetwork:
        mtu: 9000
        nodeAddressAutodetectionV4:
          interface: enp1s0np0

I then did a kubectl apply and nothing has happened.

The documentation on this page really isn't clear what exactly I am supposed to do.

[rbrtbnfgl]

You don't need to apply it through kubectl it's enough to add that file on the path that I mentioned on my previous comment and start RKE2.

[manuelbuil]

interface: enp1s0np0

I followed the Quick Start guide. And then I created the following file at rke2-calico-config.yaml to force it to use the right interface:

---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-calico
  namespace: kube-system
spec:
  valuesContent: |-
    installation:
      calicoNetwork:
        mtu: 9000
        nodeAddressAutodetectionV4:
          interface: enp1s0np0

I then did a kubectl apply and nothing has happened.

The documentation on this page really isn't clear what exactly I am supposed to do.

Doing this is dangerous because you are specifying that Calico should look for interface: enp1s0np0 in ALL nodes. I bet you don't have that interface in your windows node ;). I think it would be easier if you deploy rke2 setting the correct node-ip

[nadenf]

Blocked on #4403

[nadenf]

/etc/rancher/rke2/config.yaml:

cni: calico
node-ip: 192.168.2.2

And after restarting/killing etc. it still shows old value:

projectcalico.org/IPv4Address: 100.92.24.103/32
projectcalico.org/IPv4VXLANTunnelAddr: 10.42.69.192
rke2.io/internal-ip: 192.168.2.2
rke2.io/node-args: ["server","--cni","calico","--node-ip","192.168.2.2"]

I am assuming by the details above that the issue is the Calico autodetection.

[nadenf]

So this is the file used to configure Calico:
https://github.com/rancher/rke2-charts/blob/main-source/packages/rke2-calico/generated-changes/patch/values.yaml.patch

And what I need is to be able to configure the behaviour described here i.e. use node IP instead of auto detect:
https://docs.tigera.io/calico/latest/networking/ipam/ip-autodetection

Is it worth making node IP the default ?

[rbrtbnfgl]

if you start RKE2 with the following calico helm configuration should be working on your case:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-calico
  namespace: kube-system
spec:
  valuesContent: |-
    installation:
      calicoNetwork:
         nodeAddressAutodetectionV4:
             kubernetes: NodeInternalIP

[nadenf]

Above Applied:

root@harana-server:/home/naden# vi /var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml
root@harana-server:/home/naden# kubectl apply -f /var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml
helmchartconfig.helm.cattle.io/rke2-calico configured
root@harana-server:/home/naden# pkill -f rke2
root@harana-server:/home/naden# ps -ef | grep rke2
root     2738133 2580903  0 09:24 pts/1    00:00:00 grep --color=auto rke2
systemctl start rke2-server

kubectl describe HelmChartConfig/rke2-calico -n kube-system

Spec:
  Values Content:  installation:
  calicoNetwork:
     nodeAddressAutodetectionV4:
         kubernetes: NodeInternalIP

kubectl describe node/harana-server

projectcalico.org/IPv4Address: 100.92.24.103/32
projectcalico.org/IPv4VXLANTunnelAddr: 10.42.69.192
rke2.io/encryption-config-hash: start-19887c44fcbeb0de90dfdfd85f289b230c1e93f0da60555baaeb5c3f66064d2f
rke2.io/hostname: harana-server
rke2.io/internal-ip: 192.168.2.2
rke2.io/node-args: ["server","--cni","calico","--node-ip","192.168.2.2"]

[davidhrbac]

See #1353 (comment) - working solution.

[caroline-suse-rancher]

Closing, as this issue has a known-working workaround