VFs may get reseted after being allocated by other pod

[ormergi]

What happened?

In the scenario where pods with SR-IOV interface from the same resource pool are created and deleted a few times,
the underlying VF may end up with a default configuration instead of the desired one (i.e: no MAC address, no VLAN).

What did you expect to happen?

The pod underlying VF is to be configured correctly.

What are the minimal steps needed to reproduce the bug?

Terminal 1: Monitor SR-IOV VFs state on node worker1
Terminal 2:

• Create a pod test with:
  • nodeSelector to node worker1.
  • SR-IOV interface with MAC address 02:02:02:02:02.
• Wait for the pod to be ready.
• Delete the pod test in the background and immediately create a similar pod test2.
• Wait for pod test2 to be ready.

After a few iterations, we saw that the pod is Running but looking at the node VFs
it shows that it's not configured with the desired MAC address.

Anything else we need to know?

Scripts and manifests I used for reproducing the issue:
https://gist.github.com/ormergi/3ddbf901ddc95baf316b604994285a69

It seems that when the pod (1) is deleted and CNI cmdDEL command been executed, it will reset the VF whether it's been allocated by another pod or not.

Also, it's not guaranteed that as soon as a pod is disposed its underlying VF is reseted.
It takes 1-2 seconds for it to reset, which seem odd because I would expect all its resource to be free when it's gone.

Component Versions

Component	Version
SR-IOV CNI Plugin	v2.6
Multus	v3.8
SR-IOV Network Device Plugin	v3.3
Kubernetes	v1.22.2
OS	RHEL 8.6

Config Files

Config file locations may be config dependent.

CNI config (Try '/etc/cni/net.d/')

cat /etc/cni/net.d/00-multus.conf
{ "cniVersion": "0.3.1", "name": "multus-cni-network", "type": "multus", "capabilities": {"portMappings": true}, "logLevel": "debug", "logFile": "/var/log/multus.log", "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig", "delegates": [ { "cniVersion": "0.3.1", "name": "kindnet", "plugins": [ { "type": "ptp", "ipMasq": false, "ipam": { "type": "host-local", "dataDir": "/run/cni-ipam-state", "routes": [ { "dst": "0.0.0.0/0" } ], "ranges": [ [ { "subnet": "10.244.2.0/24" } ] ] } , "mtu": 1500 }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } ] }

cat /etc/cni/net.d/10-kindnet.conflist
{
        "cniVersion": "0.3.1",
        "name": "kindnet",
        "plugins": [
        {
                "type": "ptp",
                "ipMasq": false,
                "ipam": {
                        "type": "host-local",
                        "dataDir": "/run/cni-ipam-state",
                        "routes": [
                                { "dst": "0.0.0.0/0" }
                        ],
                        "ranges": [
                                [ { "subnet": "10.244.2.0/24" } ]
                        ]
                }
                ,
                "mtu": 1500
        },
        {
                "type": "portmap",
                "capabilities": {
                        "portMappings": true
                }
        }
        ]
}

Device pool config file location (Try '/etc/pcidp/config.json')

{
  "resourceList": [{
    "resourceName": "sriov_net",
    "selectors": {
      "drivers": ["vfio-pci"],
      "pfNames": ["enp4s0f0","enp4s0f1"]
    }
  }]
}

Multus config (Try '/etc/cni/multus/net.d')

see ##### CNI config

Kubernetes deployment type ( Bare Metal, Kubeadm etc.)

Kind deployment

Kubeconfig file

SR-IOV Network Custom Resource Definition

kind: NetworkAttachmentDefinition
metadata:
  annotations:
    k8s.v1.cni.cncf.io/resourceName: kubevirt.io/sriov_net
  name: sriov-network
spec:
  config: "{ \n    \"cniVersion\":\"0.3.1\", \n    \"name\":\"sriov-network\",\n
    \   \"type\":\"sriov\",\n    \"vlan\":0,\n    \"spoofchk\":\"on\",\n    \"trust\":\"off\",\n
    \   \"vlanQoS\":0,\n    \"link_state\":\"enable\",\n    \"ipam\":{}\n}"

Logs

SR-IOV Network Device Plugin Logs (use kubectl logs $PODNAME)

Multus logs (If enabled. Try '/var/log/multus.log' )

Kubelet logs (journalctl -u kubelet)

Journal log including Multus logs from when the issue accord https://pastebin.com/eGhyrevk.

[ormergi]

Script and manifests I used in order to reproduce this https://gist.github.com/ormergi/3ddbf901ddc95baf316b604994285a69.
Journal log from one of the iterations when the issue occurred https://pastebin.com/eGhyrevk.

[SchSeba]

Hi @ormergi can this be related to #126 ?

[ormergi]

@SchSeba thanks for looking at this.
It seems that #126 is about kubelet won't keep the pod alive so that the plugin will manage to reach its netns and do its thing.
In our case, the plugin executes as we can see the VFs get reset after the pod is allegedly disposed though.

[SchSeba]

@ormergi will you be able to provide the logs from multus? that is the only way we will be able to try and trace the issue I think

[ormergi]

Script and manifests I used in order to reproduce this https://gist.github.com/ormergi/3ddbf901ddc95baf316b604994285a69. Journal log from one of the iterations when the issue occurred https://pastebin.com/eGhyrevk.

@SchSeba are these logs good enough https://pastebin.com/eGhyrevk?
This is a journal log of the node since the pod was created + 1 minute before, it also includes Multus logs.

EDIT: FWIW its much easier to review these logs in vscode.

[ormergi]

@adrianchiris is there a planned release so we could consume the fixed version? 🙂