Skip to content

nvpassthrough: bind every PCI function of a GPU, not just the first auxiliary one - #115

Open
jjacobelli wants to merge 1 commit into
NVIDIA:mainfrom
jjacobelli:fix/bind-all-gpu-functions
Open

nvpassthrough: bind every PCI function of a GPU, not just the first auxiliary one#115
jjacobelli wants to merge 1 commit into
NVIDIA:mainfrom
jjacobelli:fix/bind-all-gpu-functions

Conversation

@jjacobelli

Copy link
Copy Markdown

Follow-up to NVIDIA/k8s-driver-manager#254.

Problem

VFIO assigns an entire IOMMU group to a guest and refuses the group unless every
device in it is bound to a vfio driver or to no driver at all. A discrete NVIDIA
GPU is a multi-function PCI device: .0 VGA/3D, .1 HDMI audio, and on
Turing-era boards .2 (VirtualLink USB xHCI) and .3 (USB-C UCSI).

getGraphicsAuxDev() could not find all of those:

  • it returned on the first match, so at most one auxiliary function was ever bound;
  • it discovered functions only via consumer:pci: device links. The HDA audio
    function creates one, but the VirtualLink xHCI and USB-C UCSI functions do not,
    so they were structurally invisible;
  • it returned early unless the device was PCIVgaControllerClass, so a GPU
    enumerating as a 3D controller (0x030200) got no auxiliary handling at all.

In practice this leaves xhci_hcd owning .2. The IOMMU group is then not
viable and passthrough fails with vfio: group N is not viable.

Fix

Replace it with getAuxDevices(), which returns every other function of the same
physical card. Functions are discovered both by enumerating siblings that share a
domain:bus:device and by following consumer links; the two sets are unioned and
de-duplicated.

Siblings are scoped to the physical card rather than to the IOMMU group
deliberately: where ACS is unavailable a group can span a whole root port, and
unrelated devices must not be unbound from their drivers. A vendor check against
nvpci.PCINvidiaVendorID guards against touching anything that is not an NVIDIA
function.

BindToVFIODriver() and Unbind() now iterate over all of them.

getAuxDevices() takes the devices root as a parameter so the discovery logic is
testable against a fake sysfs tree.

Tests

Adds table-driven tests for the discovery logic covering multi-function boards,
single-function GPUs, foreign-vendor siblings, devices on other slots, class
handling (VGA / 3D controller / NVSwitch), and the de-duplication of a function
reachable both as a sibling and via a consumer link. A second test asserts the
currently-bound driver is reported per function, which is what lets
BindToVFIODriver() skip functions already on the vfio driver.

make assert-fmt goimports vet check-vendor and go test ./pkg/nvpassthrough/
pass; no dependency or vendor changes.

…uxiliary one

VFIO assigns an entire IOMMU group to a guest and refuses the group unless every
device in it is bound to a vfio driver or to no driver at all. A discrete NVIDIA
GPU is a multi-function PCI device: .0 VGA/3D, .1 HDMI audio, and on Turing-era
boards .2 (VirtualLink USB xHCI) and .3 (USB-C UCSI).

getGraphicsAuxDev() could not find all of those:

  * it returned on the first match, so at most one auxiliary function was ever
    bound;
  * it discovered functions only via "consumer:pci:" device links. The HDA audio
    function creates one, but the VirtualLink xHCI and USB-C UCSI functions do
    not, so they were structurally invisible;
  * it returned early unless the device was PCIVgaControllerClass, so a GPU
    enumerating as a 3D controller (0x030200) got no auxiliary handling at all.

In practice this leaves xhci_hcd owning .2. The IOMMU group is then not viable
and passthrough fails with "vfio: group N is not viable".

Replace it with getAuxDevices(), which returns every other function of the same
physical card. Functions are discovered both by enumerating siblings that share
a domain:bus:device and by following consumer links, and the two sets are
unioned and de-duplicated. Siblings are scoped to the card rather than to the
IOMMU group deliberately: where ACS is unavailable a group can span a whole root
port, and unrelated devices must not be unbound from their drivers. A vendor
check guards against touching anything that is not an NVIDIA function.

BindToVFIODriver() and Unbind() now iterate over all of them.

Adds table-driven tests for the discovery logic covering multi-function boards,
single-function GPUs, foreign-vendor siblings, devices on other slots, and the
de-duplication of a function reachable both ways.

Signed-off-by: Jordan Jacobelli <jjacobelli@nvidia.com>
@jjacobelli
jjacobelli marked this pull request as ready for review September 10, 2026 12:59
Comment on lines +364 to +373
if _, err := os.Stat(path); err != nil {
// The function is not present on this host; there is nothing to bind.
return nil
}
// Auxiliary functions of a GPU are by definition the same vendor. Checking guards
// against acting on a device we did not intend to touch.
isNvidia, err := isNvidiaVendor(path)
if err != nil || !isNvidia {
return nil
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if _, err := os.Stat(path); err != nil {
// The function is not present on this host; there is nothing to bind.
return nil
}
// Auxiliary functions of a GPU are by definition the same vendor. Checking guards
// against acting on a device we did not intend to touch.
isNvidia, err := isNvidiaVendor(path)
if err != nil || !isNvidia {
return nil
}
if _, err := os.Stat(path); err != nil {
if os.IsNotExist(err) {
// The function is not present on this host; there is nothing to bind.
return nil
}
return fmt.Errorf("failed to stat auxiliary device %s: %w", address, err)
}
// Auxiliary functions of a GPU are by definition the same vendor. Checking guards
// against acting on a device we did not intend to touch.
isNvidia, err := isNvidiaVendor(path)
if err != nil {
return fmt.Errorf("failed to check vendor for auxiliary device %s: %w", address, err)
}
if !isNvidia {
return nil
}

}

// isNvidiaVendor reports whether the PCI device at devicePath is an NVIDIA device.
func isNvidiaVendor(devicePath string) (bool, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also handle the FileNotFound here? Is the FileNotFound error to be tolerated instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants