nvpassthrough: bind every PCI function of a GPU, not just the first auxiliary one - #115
Open
jjacobelli wants to merge 1 commit into
Open
nvpassthrough: bind every PCI function of a GPU, not just the first auxiliary one#115jjacobelli wants to merge 1 commit into
jjacobelli wants to merge 1 commit into
Conversation
…uxiliary one
VFIO assigns an entire IOMMU group to a guest and refuses the group unless every
device in it is bound to a vfio driver or to no driver at all. A discrete NVIDIA
GPU is a multi-function PCI device: .0 VGA/3D, .1 HDMI audio, and on Turing-era
boards .2 (VirtualLink USB xHCI) and .3 (USB-C UCSI).
getGraphicsAuxDev() could not find all of those:
* it returned on the first match, so at most one auxiliary function was ever
bound;
* it discovered functions only via "consumer:pci:" device links. The HDA audio
function creates one, but the VirtualLink xHCI and USB-C UCSI functions do
not, so they were structurally invisible;
* it returned early unless the device was PCIVgaControllerClass, so a GPU
enumerating as a 3D controller (0x030200) got no auxiliary handling at all.
In practice this leaves xhci_hcd owning .2. The IOMMU group is then not viable
and passthrough fails with "vfio: group N is not viable".
Replace it with getAuxDevices(), which returns every other function of the same
physical card. Functions are discovered both by enumerating siblings that share
a domain:bus:device and by following consumer links, and the two sets are
unioned and de-duplicated. Siblings are scoped to the card rather than to the
IOMMU group deliberately: where ACS is unavailable a group can span a whole root
port, and unrelated devices must not be unbound from their drivers. A vendor
check guards against touching anything that is not an NVIDIA function.
BindToVFIODriver() and Unbind() now iterate over all of them.
Adds table-driven tests for the discovery logic covering multi-function boards,
single-function GPUs, foreign-vendor siblings, devices on other slots, and the
de-duplication of a function reachable both ways.
Signed-off-by: Jordan Jacobelli <jjacobelli@nvidia.com>
jjacobelli
marked this pull request as ready for review
September 10, 2026 12:59
tariq1890
reviewed
Sep 11, 2026
Comment on lines
+364
to
+373
| if _, err := os.Stat(path); err != nil { | ||
| // The function is not present on this host; there is nothing to bind. | ||
| return nil | ||
| } | ||
| // Auxiliary functions of a GPU are by definition the same vendor. Checking guards | ||
| // against acting on a device we did not intend to touch. | ||
| isNvidia, err := isNvidiaVendor(path) | ||
| if err != nil || !isNvidia { | ||
| return nil | ||
| } |
Contributor
There was a problem hiding this comment.
Suggested change
| if _, err := os.Stat(path); err != nil { | |
| // The function is not present on this host; there is nothing to bind. | |
| return nil | |
| } | |
| // Auxiliary functions of a GPU are by definition the same vendor. Checking guards | |
| // against acting on a device we did not intend to touch. | |
| isNvidia, err := isNvidiaVendor(path) | |
| if err != nil || !isNvidia { | |
| return nil | |
| } | |
| if _, err := os.Stat(path); err != nil { | |
| if os.IsNotExist(err) { | |
| // The function is not present on this host; there is nothing to bind. | |
| return nil | |
| } | |
| return fmt.Errorf("failed to stat auxiliary device %s: %w", address, err) | |
| } | |
| // Auxiliary functions of a GPU are by definition the same vendor. Checking guards | |
| // against acting on a device we did not intend to touch. | |
| isNvidia, err := isNvidiaVendor(path) | |
| if err != nil { | |
| return fmt.Errorf("failed to check vendor for auxiliary device %s: %w", address, err) | |
| } | |
| if !isNvidia { | |
| return nil | |
| } |
tariq1890
reviewed
Sep 11, 2026
| } | ||
|
|
||
| // isNvidiaVendor reports whether the PCI device at devicePath is an NVIDIA device. | ||
| func isNvidiaVendor(devicePath string) (bool, error) { |
Contributor
There was a problem hiding this comment.
Should we also handle the FileNotFound here? Is the FileNotFound error to be tolerated instead?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to NVIDIA/k8s-driver-manager#254.
Problem
VFIO assigns an entire IOMMU group to a guest and refuses the group unless every
device in it is bound to a vfio driver or to no driver at all. A discrete NVIDIA
GPU is a multi-function PCI device:
.0VGA/3D,.1HDMI audio, and onTuring-era boards
.2(VirtualLink USB xHCI) and.3(USB-C UCSI).getGraphicsAuxDev()could not find all of those:consumer:pci:device links. The HDA audiofunction creates one, but the VirtualLink xHCI and USB-C UCSI functions do not,
so they were structurally invisible;
PCIVgaControllerClass, so a GPUenumerating as a 3D controller (
0x030200) got no auxiliary handling at all.In practice this leaves
xhci_hcdowning.2. The IOMMU group is then notviable and passthrough fails with
vfio: group N is not viable.Fix
Replace it with
getAuxDevices(), which returns every other function of the samephysical card. Functions are discovered both by enumerating siblings that share a
domain:bus:deviceand by following consumer links; the two sets are unioned andde-duplicated.
Siblings are scoped to the physical card rather than to the IOMMU group
deliberately: where ACS is unavailable a group can span a whole root port, and
unrelated devices must not be unbound from their drivers. A vendor check against
nvpci.PCINvidiaVendorIDguards against touching anything that is not an NVIDIAfunction.
BindToVFIODriver()andUnbind()now iterate over all of them.getAuxDevices()takes the devices root as a parameter so the discovery logic istestable against a fake sysfs tree.
Tests
Adds table-driven tests for the discovery logic covering multi-function boards,
single-function GPUs, foreign-vendor siblings, devices on other slots, class
handling (VGA / 3D controller / NVSwitch), and the de-duplication of a function
reachable both as a sibling and via a consumer link. A second test asserts the
currently-bound driver is reported per function, which is what lets
BindToVFIODriver()skip functions already on the vfio driver.make assert-fmt goimports vet check-vendorandgo test ./pkg/nvpassthrough/pass; no dependency or vendor changes.