Superfly NVSwitch bringup

[sdake]

This issue tracks the bring-up of NVSwitch on the Superfly Platform and Infrastructure.

cc @draper7 @tqbf @mrkurt

[sdake]

Hi gang,

Installed kernel 6.1.0-9.

Using debian 11 bullseye 5.10-23 AMD kernel results in a segfault during SEV setup.

Running with older kernels that may have bugs fixed upstream burns my flow. I installed Debian's 6.1-9 kernel.

sdake@nice:~$ hostnamectl
   Static hostname: nice
         Icon name: computer-server
           Chassis: server
        Machine ID: d2c1a8d880ec4ce3b7e16d718ba2bd34
           Boot ID: d52758ded0d34bbe863a1afbc74806f2
  Operating System: Debian GNU/Linux 11 (bullseye)
            Kernel: Linux 6.1.0-9-amd64
      Architecture: x86-64

It appears older kernels cannot detect SEV settings in the UEFI firmware. the 6.1 kernel displays this:

ccp 0000:32:00.1: SEV: memory encryption not enabled by BIOS

[sdake]

Install kernel-6.1.0-0 on Debian 11.

The guide is missing two critical steps required to build Nvidia kernel modules. First, pause using the guide prior to sudo apt upgrade.

Add this to the pin list in /etc/apt/preferences

Package: linux-headers-6.1
Pin:release a=unstable
Pin-Priority: 1000

And install the kernel headers:

sudo apt install linux-headers-6.1.0-0.deb11.7-amd64

Then finish the guide and reboot.

sudo apt upgrade

^^ I am not sure the above sequence works completely. To get things moving, I set the following:

Set sources in /etc/apt/sources.list:

deb http://deb.debian.org/debian/ bullseye main contrib
deb http://deb.debian.org/debian/ bullseye-updates main contrib
deb http://deb.debian.org/debian bullseye-backports main contrib non-free
deb http://security.debian.org/debian-security bullseye-security main contrib
deb http://deb.debian.org/debian unstable main contrib non-free

deb-src http://deb.debian.org/debian/ bullseye main contrib
deb-src http://security.debian.org/debian-security bullseye-security main contrib
deb-src http://deb.debian.org/debian/ bullseye-updates main contrib
deb-src http://deb.debian.org/debian unstable main contrib non-free

Set pins in /etc/apt/preferences:

sdake@beast-06:~$ cat /etc/apt/preferences
Package: *
Pin: release a=bullseye
Pin-Priority: 500

Package: linux-image-amd64
Pin:release a=unstable
Pin-Priority: 1000

Package: *
Pin: release a=unstable
Pin-Priority: 100

Package: linux-headers-6.1
Pin:release a=unstable
Pin-Priority: 1000

Install 6.1 kernel (order matters):

sudo apt update
sudo apt install linux-headers-6.1.0-0.deb11.7-amd64
sudo apt upgrade

[sdake]

taking a look at the dmesg differences between a related platform that works, versus one the one under test:

sdake@beast-06 ~> sudo dmesg | cut -c16- > /tmp/dmesg-aw.txt
sdake@nice:~$ sudo dmesg | cut -c16- > /tmp/dmesg-fly.txt

[sdake]

Trying to answer what this kernel log means from the AW system:

AMD-Vi: Using global IVHD EFR:0x59f77efa2094ade, EFR2:0x0

The following resources were used to deduce the adronym IVHD:

I/O Virtualization Hardware Definition (IVHD): An IVHD describes the capabilities and configuration of IOMMU hardware as well as system I/O topology associated with each IOMMU hardware.

While the NVSwitch system does not. This led to the following content:
PCI Passthrough walkthrough
Deep dive into internals

Interesting finds along the way:
Optimus GPU manager
Forum for GPU passthrough BAR errors
Documentation change to korg.

[sdake]

Kernel Lore

[sdake]

I paused the comparison of the two kernel dmesg logs. It is evident that the Broadcom PEX88048 is not enumerating properly in Linux. None of the devices behind this bus type enumerate at all!

[sdake]

After more investigation, I found the PEX switches are PEX88096.

As part of the process of bringup, I was attempting to target Linux's PCI enumeration to function correctly.

I modified the file:

/etc/default/grub.d/aw.cfg

And set the boot flag pci=realloc,nocrs,nommconf. This seems to have blocked the bootup. I don't have console access.

Would someone mind modifying the file or removing it, and running sudo update-grub that has console access?

Thank you,
-steve

[sdake]

After @draper7 modified the dropin grub file, I tried a variety of the following. Unfortunately, the PCI bus does not enumerate properly.

Next steps:

1. try pci=realloc with pci=assign-busses,pci_scan_all,nocrs
2. try disabling ACPI
3. finish reading the kernel/drivers/pci source code.
4. Map the bus as it comes from an early PCI dump (as configured by BIOS).

it is viable and possible that a UEFI setting is misconfigured. I am running the latest 6.1 kernel, and find it unlikely it is the cause, although I will take a look through the default kernel config.

It is very clear the BroadCom PEX devices are failing to map.

I am not sure WHY they are failing to map. This is why I need the early pci dump.

I do think it would be helpful to accept other suggestions, as always.

sdake@nice:~$ sudo cat /etc/default/grub.d/aw.cfg
sudo: unable to resolve host nice: Name or service not known
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX modprobe.blacklist=nouveau biosdevname=0 net.ifnames=1 cgroup_enable=memory swapaccount=1 pci=assign-busses,pcie_scan_all,nocrs"

sdake@nice:~$ sudo dmesg | grep pci | grep failed
sudo: unable to resolve host nice: Name or service not known
[    0.769730] pci 0000:02:04.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.769733] pci 0000:02:08.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.769740] pci 0000:02:04.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.769742] pci 0000:02:08.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.769751] pci 0000:04:08.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769754] pci 0000:04:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769760] pci 0000:04:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769762] pci 0000:04:08.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769838] pci 0000:0a:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.769841] pci 0000:0a:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.769843] pci 0000:0b:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769847] pci 0000:0b:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769887] pci 0000:0e:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.769890] pci 0000:0e:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.769892] pci 0000:0f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769895] pci 0000:0f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770141] pci 0000:42:04.0: BAR 15: failed to assign [mem size 0x00900000 64bit pref]
[    0.770144] pci 0000:42:08.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770146] pci 0000:42:08.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770152] pci 0000:42:04.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770154] pci 0000:42:08.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770156] pci 0000:42:08.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770201] pci 0000:47:00.0: BAR 15: failed to assign [mem size 0x00900000 64bit pref]
[    0.770204] pci 0000:47:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770207] pci 0000:48:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770209] pci 0000:48:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770210] pci 0000:48:18.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770212] pci 0000:48:1c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770216] pci 0000:48:1c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770218] pci 0000:48:18.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770219] pci 0000:48:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770221] pci 0000:48:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770288] pci 0000:4e:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770290] pci 0000:4e:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770292] pci 0000:4e:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770294] pci 0000:4e:00.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770295] pci 0000:4f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770297] pci 0000:4f:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770299] pci 0000:4f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770301] pci 0000:4f:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770597] pci 0000:90:03.1: BAR 13: failed to assign [io  size 0x6000]
[    0.770600] pci 0000:90:03.1: BAR 13: failed to assign [io  size 0x6000]
[    0.770601] pci 0000:91:00.0: BAR 13: failed to assign [io  size 0x6000]
[    0.770603] pci 0000:91:00.0: BAR 13: failed to assign [io  size 0x6000]
[    0.770606] pci 0000:92:04.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770608] pci 0000:92:04.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770610] pci 0000:92:08.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770612] pci 0000:92:08.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770614] pci 0000:92:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.770616] pci 0000:92:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770618] pci 0000:92:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770620] pci 0000:92:04.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770622] pci 0000:92:04.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770624] pci 0000:92:08.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770625] pci 0000:92:08.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770627] pci 0000:92:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770628] pci 0000:92:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770630] pci 0000:92:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.770632] pci 0000:93:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.770633] pci 0000:93:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.770637] pci 0000:94:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770640] pci 0000:94:08.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770642] pci 0000:94:08.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770644] pci 0000:94:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770646] pci 0000:94:0c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770647] pci 0000:94:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770649] pci 0000:94:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770650] pci 0000:94:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770652] pci 0000:94:0c.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770654] pci 0000:94:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770656] pci 0000:94:0c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770658] pci 0000:94:0c.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770660] pci 0000:94:08.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770662] pci 0000:94:08.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770663] pci 0000:94:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770665] pci 0000:94:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770667] pci 0000:94:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770668] pci 0000:94:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770743] pci 0000:9a:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770744] pci 0000:9a:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770746] pci 0000:9a:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770748] pci 0000:9a:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770749] pci 0000:9a:00.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770751] pci 0000:9a:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770753] pci 0000:9b:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770754] pci 0000:9b:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770756] pci 0000:9b:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770758] pci 0000:9b:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770759] pci 0000:9b:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770761] pci 0000:9b:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770801] pci 0000:9e:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770803] pci 0000:9e:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770804] pci 0000:9e:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770806] pci 0000:9e:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770807] pci 0000:9e:00.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770809] pci 0000:9e:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770811] pci 0000:9f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770813] pci 0000:9f:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770814] pci 0000:9f:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770816] pci 0000:9f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770817] pci 0000:9f:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770819] pci 0000:9f:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771036] pci 0000:d0:03.1: BAR 13: failed to assign [io  size 0x6000]
[    0.771038] pci 0000:d0:03.1: BAR 13: failed to assign [io  size 0x6000]
[    0.771040] pci 0000:d1:00.0: BAR 13: failed to assign [io  size 0x6000]
[    0.771041] pci 0000:d1:00.0: BAR 13: failed to assign [io  size 0x6000]
[    0.771043] pci 0000:d2:04.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.771045] pci 0000:d2:04.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.771047] pci 0000:d2:08.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.771049] pci 0000:d2:08.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.771051] pci 0000:d2:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.771053] pci 0000:d2:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771054] pci 0000:d2:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771056] pci 0000:d2:04.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.771058] pci 0000:d2:04.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.771059] pci 0000:d2:08.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.771061] pci 0000:d2:08.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.771063] pci 0000:d2:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771065] pci 0000:d2:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771066] pci 0000:d2:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.771068] pci 0000:d3:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.771069] pci 0000:d3:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.771072] pci 0000:d4:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771074] pci 0000:d4:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771076] pci 0000:d4:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771078] pci 0000:d4:0c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771080] pci 0000:d4:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771081] pci 0000:d4:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771082] pci 0000:d4:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771084] pci 0000:d4:0c.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771086] pci 0000:d4:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771088] pci 0000:d4:0c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771089] pci 0000:d4:0c.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771090] pci 0000:d4:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771092] pci 0000:d4:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771094] pci 0000:d4:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771095] pci 0000:d4:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771097] pci 0000:d4:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771172] pci 0000:da:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.771174] pci 0000:da:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.771176] pci 0000:da:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771177] pci 0000:da:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.771179] pci 0000:da:00.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.771181] pci 0000:da:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771182] pci 0000:db:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771184] pci 0000:db:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771185] pci 0000:db:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771187] pci 0000:db:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771189] pci 0000:db:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771190] pci 0000:db:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771230] pci 0000:de:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.771231] pci 0000:de:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.771233] pci 0000:de:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771235] pci 0000:de:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.771236] pci 0000:de:00.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.771238] pci 0000:de:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771240] pci 0000:df:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771241] pci 0000:df:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771243] pci 0000:df:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771244] pci 0000:df:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771246] pci 0000:df:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771247] pci 0000:df:10.0: BAR 13: failed to assign [io  size 0x1000]

[sdake]

Gigabyte Manual

[sdake]

Gigabyte Block Diagram

[sdake]

After @draper7 modified the dropin grub file, I tried a variety of the following. Unfortunately, the PCI bus does not enumerate properly.

Next steps:

1. try pci=realloc with pci=assign-busses,pci_scan_all,nocrs
2. try disabling ACPI
3. finish reading the kernel/drivers/pci source code.
4. Map the bus as it comes from an early PCI dump (as configured by BIOS).

it is viable and possible that a UEFI setting is misconfigured. I am running the latest 6.1 kernel, and find it unlikely it is the cause, although I will take a look through the default kernel config.

It is very clear the BroadCom PEX devices are failing to map.

I am not sure WHY they are failing to map. This is why I need the early pci dump.

I do think it would be helpful to accept other suggestions, as always.

sdake@nice:~$ sudo cat /etc/default/grub.d/aw.cfg
sudo: unable to resolve host nice: Name or service not known
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX modprobe.blacklist=nouveau biosdevname=0 net.ifnames=1 cgroup_enable=memory swapaccount=1 pci=assign-busses,pcie_scan_all,nocrs"

sdake@nice:~$ sudo dmesg | grep pci | grep failed
sudo: unable to resolve host nice: Name or service not known
[    0.769730] pci 0000:02:04.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.769733] pci 0000:02:08.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.769740] pci 0000:02:04.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.769742] pci 0000:02:08.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.769751] pci 0000:04:08.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769754] pci 0000:04:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769760] pci 0000:04:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769762] pci 0000:04:08.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769838] pci 0000:0a:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.769841] pci 0000:0a:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.769843] pci 0000:0b:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769847] pci 0000:0b:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769887] pci 0000:0e:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.769890] pci 0000:0e:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.769892] pci 0000:0f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.769895] pci 0000:0f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770141] pci 0000:42:04.0: BAR 15: failed to assign [mem size 0x00900000 64bit pref]
[    0.770144] pci 0000:42:08.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770146] pci 0000:42:08.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770152] pci 0000:42:04.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770154] pci 0000:42:08.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770156] pci 0000:42:08.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770201] pci 0000:47:00.0: BAR 15: failed to assign [mem size 0x00900000 64bit pref]
[    0.770204] pci 0000:47:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770207] pci 0000:48:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770209] pci 0000:48:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770210] pci 0000:48:18.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770212] pci 0000:48:1c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770216] pci 0000:48:1c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770218] pci 0000:48:18.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770219] pci 0000:48:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770221] pci 0000:48:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770288] pci 0000:4e:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770290] pci 0000:4e:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770292] pci 0000:4e:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770294] pci 0000:4e:00.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770295] pci 0000:4f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770297] pci 0000:4f:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770299] pci 0000:4f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770301] pci 0000:4f:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770597] pci 0000:90:03.1: BAR 13: failed to assign [io  size 0x6000]
[    0.770600] pci 0000:90:03.1: BAR 13: failed to assign [io  size 0x6000]
[    0.770601] pci 0000:91:00.0: BAR 13: failed to assign [io  size 0x6000]
[    0.770603] pci 0000:91:00.0: BAR 13: failed to assign [io  size 0x6000]
[    0.770606] pci 0000:92:04.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770608] pci 0000:92:04.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770610] pci 0000:92:08.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770612] pci 0000:92:08.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770614] pci 0000:92:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.770616] pci 0000:92:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770618] pci 0000:92:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770620] pci 0000:92:04.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770622] pci 0000:92:04.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770624] pci 0000:92:08.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770625] pci 0000:92:08.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770627] pci 0000:92:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770628] pci 0000:92:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770630] pci 0000:92:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.770632] pci 0000:93:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.770633] pci 0000:93:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.770637] pci 0000:94:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770640] pci 0000:94:08.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770642] pci 0000:94:08.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770644] pci 0000:94:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770646] pci 0000:94:0c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770647] pci 0000:94:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770649] pci 0000:94:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770650] pci 0000:94:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770652] pci 0000:94:0c.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770654] pci 0000:94:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770656] pci 0000:94:0c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770658] pci 0000:94:0c.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770660] pci 0000:94:08.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770662] pci 0000:94:08.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770663] pci 0000:94:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770665] pci 0000:94:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770667] pci 0000:94:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770668] pci 0000:94:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770743] pci 0000:9a:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770744] pci 0000:9a:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770746] pci 0000:9a:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770748] pci 0000:9a:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770749] pci 0000:9a:00.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770751] pci 0000:9a:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770753] pci 0000:9b:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770754] pci 0000:9b:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770756] pci 0000:9b:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770758] pci 0000:9b:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770759] pci 0000:9b:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770761] pci 0000:9b:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770801] pci 0000:9e:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.770803] pci 0000:9e:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.770804] pci 0000:9e:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770806] pci 0000:9e:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.770807] pci 0000:9e:00.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.770809] pci 0000:9e:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770811] pci 0000:9f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770813] pci 0000:9f:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770814] pci 0000:9f:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.770816] pci 0000:9f:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.770817] pci 0000:9f:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.770819] pci 0000:9f:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771036] pci 0000:d0:03.1: BAR 13: failed to assign [io  size 0x6000]
[    0.771038] pci 0000:d0:03.1: BAR 13: failed to assign [io  size 0x6000]
[    0.771040] pci 0000:d1:00.0: BAR 13: failed to assign [io  size 0x6000]
[    0.771041] pci 0000:d1:00.0: BAR 13: failed to assign [io  size 0x6000]
[    0.771043] pci 0000:d2:04.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.771045] pci 0000:d2:04.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.771047] pci 0000:d2:08.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.771049] pci 0000:d2:08.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.771051] pci 0000:d2:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.771053] pci 0000:d2:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771054] pci 0000:d2:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771056] pci 0000:d2:04.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.771058] pci 0000:d2:04.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.771059] pci 0000:d2:08.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.771061] pci 0000:d2:08.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.771063] pci 0000:d2:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771065] pci 0000:d2:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771066] pci 0000:d2:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.771068] pci 0000:d3:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.771069] pci 0000:d3:00.0: BAR 13: failed to assign [io  size 0x4000]
[    0.771072] pci 0000:d4:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771074] pci 0000:d4:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771076] pci 0000:d4:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771078] pci 0000:d4:0c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771080] pci 0000:d4:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771081] pci 0000:d4:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771082] pci 0000:d4:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771084] pci 0000:d4:0c.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771086] pci 0000:d4:0c.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771088] pci 0000:d4:0c.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771089] pci 0000:d4:0c.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771090] pci 0000:d4:08.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771092] pci 0000:d4:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771094] pci 0000:d4:04.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771095] pci 0000:d4:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771097] pci 0000:d4:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771172] pci 0000:da:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.771174] pci 0000:da:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.771176] pci 0000:da:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771177] pci 0000:da:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.771179] pci 0000:da:00.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.771181] pci 0000:da:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771182] pci 0000:db:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771184] pci 0000:db:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771185] pci 0000:db:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771187] pci 0000:db:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771189] pci 0000:db:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771190] pci 0000:db:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771230] pci 0000:de:00.0: BAR 14: failed to assign [mem size 0x00300000]
[    0.771231] pci 0000:de:00.0: BAR 15: failed to assign [mem size 0x00300000 64bit pref]
[    0.771233] pci 0000:de:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771235] pci 0000:de:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.771236] pci 0000:de:00.0: BAR 15: failed to assign [mem size 0x00100000 64bit pref]
[    0.771238] pci 0000:de:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771240] pci 0000:df:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771241] pci 0000:df:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771243] pci 0000:df:10.0: BAR 13: failed to assign [io  size 0x1000]
[    0.771244] pci 0000:df:10.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.771246] pci 0000:df:10.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[    0.771247] pci 0000:df:10.0: BAR 13: failed to assign [io  size 0x1000]

The real problem with the above kernel command line is ACPI's bus configuration was not honored, so the PCI busses were "ignored". After reverting, same problem persists.