Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aarch64 Support #198

Open
8 of 9 tasks
retrage opened this issue Sep 7, 2022 · 10 comments
Open
8 of 9 tasks

aarch64 Support #198

retrage opened this issue Sep 7, 2022 · 10 comments

Comments

@retrage
Copy link
Contributor

retrage commented Sep 7, 2022

This issue tracks aarch64 support.

I am working on an experimental implementation of aarch64 support for Rust Hypervisor Firmware.
The code is available at:

Current Status

The changes in this branch have been tested to work using QEMU aarch64 virt.
Similar to x86_64, you can boot the Linux image by specifying the firmware with -kernel. The major changes are as follows:

  • Add target and linker scripts for aarch64.
  • Add BootInfo for FDT.
  • Support for PCI configuration space access using MMIO on aarch64
  • Add virtio-mmio driver (only for QEMU)

I will propose these changes in multiple PRs.

To-Do

Known Issues

This aarch64 support has the following issues.

Cloud Hypervisor is not yet supported.

This aarch64 support does not yet support Cloud Hypervisor. This is because I do not have an aarch64 machine that supports GICv3 or later and cannot test it. Please let me know if there is a good test environment available.

Update: It works with a custom Ubuntu bionic cloud image.

objcopy is required as post-processing after build.

QEMU aarch64 virt has an emulator loader that behaves differently depending on the type of binary passed with -kernel. When the binary is not ELF, QEMU executes the binary with the first address of the FDT passed in the x0 register. You need to run objcopy as a post-processing step to convert it to raw binary, as follows:
This post process is not needed by generating as a raw binary at link time. See eaed071.

Run QEMU with the binary as follows:

qemu-system-aarch64 \
  -machine virt \
  -cpu cortex-a53 \
  -m 8G \
  -nographic \
  -serial mon:stdio \
  -drive id=disk,file=$(AARCH64_IMG),if=none \
  -device virtio-blk-pci,drive=disk,disable-legacy=on \
  -kernel target/aarch64-unknown-none/debug/hypervisor-fw
@rbradford
Copy link
Member

Cool, perhaps @MrXinWang or @michael2012z could help. Maybe we could add GICv2 support to CH, not sure how intrusive that would be.

@michael2012z
Copy link
Member

Awesome!

I will reach in Slack for the test environment.

The problem of GICv2 is that it does not support MSI. But in CH the virtio-PCI requires MSI. So enabling GICv2 is not very meaningful, too little devices can be supported.

@retrage
Copy link
Contributor Author

retrage commented Sep 26, 2022

Here is some updates:
I'm working on support for the aarch64 CH using the test environment. (Thank you, @michael2012z!) With a custom Ubuntu bionic cloud image used in CH integration tests, it starts Linux kernel via GRUB:

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x413fd0c1][440/1959]
[    0.000000] Linux version 4.15.0-106-generic (buildd@bos02-arm64-071) (gcc ve
rsion 7.5.0 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)) #107-Ubuntu SMP Thu Jun 4 11:2
8:55 UTC 2020 (Ubuntu 4.15.0-106.107-generic 4.15.18)                           
[    0.000000] Machine model: linux,dummy-virt                                  
[    0.000000] debug: skip boot console de-registration.                        
[    0.000000] earlycon: pl11 at MMIO 0x0000000009000000 (options '')           
[    0.000000] bootconsole [pl11] enabled                                       
[    0.000000] efi: Getting EFI parameters from FDT:                            
[    0.000000] efi: EFI v2.80 by                                                
[    0.000000] efi:                                                             
[    0.000000] NUMA: No NUMA configuration found                                
[    0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x00000000fbffffff
]                                                                               
[    0.000000] NUMA: NODE_DATA [mem 0xf3fea900-0xf3fedbff]                      
[    0.000000] Zone ranges:                                                     
[    0.000000]   DMA      [mem 0x0000000040000000-0x00000000fbffffff]           
[    0.000000]   Normal   empty                                                 
[    0.000000] Movable zone start for each node                                 
[    0.000000] Early memory node ranges                                         
[    0.000000]   node   0: [mem 0x0000000040000000-0x00000000403fffff]
[    0.000000]   node   0: [mem 0x0000000040400000-0x000000004047ffff]
[    0.000000]   node   0: [mem 0x0000000040480000-0x00000000405d7fff]

However, Linux boot process is aborted by unhandled kernel paging request:

[    0.540102] Unable to handle kernel paging request at virtual address 4049457
0                                                                               
[    0.540102] Unable to handle kernel paging request at virtual address 4049457
0                                                                               
[    0.542886] Mem abort info:                                                  
[    0.542886] Mem abort info:                                                  
[    0.543972]   ESR = 0x86000005                                               
[    0.543972]   ESR = 0x86000005                                               
[    0.545142]   Exception class = IABT (current EL), IL = 32 bits              
[    0.545142]   Exception class = IABT (current EL), IL = 32 bits              
[    0.547437]   SET = 0, FnV = 0                                               
[    0.547437]   SET = 0, FnV = 0                                               
[    0.548624]   EA = 0, S1PTW = 0                                              
[    0.548624]   EA = 0, S1PTW = 0
[    0.549820] [0000000040494570] user address but active_mm is swapper
[    0.549820] [0000000040494570] user address but active_mm is swapper
[    0.552303] Internal error: Oops: 86000005 [#1] SMP
[    0.552303] Internal error: Oops: 86000005 [#1] SMP

It looks like the virtual address is in the EFI runtime services region, which is still not handled correctly.

Here is the working branch: https://github.com/retrage/rust-hypervisor-firmware/tree/aarch64-ch-support

Anyway, it's not so far from booting properly.

Update: It works with a custom Ubuntu bionic cloud image.

@retrage
Copy link
Contributor Author

retrage commented Mar 14, 2023

I noticed that Linux boot on CH/aarch64 has been broken since
88d7dd2. It seems to be a PCI related bug.

@rbradford
Copy link
Member

@retrage Are you only testing with CH or do you also have some QEMU support for aarch64?

@rbradford
Copy link
Member

rbradford commented Mar 14, 2023

This is the output I get when running with main:

root@ci-cloud-hypervisor-arm64-02:~/testing.rbradford/cloud-hypervisor# target/debug/cloud-hypervisor --serial tty --console off --kernel ~/testing.rbradford/rust-hypervisor-firmware/target/aarch64-unknown-none/release/hypervisor-fw --disk path=../jammy-server-cloudimg-arm64.raw --disk path=/tmp/ubuntu-cloudinit.img 

Booting with FDT
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=0 device=0 in slot=1
Found PCI device vendor=8086 device=d57 in slot=2
Found PCI device vendor=0 device=0 in slot=3
Found PCI device vendor=8086 device=d57 in slot=4
Found PCI device vendor=0 device=0 in slot=5
Found PCI device vendor=8086 device=d57 in slot=6
Found PCI device vendor=0 device=0 in slot=7
Found PCI device vendor=8086 device=d57 in slot=8
Found PCI device vendor=0 device=0 in slot=9
Found PCI device vendor=8086 device=d57 in slot=10
Found PCI device vendor=0 device=0 in slot=11
Found PCI device vendor=8086 device=d57 in slot=12
Found PCI device vendor=0 device=0 in slot=13
Found PCI device vendor=8086 device=d57 in slot=14
Found PCI device vendor=0 device=0 in slot=15
Found PCI device vendor=1af4 device=1042 in slot=16
Found PCI device vendor=0 device=0 in slot=17
Found PCI device vendor=1af4 device=1042 in slot=18
Found PCI device vendor=0 device=0 in slot=19
Found PCI device vendor=1af4 device=1042 in slot=20
Found PCI device vendor=0 device=0 in slot=21
Found PCI device vendor=1af4 device=1042 in slot=22
Found PCI device vendor=0 device=0 in slot=23
Found PCI device vendor=1af4 device=1042 in slot=24
Found PCI device vendor=0 device=0 in slot=25
Found PCI device vendor=1af4 device=1042 in slot=26
Found PCI device vendor=0 device=0 in slot=27
Found PCI device vendor=1af4 device=1042 in slot=28
Found PCI device vendor=0 device=0 in slot=29
Found PCI device vendor=1af4 device=1042 in slot=30
Found PCI device vendor=0 device=0 in slot=31
PCI Device: 0:16.0 1af4:1042
Bar: type=MemorySpace32 address=0x2ff80000 size=0x80000
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Updated BARs: type=MemorySpace32 address=2ff80000 size=80000
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Virtio block device configured. Capacity: 4612096 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: \EFI\BOOT\BOOTAA64.EFI

The PCI bus output looks very strange...if I go back to 411ca73 it still looks peculiar and I have the same issue with booting

root@ci-cloud-hypervisor-arm64-02:~/testing.rbradford/cloud-hypervisor# target/debug/cloud-hypervisor --serial tty --console off --kernel ~/testing.rbradford/rust-hypervisor-firmware/target/aarch64-unknown-none/release/hypervisor-fw --disk path=../jammy-server-cloudimg-arm64.raw --disk path=/tmp/ubuntu-cloudinit.img

Booting with FDT
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=0 device=0 in slot=1
Found PCI device vendor=8086 device=d57 in slot=2
Found PCI device vendor=0 device=0 in slot=3
Found PCI device vendor=8086 device=d57 in slot=4
Found PCI device vendor=0 device=0 in slot=5
Found PCI device vendor=8086 device=d57 in slot=6
Found PCI device vendor=0 device=0 in slot=7
Found PCI device vendor=8086 device=d57 in slot=8
Found PCI device vendor=0 device=0 in slot=9
Found PCI device vendor=8086 device=d57 in slot=10
Found PCI device vendor=0 device=0 in slot=11
Found PCI device vendor=8086 device=d57 in slot=12
Found PCI device vendor=0 device=0 in slot=13
Found PCI device vendor=8086 device=d57 in slot=14
Found PCI device vendor=0 device=0 in slot=15
Found PCI device vendor=1af4 device=1042 in slot=16
Found PCI device vendor=0 device=0 in slot=17
Found PCI device vendor=1af4 device=1042 in slot=18
Found PCI device vendor=0 device=0 in slot=19
Found PCI device vendor=1af4 device=1042 in slot=20
Found PCI device vendor=0 device=0 in slot=21
Found PCI device vendor=1af4 device=1042 in slot=22
Found PCI device vendor=0 device=0 in slot=23
Found PCI device vendor=1af4 device=1042 in slot=24
Found PCI device vendor=0 device=0 in slot=25
Found PCI device vendor=1af4 device=1042 in slot=26
Found PCI device vendor=0 device=0 in slot=27
Found PCI device vendor=1af4 device=1042 in slot=28
Found PCI device vendor=0 device=0 in slot=29
Found PCI device vendor=1af4 device=1042 in slot=30
Found PCI device vendor=0 device=0 in slot=31
PCI Device: 0:16.0 1af4:1042
Bar: type=MemorySpace32 address=2ff80000
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Virtio block device configured. Capacity: 4612096 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: /EFI/BOOT/BOOTAA64.EFI

@retrage
Copy link
Contributor Author

retrage commented Mar 14, 2023

I test it with CH only.

@retrage
Copy link
Contributor Author

retrage commented Mar 14, 2023

Here is output:

cloud-hypervisor/target/release/cloud-hypervisor --kernel rust-hypervisor-firmware/target/aarch64-unknown-none/debug/hypervisor-fw.bin --disk path=/home/retrage/workloads/bionic-server-cloudimg-arm64.raw --cpus boot=1 --memory size=4G --console off --serial tty -v -v -v --log-file ch-debug.log

Booting with FDT                                                                
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=0 device=0 in slot=1       
Found PCI device vendor=8086 device=d57 in slot=2
Found PCI device vendor=0 device=0 in slot=3       
Found PCI device vendor=8086 device=d57 in slot=4
Found PCI device vendor=0 device=0 in slot=5       
Found PCI device vendor=8086 device=d57 in slot=6
Found PCI device vendor=0 device=0 in slot=7       
Found PCI device vendor=8086 device=d57 in slot=8
Found PCI device vendor=0 device=0 in slot=9
Found PCI device vendor=8086 device=d57 in slot=10                  
Found PCI device vendor=0 device=0 in slot=11                                  
Found PCI device vendor=8086 device=d57 in slot=12
Found PCI device vendor=0 device=0 in slot=13
Found PCI device vendor=8086 device=d57 in slot=14
Found PCI device vendor=0 device=0 in slot=15
Found PCI device vendor=1af4 device=1042 in slot=16
Found PCI device vendor=0 device=0 in slot=17
Found PCI device vendor=1af4 device=1042 in slot=18
Found PCI device vendor=0 device=0 in slot=19
Found PCI device vendor=1af4 device=1042 in slot=20
Found PCI device vendor=0 device=0 in slot=21
Found PCI device vendor=1af4 device=1042 in slot=22
Found PCI device vendor=0 device=0 in slot=23
Found PCI device vendor=1af4 device=1042 in slot=24
Found PCI device vendor=0 device=0 in slot=25
Found PCI device vendor=1af4 device=1042 in slot=26
Found PCI device vendor=0 device=0 in slot=27
Found PCI device vendor=1af4 device=1042 in slot=28
Found PCI device vendor=0 device=0 in slot=29
Found PCI device vendor=1af4 device=1042 in slot=30
Found PCI device vendor=0 device=0 in slot=31
PCI Device: 0:16.0 1af4:1042
PANIC: panicked at 'attempt to add with overflow', src/pci.rs:286:36

@rbradford
Copy link
Member

The PCI device enumeration looks really wrong:

This is what I get for RV64 on QEMU (which will be using the same MMIO ECAM code.)

Starting on RV64 0x0 0x9fe00000 
Memory region 512MiB@0x80000000

Booting with FDT
Found PCI device vendor=1b36 device=8 in slot=0
Found PCI device vendor=1af4 device=1042 in slot=16
PCI Device: 0:16.0 1af4:1042
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x1000
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace64 address=0x0 size=0x4000
Bar: type=Unused address=0x0 size=0x0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=40000000 size=1000
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace64 address=40004000 size=4000
Updated BARs: type=Unused address=0 size=0
Virtio block device configured. Capacity: 262144 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: \EFI\BOOT\BOOTRISCV64.EFI

@rbradford
Copy link
Member

#233 fixes the erroneous bus reporting:

root@ci-cloud-hypervisor-arm64-02:~/testing.rbradford/cloud-hypervisor# target/debug/cloud-hypervisor --serial tty --console off --kernel ~/testing.rbradford/rust-hypervisor-firmware/target/aarch64-unknown-none/release/hypervisor-fw --disk path=../jammy-server-cloudimg-arm64.raw --disk path=/tmp/ubuntu-cloudinit.img

Booting with FDT
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=1af4 device=1042 in slot=1
Found PCI device vendor=1af4 device=1042 in slot=2
Found PCI device vendor=1af4 device=1044 in slot=3
PCI Device: 0:1.0 1af4:1042
Bar: type=MemorySpace32 address=0x2ff80000 size=0x80000
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Updated BARs: type=MemorySpace32 address=2ff80000 size=80000
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Virtio block device configured. Capacity: 4612096 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: \EFI\BOOT\BOOTAA64.EFI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants