-
Notifications
You must be signed in to change notification settings - Fork 18
Add virtio-socket device implementation #161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
dreamliner787-9
wants to merge
19
commits into
main
Choose a base branch
from
virtio_socket_rebased
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 11 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
ab7bcdb
examples/virtio-socket: checkpoint port over from virtio example and …
dreamliner787-9 1f80fb4
virtio/socket: add virtio-socket device implementation
dreamliner787-9 2a42530
examples/virtio-socket: add test programs and checkpoint everything w…
dreamliner787-9 b5d3634
examples/virtio-socket: add missing virtio socket device to dts
dreamliner787-9 f98ed06
virtio/socket: clean ups + docs
dreamliner787-9 4f3fe6d
examples/virtio-socket: add odroidc4 support and make sender sleep lo…
dreamliner787-9 114dd45
virtio/socket: integrate tooling
dreamliner787-9 d29f72c
examples/virtio-socket: fix dts node name and improve logging of exam…
dreamliner787-9 77ffd07
examples/virtio-socket: update docs
dreamliner787-9 2d34827
examples/virtio: download correct kernel
dreamliner787-9 c192a99
examples/virtio-socket: improve debug prints
dreamliner787-9 e7f359c
vmm.mk: make indentation consistent
dreamliner787-9 bdc6b65
examples/virtio: converge with examples/virtio-socket
dreamliner787-9 c85b778
examples/virtio: fix licences
dreamliner787-9 dd6c196
virtio/socket: style + trailing spaces
dreamliner787-9 5d24ad7
examples/virtio: style + trailing whitespaces
dreamliner787-9 6d549ea
examples/virtio: trailing whitespaces
dreamliner787-9 0ed5653
examples/virtio: fix readme trailing whitespace
dreamliner787-9 4155c85
virtio/socket: fix a spec non-compliance issue where a reset packet m…
dreamliner787-9 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| # | ||
| # Copyright 2024, UNSW | ||
| # | ||
| # SPDX-License-Identifier: BSD-2-Clause | ||
| # | ||
|
|
||
| BUILD_DIR ?= build | ||
| export MICROKIT_CONFIG ?= debug | ||
|
|
||
| ifeq ($(strip $(MICROKIT_SDK)),) | ||
| $(error MICROKIT_SDK must be specified) | ||
| endif | ||
| export override MICROKIT_SDK:=$(abspath $(MICROKIT_SDK)) | ||
|
|
||
| # Both QEMU and Maaxboard uses the same guest device tree as the guest does not | ||
| # need to access any physical device. The only difference is that QEMU is GICv2 while | ||
| # the Maaxboard is GICv3. We apply a GIC overlay for the correct version on the board | ||
| # at build time. | ||
| # TODO: we should probably determine this better if we have lots of supported platforms. | ||
| ifeq ($(strip $(MICROKIT_BOARD)), maaxboard) | ||
| export UART_DRIVER := imx | ||
| export CPU := cortex-a55 | ||
| export GIC_DT_OVERLAY := gic_v3_overlay.dts | ||
| else ifeq ($(strip $(MICROKIT_BOARD)), odroidc4) | ||
| export UART_DRIVER := meson | ||
| export CPU := cortex-a55 | ||
| export GIC_DT_OVERLAY := gic_400_overlay.dts | ||
| else ifeq ($(strip $(MICROKIT_BOARD)), qemu_virt_aarch64) | ||
| export UART_DRIVER := arm | ||
| export CPU := cortex-a53 | ||
| export GIC_DT_OVERLAY := gic_v2_overlay.dts | ||
| QEMU := qemu-system-aarch64 | ||
| else | ||
| $(error Unsupported MICROKIT_BOARD given) | ||
| endif | ||
|
|
||
| # All platforms use the same Linux and initrd images. | ||
| export LINUX := 372b2c56f55f2d461e4570d79332298e0f7ecce4-linux | ||
| export INITRD := b6a276df6a0e39f76bc8950e975daa2888ad83df-rootfs.cpio.gz | ||
|
|
||
| export BUILD_DIR:=$(abspath $(BUILD_DIR)) | ||
| export MICROKIT_SDK:=$(abspath $(MICROKIT_SDK)) | ||
| export VIRTIO_EXAMPLE:=$(abspath .) | ||
|
|
||
| export OBJCOPY := llvm-objcopy | ||
| export TARGET := aarch64-none-elf | ||
| export CC := clang | ||
| export CC_USERLEVEL := zig cc | ||
| export LD := ld.lld | ||
| export AS := llvm-as | ||
| export AR := llvm-ar | ||
| export DTC := dtc | ||
| export RANLIB := llvm-ranlib | ||
| export MICROKIT_TOOL ?= $(MICROKIT_SDK)/bin/microkit | ||
| export SDDF=$(abspath ../../dep/sddf) | ||
| export LIBVMM=$(abspath ../../) | ||
|
|
||
| IMAGE_FILE := $(BUILD_DIR)/loader.img | ||
| REPORT_FILE := $(BUILD_DIR)/report.txt | ||
|
|
||
| all: $(IMAGE_FILE) | ||
|
|
||
| qemu $(IMAGE_FILE) $(REPORT_FILE) clean clobber: $(BUILD_DIR)/Makefile FORCE | ||
| $(MAKE) -C $(BUILD_DIR) MICROKIT_SDK=$(MICROKIT_SDK) $(notdir $@) | ||
|
|
||
| $(BUILD_DIR)/Makefile: virtio-socket.mk | ||
| mkdir -p $(BUILD_DIR) | ||
| cp virtio-socket.mk $@ | ||
|
|
||
| FORCE: | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,140 @@ | ||
| <!-- | ||
| Copyright 2025, UNSW | ||
| SPDX-License-Identifier: CC-BY-SA-4.0 | ||
| --> | ||
|
|
||
| # Using multiple virtIO devices with a Linux guest | ||
|
|
||
| This example demoes libvmm's VirtIO socket device implementation that allow | ||
| inter-guests communication without Ethernet or IP protocols. | ||
|
|
||
| We have two Linux VMs, a sender and receiver as follows: | ||
| ``` | ||
dreamliner787-9 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| --------- VM --------- --------- VM --------- | ||
| Userland: Userland: | ||
| vsock_recv.elf vsock_send.elf | ||
| ^ | | ||
| | v | ||
| Kernel: Kernel: | ||
| virtio MMIO vsock virtio MMIO vsock | ||
| ^ | | ||
| | v | ||
| --------- VMM -------- --------- VMM -------- | ||
| virtio vsock device /-- virtio vsock device | ||
| ^ / | ||
| | / | ||
| rx buffer <----------------/ rx buffer | ||
| ``` | ||
|
|
||
| In sender VM, `vsock_send.elf` will send 32k worth of data through virtIO socket | ||
| to `vsock_recv.elf` in the receiver VM. Then, the receiver VM will verify that the | ||
| data are all correct and both programs will quit. | ||
|
|
||
| Each virtIO socket device have a small receive buffer that the peer can write to | ||
| to send data. The buffer size is advertised to both the sender and receiver via the | ||
| `buf_alloc` field of the packet header for the guest driver to split up packets | ||
| as necessary. | ||
|
|
||
| Here is a demo of that process happening (receiver is red, sender is green): | ||
|  | ||
|
|
||
| The example currently works on the following platforms: | ||
|
|
||
| * QEMU virt AArch64 | ||
| * Avnet MaaXBoard | ||
| * HardKernel OdroidC4 | ||
|
|
||
| ### Metaprogram | ||
|
|
||
| Unlike the other examples, this one uses a metaprogram (`meta.py`) with | ||
| the [sdfgen](https://github.com/au-ts/microkit_sdf_gen) tooling to generate the | ||
| System Description File (SDF) and other necessary artefacts. Previously, | ||
| SDFs were written manually, along with C headers for sDDF-specific configurations, | ||
| but this approach was tedious and error-prone. Wit this tooling, we can describe | ||
dreamliner787-9 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| the system at a higher level, automating the generation of system-specific data. | ||
|
|
||
| ## Dependencies | ||
|
|
||
| In addition to the dependencies outlined in the top-level README, the following | ||
| dependencies are needed: | ||
| * sdfgen (for generating the System Description File with a metaprogram). | ||
|
|
||
| ### Linux | ||
|
|
||
| <!-- TODO bump sdfgen --> | ||
| On apt based Linux distributions run the following commands: | ||
| ```sh | ||
| pip3 install sdfgen==0.23.1 | ||
| ``` | ||
|
|
||
| If you get error: `externally-managed-environment` when installing via pip, instead run: | ||
| ```sh | ||
| pip3 install --break-system-packages sdfgen==0.23.1 | ||
| ``` | ||
|
|
||
| ### macOS | ||
|
|
||
| On macOS, you can install the dependencies via Homebrew: | ||
| ```sh | ||
| pip3 install sdfgen==0.23.1 | ||
| ``` | ||
|
|
||
| If you get error: `externally-managed-environment` when installing via pip, instead run: | ||
| ```sh | ||
| pip3 install --break-system-packages sdfgen==0.23.1 | ||
| ``` | ||
|
|
||
| ### Nix | ||
|
|
||
| The top-level `shell.nix` has everything necessary: | ||
| ```sh | ||
| nix-shell ../../shell.nix | ||
| ``` | ||
|
|
||
| ## Building | ||
|
|
||
| ```sh | ||
| make MICROKIT_BOARD=<BOARD> MICROKIT_SDK=/path/to/sdk | ||
| ``` | ||
|
|
||
| Where `<BOARD>` is one of: | ||
| * `qemu_virt_aarch64` | ||
| * `maaxboard` | ||
| * `odroidc4` | ||
|
|
||
| Other configuration options can be passed to the Makefile such as `CONFIG` | ||
| and `BUILD_DIR`, see the Makefile for details. | ||
|
|
||
| By default the build system fetches the Linux kernel and initrd images from | ||
| Trustworthy Systems' website. To use your own images, specify `LINUX` and/or | ||
| `INITRD`. For example: | ||
|
|
||
| ```sh | ||
| make MICROKIT_BOARD=qemu_virt_aarch64 MICROKIT_SDK=/path/to/sdk LINUX=/path/to/linux INITRD=/path/to/initrd | ||
| ``` | ||
|
|
||
| If you would like to simulate the QEMU board you can run the following command: | ||
| ```sh | ||
| make MICROKIT_BOARD=qemu_virt_aarch64 MICROKIT_SDK=/path/to/sdk qemu | ||
| ``` | ||
|
|
||
| This will build the example code as well as run the QEMU command to simulate a | ||
| system running the whole system. | ||
|
|
||
| ## Linux kernel notes | ||
| If you decide to BYO kernel, you will need to make sure these configuration | ||
| values were set when your kernel was built: | ||
| ``` | ||
| CONFIG_VSOCKETS=y | ||
| CONFIG_VSOCKETS_LOOPBACK=y | ||
| CONFIG_VIRTIO_VSOCKETS=y | ||
| CONFIG_VIRTIO_VSOCKETS_COMMON=y | ||
| ``` | ||
|
|
||
| It is recommended that you use at least Linux v6.13 as previous versions have an | ||
| issue where if you try to send more data than the underlying buffer can accomodate, | ||
| the kernel driver won't split up your data and the guests will hang forever waiting | ||
| for a transfer that will never take place to complete. | ||
|
|
||
| ## Hardware set up | ||
| Since this example does not touch any real hardware, no setup is necessary. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| <!-- | ||
| Copyright 2025, UNSW | ||
| SPDX-License-Identifier: CC-BY-SA-4.0 | ||
| --> | ||
|
|
||
| # Linux kernel image | ||
|
|
||
| ## Linux kernel | ||
|
|
||
| ### Details | ||
| * Image name: `linux` | ||
| * Config name: `linux_config` | ||
| * Git remote: https://github.com/torvalds/linux.git | ||
| * Tag: v6.13 (commit hash: `ffd294d346d185b70e28b1a28abe367bbfe53c04`) | ||
| * Toolchain: `aarch64-none-elf` | ||
| * Version: (Arm GNU Toolchain 12.2.Rel1 (Build arm-12.24)) 12.2.1 20221205 | ||
|
|
||
| You can also get the Linux config used after booting by running the following | ||
| command in userspace: `zcat /proc/config.gz`. This is a kernel minimally configured with a | ||
| IPv4 network stack + DNS + DHCP and virtIO device drivers. | ||
|
|
||
| ### Instructions for reproducing | ||
| ``` | ||
| git clone --depth 1 --branch v6.13 https://github.com/torvalds/linux.git | ||
| cp config linux/.config | ||
| make -C linux ARCH=arm64 CROSS_COMPILE=aarch64-none-elf- all -j$(nproc) | ||
| ``` | ||
|
|
||
| The path to the image is: `linux/arch/arm64/boot/Image`. | ||
|
|
||
| ## Buildroot RootFS image | ||
|
|
||
| ### Details | ||
| * Image name: `rootfs.cpio.gz` | ||
| * Config name: `buildroot_config` | ||
| * Version: 2022.08-rc2 | ||
|
|
||
| ### Instructions for reproducing | ||
|
|
||
| ``` | ||
| wget https://buildroot.org/downloads/buildroot-2022.08-rc2.tar.xz | ||
| tar xvf buildroot-2022.08-rc2.tar.xz | ||
| cp buildroot_config buildroot-2022.08-rc2/.config | ||
| make -C buildroot-2022.08-rc2 | ||
| ``` | ||
|
|
||
| The root filesystem will be located at: `buildroot-2022.08-rc2/output/images/rootfs.cpio.gz` along | ||
| with the other buildroot artefacts. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.