Add Qemu/integration platform (#222)

* Copy paste some stuff over for qemu

WIP commit to integrate a dedicated integration
target based of generic arm Qemu and to fix
the broken simulator

Signed-off-by: Marvin Drees <[email protected]>

* Add 32bit and 64bit qemu platform folders

Add basic Qemu platform folders to allow both
integration and the simulator to be dedicated
targets, also allow testing on both 32bit and 64bit

Signed-off-by: Marvin Drees <[email protected]>

* Add block device support to loader

Make the boot/ directory a go submodule to be able to
use u-root as a module dependency and add support
to also mount block devices and not only MTD flash.

Signed-off-by: Marvin Drees <[email protected]>

* Switch from u-root to gobusybox+u-root

Because u-root wasn't able to properly create the
busybox binary now that we use go modules I switched to using
gobusybox instead which in turn wasn't able to work properly
with direct code dependencies in u-bmc on u-root.

The loader is already in its own submodule to account
for that, othre parts still need adoption now!

Signed-off-by: Marvin Drees <[email protected]>

* Restructure taskfile flow and directories

Qemu needs different components from the buildsystem
than e.g. ast socs need so we make more use of the
Platform.yml file now and split tasks up further.
In addition we don't need go vendoring anymore so
removed these from the gitignores and taskfiles.
Finally the streamline the linux configs for qemu
a bit more. 32bit still has some issues for now.

Signed-off-by: Marvin Drees <[email protected]>

* Update go.mod

Update go.mod and go.sum to go 1.17

Signed-off-by: Marvin Drees <[email protected]>

* Fix paths in taskfiles

Some paths were wrong and have been fixed now.
This commit is part of adding Qemu support
and very much WIP. The added kexec binary is
only for testing purposes and will be removed
soon.

Signed-off-by: Marvin Drees <[email protected]>

* Add boot/mounter

In order to account for future rootfs layouts
that need a readonly and readwrite part a new
mounter is introduced which is loaded by the
runtime kernel initramfs and executes the main init.

Signed-off-by: Marvin Drees <[email protected]>

* Add boot/cpio to pack the initrd

The boot/cpio tool uses u-roots cpio library
to assemble an initrd for both the LinuxBoot
and runtime kernel. This removes the need for
and external cpio command and makes creating the
initrds much easier.

Signed-off-by: Marvin Drees <[email protected]>

* Fix boot/loader implementation of kexec

The loader tool can now be used in both the LinuxBoot
and runtime initrd and does not rely on having a kexec
binary in the image as we already got proper kexec_load
support on both arm and arm64. On arm64 the loader tries
kexec_file_load first and falls back to kexec_load if it
did not succeed.

Signed-off-by: Marvin Drees <[email protected]>

* Remove nested go.mod

For some reason gobusybox now works without the workaround
of having a nested go.mod. I won't question it...

Signed-off-by: Marvin Drees <[email protected]>

* Update CircleCI to go 1.17.1

Update the circleci config to remove some unnecessary
steps like go vendor and update the go version to match
the one marked in the go.mod files

Signed-off-by: Marvin Drees <[email protected]>

* Update README

Signed-off-by: Marvin Drees <[email protected]>

* Satisfy gofmt

As we updated go to 1.17 we got slightly different
requirements with gofmt, those are now fixed.

Signed-off-by: Marvin Drees <[email protected]>

* Change qemu-virt-a72 toolchain

As CircleCI and regular Ubuntu images don't feature a
musl toolchain we switch back to gnu for the kernel.

Signed-off-by: Marvin Drees <[email protected]>

* Compile erofs-utils in CircleCI

Because the current CircleCI image does not contain
a recent enough erofs-utils version we need to compile
one from source there.

Signed-off-by: Marvin Drees <[email protected]>

Author: MDr164

.circleci/config.yml

@@ -31,10 +31,8 @@ workflows:
 jobs:
   clean-code:
     docker:
-      - image: cimg/go:1.16.7
+      - image: cimg/go:1.17.1
     working_directory: ~/go/src/github.com/u-root/u-bmc
-    environment:
-      - CGO_ENABLED: 0
     steps:
       - checkout
       - run:
@@ -48,26 +46,21 @@ jobs:
           command: |
             ssh-keygen -t rsa -b 4096 -C "testkey" -N "testpass" -f config/ssh_keys
             touch i_agree_to_the_acme_terms
-            echo quanta-f06-leopard-ddr3 > TARGET
+            echo qemu-virt-a72 > TARGET
             task config:generate
-      - run:
-          name: vendor
-          command: go mod vendor
       - run:
           name: vet
-          command: go vet $(go list ./... | grep -v vendor/)
+          command: go vet $(go list ./...)
       - run:
           name: gofmt
-          command: test -z "$(gofmt -s -l $(go list ./... | grep -v vendor/ | cut -f 4- -d '/'))"
+          command: test -z "$(gofmt -s -l $(go list ./... | cut -f 4- -d '/'))"
       - run:
           name: ineffassign
           command: ineffassign .
   test:
     docker:
-      - image: cimg/go:1.16.7
+      - image: cimg/go:1.17.1
     working_directory: ~/go/src/github.com/u-root/u-bmc
-    environment:
-      - CGO_ENABLED: 0
     steps:
       - checkout
       - run:
@@ -78,7 +71,7 @@ jobs:
           command: |
             ssh-keygen -t rsa -b 4096 -C "testkey" -N "testpass" -f config/ssh_keys
             touch i_agree_to_the_acme_terms
-            echo quanta-f06-leopard-ddr3 > TARGET
+            echo qemu-virt-a72 > TARGET
             task config:generate
       - run:
           name: Test all
@@ -88,10 +81,8 @@ jobs:
           command: task coverage
   race:
     docker:
-      - image: cimg/go:1.16.7
+      - image: cimg/go:1.17.1
     working_directory: ~/go/src/github.com/u-root/u-bmc
-    environment:
-      - CGO_ENABLED: 1
     steps:
       - checkout
       - run:
@@ -102,17 +93,15 @@ jobs:
           command: |
             ssh-keygen -t rsa -b 4096 -C "testkey" -N "testpass" -f config/ssh_keys
             touch i_agree_to_the_acme_terms
-            echo quanta-f06-leopard-ddr3 > TARGET
+            echo qemu-virt-a72 > TARGET
             task config:generate
       - run:
           name: Race detector
           command: task race
   compile:
     docker:
-      - image: cimg/go:1.16.7
+      - image: cimg/go:1.17.1
     working_directory: ~/go/src/github.com/u-root/u-bmc
-    environment:
-      - CGO_ENABLED: 0
     steps:
       - checkout
       - run:
@@ -123,43 +112,57 @@ jobs:
           command: |
             ssh-keygen -t rsa -b 4096 -C "testkey" -N "testpass" -f config/ssh_keys
             touch i_agree_to_the_acme_terms
-            echo quanta-f06-leopard-ddr3 > TARGET
+            echo qemu-virt-a72 > TARGET
             task config:generate
       - run:
           name: build all commands
           command: |
             cd cmd
             go install -a ./...
             cd ../
-            go install -a ./platform/quanta-f06-leopard-ddr3/cmd/uinit
+            go install -a ./platform/qemu-virt-a72/cmd/uinit
   image:
     docker:
-      - image: cimg/go:1.16.7
+      - image: cimg/go:1.17.1
     working_directory: ~/go/src/github.com/u-root/u-bmc
-    environment:
-      - CGO_ENABLED: 0
     steps:
       - checkout
       - run:
           name: Install dependencies
-          # Workaround for u-root only needed until proper modules support is in place
           command: |
-            sudo apt-get update
-            sudo apt-get install gcc-arm-none-eabi mtd-utils fakeroot flex bison device-tree-compiler bc libssl-dev zlib1g-dev libglib2.0-dev libpixman-1-dev unzip cpio patch libelf-dev qemu-kvm
+            sudo apt update
+            sudo apt install gcc-aarch64-linux-gnu fakeroot flex bison bc patch libssl-dev zlib1g-dev libelf-dev qemu-kvm
             GO111MODULE=off go get github.com/u-root/u-root
             sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin
+      # Because the included version is too old and lacks the -U flag...
+      - run:
+          name: Compile erofs-utils
+          command: |
+            sudo apt install libtool liblz4-dev uuid-dev
+            cd ~/
+            curl -O https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/snapshot/erofs-utils-1.3.tar.gz
+            tar -xzf erofs-utils-1.3.tar.gz
+            cd erofs-utils-1.3
+            ./autogen.sh
+            ./configure
+            make
+            sudo make install
+            sudo ln -sf /usr/local/bin/mkfs.erofs /usr/bin/mkfs.erofs
       - run:
           name: Create test key
           command: |
             ssh-keygen -t rsa -b 4096 -C "testkey" -N "testpass" -f config/ssh_keys
             touch i_agree_to_the_acme_terms
-            echo quanta-f06-leopard-ddr3 > TARGET
+            echo qemu-virt-a72 > TARGET
       - run:
           name: Build image
           command: task build
       - store_artifacts:
-          path: build/flash.img
-          destination: flash.img
+          path: build/img/rootfs.img
+          destination: rootfs.img
+      - store_artifacts:
+          path: build/linux/zImage.boot
+          destination: zImage.boot
       - run:
           name: Run integration tests
           command: task integration

.gitignore

@@ -1,4 +1,3 @@
-vendor/*
 build/*
 .task/*
 i_agree_to_the_acme_terms