Configuration files reorg

[gctucker]

Now that all the Jenkins-related files have been moved out of kernelci-core and into kernelci-jenkins, we should consider reorganising the configuration files we have left.

I've created a series of PRs to do that:

• T25969 config: move all *.yaml files to new config directory #585 for the YAML core config
• T25969 config: move all LAVA job templates to config/lava #586 for the Jinja2 templates for LAVA
• T25969 config/jobs/k8s: move k8s job templates to new directory #587 for the Kubernetes Jinja2 templates
• T25969 config/rootfs/debos: move all debos files to new directory #588 for the rootfs build scripts (debos...)
• config/docker: move all Dockerfiles to config/docker #600 for the Docker files

When all merged together, it gives us the config tree shown below. This could more easily be moved to a separate repository if we wanted to. Also, having all the YAML config files in a dedicated core directory means we can more easily add some and merge all the configuration data into a single namespace, to be able to break things down arbitrarily. For example, we could split out the kselftest and LTP definitions from test-definitions.yaml, or have specific files for the Android and stable list of configs split out of build-configs.yaml. It should also make it more flexible for other users of the kernelci Python package to tailor their configuration files for their use-cases.

Before we proceed with this:

• Are there any other files in kernelci-core or elsewhere that should be included in this config file tree?
• Should the tree be shaped differently? (e.g. should lava and k8s templates be grouped under jobs?)
• Are there any files that should not be moved to the config tree? (e.g. should we keep the Docker files tied to kernelci-core?)

$ tree config
config
├── core
│   ├── build-configs.yaml
│   ├── db-configs.yaml
│   ├── lab-configs.yaml
│   ├── rootfs-configs.yaml
│   └── test-configs.yaml
├── docker
│   ├── build-and-push.sh
│   ├── build-base
│   │   └── Dockerfile
│   ├── build-k8s
│   │   └── Dockerfile
│   ├── clang-10
│   │   └── Dockerfile
│   ├── clang-11
│   │   └── Dockerfile
│   ├── debos
│   │   └── Dockerfile
│   ├── dt-validation
│   │   └── Dockerfile
│   ├── gcc-8_arc
│   │   └── Dockerfile
│   ├── gcc-8_arm
│   │   └── Dockerfile
│   ├── gcc-8_arm64
│   │   └── Dockerfile
│   ├── gcc-8_mips
│   │   └── Dockerfile
│   ├── gcc-8_riscv64
│   │   └── Dockerfile
│   ├── gcc-8_x86
│   │   └── Dockerfile
│   └── kcidb
│       └── Dockerfile
├── k8s
│   ├── gen.py
│   ├── job-build.jinja2
│   └── wait.py
├── lava
│   ├── base
│   │   ├── kernel-ci-base.jinja2
│   │   └── kernel-ci-base-tftp-deploy.jinja2
│   ├── baseline
│   │   ├── baseline.jinja2
│   │   ├── generic-barebox-tftp-nfs-baseline-template.jinja2
│   │   ├── generic-barebox-tftp-ramdisk-baseline-template.jinja2
│   │   ├── generic-depthcharge-tftp-nfs-baseline-template.jinja2
│   │   ├── generic-depthcharge-tftp-ramdisk-baseline-template.jinja2
│   │   ├── generic-fastboot-baseline-template.jinja2
│   │   ├── generic-grub-tftp-nfs-baseline-template.jinja2
│   │   ├── generic-grub-tftp-ramdisk-baseline-template.jinja2
│   │   ├── generic-qemu-baseline-template.jinja2
│   │   ├── generic-uboot-tftp-nfs-baseline-template.jinja2
│   │   └── generic-uboot-tftp-ramdisk-baseline-template.jinja2
│   ├── boot
│   │   ├── generic-barebox-tftp-ramdisk-boot-template.jinja2
│   │   ├── generic-depthcharge-tftp-ramdisk-boot-template.jinja2
│   │   ├── generic-grub-tftp-ramdisk-boot-template.jinja2
│   │   ├── generic-ipxe-tftp-ramdisk-boot-template.jinja2
│   │   ├── generic-qemu-boot-template.jinja2
│   │   └── generic-uboot-tftp-ramdisk-boot-template.jinja2
│   ├── boot-fastboot
│   │   ├── generic-fastboot-boot-template.jinja2
│   │   └── mkbootimg.jinja2
│   ├── boot-nfs
│   │   ├── generic-barebox-tftp-nfs-template.jinja2
│   │   ├── generic-depthcharge-tftp-nfs-template.jinja2
│   │   ├── generic-grub-tftp-nfs-template.jinja2
│   │   ├── generic-ipxe-tftp-nfs-template.jinja2
│   │   └── generic-uboot-tftp-nfs-template.jinja2
│   ├── cros-ec
│   │   ├── cros-ec.jinja2
│   │   ├── generic-depthcharge-tftp-ramdisk-cros-ec-template.jinja2
│   │   └── generic-uboot-tftp-ramdisk-cros-ec-template.jinja2
│   ├── igt
│   │   ├── generic-depthcharge-tftp-ramdisk-igt.jinja2
│   │   ├── generic-grub-tftp-ramdisk-igt.jinja2
│   │   ├── generic-uboot-tftp-ramdisk-igt.jinja2
│   │   └── igt.jinja2
│   ├── kselftest
│   │   ├── generic-barebox-tftp-nfs-kselftest-template.jinja2
│   │   ├── generic-depthcharge-tftp-nfs-kselftest-template.jinja2
│   │   ├── generic-grub-tftp-nfs-kselftest-template.jinja2
│   │   ├── generic-uboot-tftp-nfs-kselftest-template.jinja2
│   │   └── kselftest.jinja2
│   ├── ltp
│   │   ├── generic-depthcharge-tftp-nfs-ltp-template.jinja2
│   │   ├── generic-grub-tftp-nfs-ltp-template.jinja2
│   │   ├── generic-uboot-tftp-nfs-ltp-template.jinja2
│   │   └── ltp.jinja2
│   ├── preempt-rt
│   │   ├── generic-uboot-tftp-nfs-preempt-rt-template.jinja2
│   │   └── preempt-rt.jinja2
│   ├── sleep
│   │   ├── generic-barebox-tftp-ramdisk-sleep-template.jinja2
│   │   ├── generic-depthcharge-tftp-ramdisk-sleep-template.jinja2
│   │   ├── generic-uboot-tftp-ramdisk-sleep-template.jinja2
│   │   ├── sleep.jinja2
│   │   ├── sleep.sh
│   │   ├── sleep.yaml
│   │   └── TODO
│   ├── usb
│   │   ├── generic-depthcharge-tftp-ramdisk-usb-template.jinja2
│   │   ├── generic-uboot-tftp-ramdisk-usb-template.jinja2
│   │   └── usb.jinja2
│   └── v4l2-compliance
│       ├── generic-depthcharge-tftp-ramdisk-v4l2-compliance-template.jinja2
│       ├── generic-qemu-v4l2-compliance-template.jinja2
│       ├── generic-uboot-tftp-ramdisk-v4l2-compliance-template.jinja2
│       └── v4l2-compliance.jinja2
└── rootfs
    └── debos
        ├── overlays
        │   ├── auto-login
        │   │   └── etc
        │   │       └── systemd
        │   │           └── system
        │   │               └── [email protected]
        │   │                   └── override.conf
        │   ├── baseline
        │   │   └── opt
        │   │       └── kernelci
        │   │           └── dmesg.sh
        │   ├── igt
        │   │   └── usr
        │   │       └── bin
        │   │           └── igt-parser.sh
        │   ├── minimal
        │   │   └── usr
        │   │       └── lib
        │   │           └── tmpfiles.d
        │   │               └── var-tmp.conf
        │   ├── networkd
        │   │   └── etc
        │   │       └── systemd
        │   │           └── network
        │   │               └── wired.network
        │   ├── prompt
        │   │   └── root
        │   └── v4l2
        │       └── usr
        │           └── bin
        │               └── v4l2-parser.sh
        ├── rootfs.yaml
        └── scripts
            ├── buster-cros-ec-tests.sh
            ├── buster-igt.sh
            ├── buster-ltp.sh
            ├── buster-v4l2.sh
            ├── create_initrd_ramdisk.sh
            ├── create_manifest.py
            ├── crush.sh
            ├── install-bootrr.sh
            ├── nothing.sh
            ├── setup-networking.sh
            ├── stretch-ltp.sh
            └── strip.sh

58 directories, 99 files

[broonie]

Makes sense to me.

[khilman]

I like it.

[patersonc]

I like the flexibility to be able to move the configs to another repo if needed. Grouping them all together now would make this much cleaner.

Will/Does KernelCI support test submission to board farms that don't use LAVA?
If so, perhaps the 'lava' job definitions could be in a 'tests' subdirectory.

[gctucker]

There is some very minimal support for non-LAVA labs at the moment. The ones currently in production typically use the backend API to check for new kernel builds every once in a while, then schedule their own tests and submit their results. We are working on enabling a publisher/subscriber mechanism for labs to receive descriptions of test jobs to run (or any kind of event).

We could potentially add support for other types of labs that can have jobs submitted to their APIs directly like we do with LAVA. In fact, Kubernetes falls into this category, for running tests in the cloud. However, those templates would most likely be specific to each system. The common test definitions to run the actual tests are hosted in test-definitions, and that should be usable by non-LAVA labs (in theory at least).

So if we start adding support for other types of "test environments", we can add other directories with templates for the data to send to them. We may group all that into a sub-directory, i.e. jobs/lava, jobs/k8s, jobs/labgrid or anything. I actually did that initially but it seemed a bit over the top with just lava and k8s. I think this might be something to potentially improve in the future though.

[gctucker]

All the PRs have been merged and the last changes are being rolled out in production today.