Remove dependency on outdated cross images

[diconico07]

Current situation

We currently use cross to build our rust binaries for all currently supported architectures (x86_64, aarch64 and armv7), cross uses the cross-compilation ability of rust paired with a C/C++ toolchain for target architecture to be able to cope with C libraries.
These toolchains are distributed through container images, cross provided images for last release are based on Ubuntu 18.04, this version of Ubuntu is EOL and we need a more recent version of packages to be able to build akri with up to date dependencies. Cross next release (which is 4 month late based on their initial milestone plan) will upgrade those to Ubuntu 20.04, but that is not enough to build akri.

This leaves us with the possibility to either build our own up-to-date toolchain images (which doesn't seems to be an easy feat), or change the way we build our artifacts.

Use "native" compilation using docker buildkit

The first solution is to directly build the artifacts as part of our Dockerfile with a file like this one (result is untested), here based on openSUSE base image to more easily compare with next "solution":

ARG RUST_VERSION=1.70
FROM registry.opensuse.org/opensuse/bci/rust:${RUST_VERSION} AS build
RUN zypper -n install libopenssl-devel systemd-devel protobuf-devel libv4l-devel && zypper clean
COPY ./ /app
WORKDIR /app
RUN cargo build --release

FROM registry.opensuse.org/opensuse/tumbleweed:latest
ARG AKRI_COMPONENT
RUN zypper -n install libopenssl3 && zypper clean
COPY --from=build /app/target/release/${AKRI_COMPONENT} ./${AKRI_COMPONENT}
ENTRYPOINT ["./${AKRI_COMPONENT}"]

You then use docker buildx build --platform linux/amd64,linux/arm64 --build-arg AKRI_COMPONENT=agent . to build.

From my tests on my laptop (x86_64 with a i9-11950H CPU) a native build takes 8 minutes and a non-native one (aarch64) around 5 hours. I've been unable to build for armv7

Pros:

• Simple to setup
• Easily reproducible "at home"
• Integrate well with GitHub Actions
• Can take advantage of docker build cache to prevent rebuilding for every component
• Can integrate a generated SBOM (built from image scanning)

Cons:

• Unable to build armv7
• Extremely slow for non-native platforms

Use build.opensuse.org to build artifacts

This solution consists on using an external service to build the artifacts (namely build.opensuse.org), this build system
is better used by creating RPMs then building an image containing those RPMs, so you can find here an attempt for a RPM build for akri here: https://build.opensuse.org/package/show/home:nbelouin/akri. and an image for the agent here: https://build.opensuse.org/package/show/home:nbelouin/akri-agent

The build system has native x86_64 builders as well as native aarch64 builders, so x86_64 build took 12 minutes and aarch64 one took 12 minutes as well. armv7 build failed.
The RPM will get automatically rebuilt if a package dependency get updated (e.g. openssl update)

There is a possible GitHub integration (Rancher Elemental project use it for example), that allows to trigger independent builds for every PR and give feedback to GitHub (not yet tested).

Pros:

• Fast
• Generate very small images easily (around 57MB for the agent)
• Can integrate a generated SBOM (built from RPM build system)

Cons:

• Integration with GitHub Actions is not straightforward
• Unable to build armv7
• Not easily reproduced "at home"

[jbpaux]

Hi,
Really interesting subject. I used buildx qui a few times on other projects but never heard of this opensuse service.

Have you also had a look on xx project ? https://github.com/tonistiigi/xx#rust

And do we know what cause armv7 build to fail ? is it rust code ? a dependency ?

[diconico07]

Hi, Really interesting subject. I used buildx qui a few times on other projects but never heard of this opensuse service.

This openSUSE service is now used by kubernetes to build their artifacts. This is also what is used to build all openSUSE

Have you also had a look on xx project ? https://github.com/tonistiigi/xx#rust

I didn't know about the xx project, will take a look at it

And do we know what cause armv7 build to fail ? is it rust code ? a dependency ?

The armv7 build fails on the opcua crate build, so only the opcua discovery handler fails to build in fact.
It seems to be some kind of error during rust code compilation specific to armv7 flavor of rustc, I didn't file a bug on rustc upstream yet as I didn't take time to try to gather more information about the issue to file a meaningful bug-report for them.

[diconico07]

About armv7, upstream kubernetes no longer build for armv7 since kubernetes/kubernetes#115742, so I've been wondering if we have any user of that armv7 flavor or if it would be better to just stop providing armv7 builds if these are basically unused.

[jbpaux]

Looking at the downloads from GHCR there are still a few downloads (similar number as arm 64bit so maybe some people download all images ?).

[kate-goldenring]

I think we should follow suit with Kubernetes and remove v7 support especially if it prevents us from improving our build system.

@diconico07 I am a big fan of reducing container size. Does strategy 1 of using opensuse base and the docker buildx toolkit reduce image size too?

I think the RPM approach is too prohibitive for the local/maintainer experience. It is helpful being able to build images locally and test and run them. But we could potentially do both, publish the RPM containers and still maintain a set of dockerfiles for local builds

[diconico07]

The way I use to reduce the container size is based on RPM (the image will only have the RPM and its dependencies), so no the pure buildx image is not lighter, it could be made lighter, but this would require more work.

I however agree that not being able to build local images is bad, if we keep all the files simple to maintain (i.e RPM specfile, RPM based Dockerfile and local build Dockerfile) it shouldn't be an issue to have both ways.

For the v7 thing I created rust-lang/rust#115674 for the issue (it won't be hard to bring it back if this get fixed).