Proposal: Adopting Docker images as distribution/installation method

[flokoe]

Hi there,

I would like to propose adopting a new distribution/installation method in addition to the existing one via the ADD directive and making the content of the tarballs also available as Docker images so that we can use the COPY directive instead. This would have the following benefits:

• The version of Docker images can be easily pinned.
• Docker tags/digest can be auto-updated via Bots like Renovate/Dependabot out-of-the-box.
• COPY directives from images can be cached.
• The --link option for the COPY directive allows rebasing, which improves build time and caching.
• Architecture selection could be automated via multi-arch images.

I see two possibilities: one image per tarball, or a single image containing the contents of the tarballs in corresponding subdirectories.

• single image:
  • Version compatibility between files.
  • Single image to update.
  • The different tarballs are kind of "hidden" (not directly mentioned in the image name).
  • Single layer.
• image per tarball:
  • The tarballs are easier to differentiate.
  • Multiple layers.
  • Users have to ensure that each image has the same version, could lead to version mismatches.

It could look something like this:

# Singe image with digest, copying multiple "tarballs" into new image.
# A multi-arch image would automatically select the correct architecture.
FROM alpine:latest

COPY --link \
  --from=just-containers/s6-overlay:v3.2.0.0@sha256:17d70c7e8e374f772c0d0539f2e794b15701a50c48cc52cebea20d96e017c95f \
  /s6-overlay-noarch /s6-overlay-arch /


# Different images per tarball.
# A multi-arch image would automatically select the correct architecture.
FROM alpine:latest

COPY --link \
  --from=just-containers/s6-overlay-noarch:v3.2.0.0@sha256:933e09abf7298e4af67d18a54391b65c207e2181e143b5489e064b3078f37aed \
  / /

COPY --link \
  --from=just-containers/s6-overlay-arch:v3.2.0.0@sha256:4d6c5e083fb05bb2700740cc0a2f26e1307afac63da3ccf3d81efa0a0e8375f2 \
  / /

File permissions are preserved and directories are merged. Any conflicts are resolved in favor of the content being added.

I would love to hear your thoughts on this. If you are interested, I could submit a PR with the necessary Dockerfiles and GitHub Actions workflows to automate the builds.

[skarnet]

Sorry, I have exactly zero available bandwidth at the moment, I'll get back to you later. @jprjr, if you're around, can you please take a look and say if it's reasonable and doesn't require too much work?

[flokoe]

Sorry, I have exactly zero available bandwidth at the moment, I'll get back to you later. @jprjr, if you're around, can you please take a look and say if it's reasonable and doesn't require too much work?

No worries. Like I said, I am happy to provide all the necessary Dockerfiles, workflows, docs etc. if you are interested.

[crazy-max]

Seems similar to this proposal #386 (comment) (dockerfile can be seen here: https://github.com/crazy-max/docker-alpine-s6/blob/master/Dockerfile)

[skarnet]

@flokoe Okay, sorry for the delay. So, I have a few comments about the benefits of that approach:

• Versions can be pinned: see, that's a direct drawback for me. Pinning is a quick and dirty fix that should be temporary but most often overstays its welcome and accumulates tech debt. I don't want people to feel like they have to pin a version of s6-overlay. I'd rather make it possible for them to trust that upgrades won't break their workflow. So, ease of pinning isn't an argument for me, on the contrary.
• Docker tags/digests can be updated by bots: I have no idea what it means 😅 Can you please explain like I'm five?
• COPY allows caching etc: yeah, sure, but the impact of this should be pretty small. Are people spending their days building images, rather than running them?
• Architecture selection: so the thing is, I absolutely do not want to be tied to the Docker way of describing architectures, which is inferior to the one we have at the moment and that allows us to target architectures that are not explicitly supported by Docker. How would the automated architecture selection play with the current system?

More generally, I said initially that I didn't want to be in charge of the s6-overlay project: I maintain it because it's useful to people, and because it's currently the most popular use of s6. I am okay with the development side of things, and we had an agreement with @jprjr that he would handle the ops side of things. But it doesn't seem like he wants to be involved any more; so if it's going to be just me, I'm going to touch the ops side as little as possible. Containers are something I'm involved with because it's where my software is used; but I don't want them to consume my time, so if it's not broken, I'm not going to fix it.

So, yeah, if you're willing to provide a proof of concept with Dockerfiles and GitHub actions that I could play with to see how it would go, sure, why not; but expect to have to provide a lot of patient explanations, and opinions and point of views that are probably rather different from your usual container software provider. Sorry for being so conservative and reluctant about this; but managing my priorities is absolutely necessary.

[skarnet]

@crazy-max It doesn't sound similar to your project; basically, what you're doing is rewriting the build system in the Dockerfile language, which I don't see the benefit of at all. Whereas if I understand correctly, @flokoe seems to want to keep the existing build, just provide different end outputs via Github actions voodoo, to integrate with various user workflows.

[socheatsok78]

I would agree with @flokoe. But only a single container image is all that we need, e.g. just-containers/s6-overlay.

We can distribute the tarballs using tags. Example:

• s6-overlay-noarch.tar.xz and s6-overlay-${arch}.tar.xz as v3.2.0.0
• s6-overlay-symlinks-noarch.tar.xz and s6-overlay-symlinks-arch.tar.xz as v3.2.0.0-symlinks
• syslogd-overlay-noarch.tar.xz as v3.2.0.0-syslogd

All that we need after that for Dockerfile:

ARG S6_OVERLAY_VERSION=v3.2.0.0
FROM containers/s6-overlay:${S6_OVERLAY_VERSION} AS s6-overlay
#FROM containers/s6-overlay:${S6_OVERLAY_VERSION}-symlinks AS s6-overlay-symlinks
#FROM containers/s6-overlay:${S6_OVERLAY_VERSION}-syslogd AS s6-overlay-syslogd

FROM alpine:latest
COPY --link --from=s6-overlay / /
#COPY --link --from=s6-overlay-symlinks / /
#COPY --link --from=s6-overlay-syslogd / /
ENTRYPOINT [ "/init" ]

This is what I've made and use personally socheatsok78/s6-overlay-distribution, previously socheatsok78/s6-overlay-installer but I found that it is inconvenience to download and install from GitHub every time a build happens.