Question: is it possible to push to Spegel?

[caruccio]

Not a problem, only a question. Just found this project and could not find this question not in PRs, issues or docs.

Is it possible to push images to one of the Spegel instances and have it distributed to the whole cluster?
My use case is to avoid the pull from the first container at all, while pushing my new image right from the CI pipeline.
One simple solution for this is that I could just create a single pod only to trigger the pull, but it seams too odd.

[phillebaba]

So currently it is not possible but it is something that I have considered. It does however add some additional complexity to the solution. Spegel will always be a stateless application which means that all images stored are ephemeral. The risk then is that you build and push an image to one node, and when that node is removed from the cluster the image disappears. This could be mitigated with having image replication management like i mention in #375 but that is right now just an idea.

Is the reason you want to do this because you want to make sure the image is only pulled once? Or is it because you want to make sure the image is cached in the cluster before you start the deployment of a new application version?

[caruccio]

Both, in fact. With a push model I can avoid the high latency of the first pull, while using high speed local network to replicate it over the rest of the cluster.

[phillebaba]

Ok I can see that. I am not opposed to the idea, its just that I need to think through it a couple of times to come up with a good plan. This would add a write component to Spegel which would add additional complexity and security concerns. One thought that I had was to use a separate image store for images that have been pushed, then when another node pulls the image it will be in Containerd and we can remove it from the local store. Implementing this could work. I would probably have to change the configuration to a file to allow for some more complex settings. It would also enable serving content from multiple sources at once also, which could be an interesting feature.

[srgvg]

I'd like to add a use case for being able to push images: a situation of air-gapped and offline installation, where no local registry is available.

Another solution for this use case could be https://zarf.dev/ which has an interesting solution to bootstraps a registry in the cluster. and then push images that are locally packaged (in a zarf archive). Whilst interesting this does not solve caching the controlplane and related components (k8s controllers, kubelet, pause container, ...). It also currently breaks my Gitops/FluxCD workflow. That latter also involves needing to cache other types of OCI artefacts (helm charts and FluxCD with yaml manifests).

[vaizki]

Airgapped user here as well with no great solutions to bootstrap & run clusters with no registry connectivity...

[phillebaba]

@srgvg funny that you mention Zarf. I am pretty aware of how Zarf does this as I work on the project as my day job. The bootstrap process of Zarf is critical for air gap end to solve the chicken and egg problem. The solution depends on a custom registry implemented in Rust whose binary needs to be under a specific size, which can make future updates complicated. I have not made up my mind but it feels like it is out of scope for Spegel to solve right now.

I am very interested in eventually adding support for pushing. A p2p registry that does not depend on persistent disks is very interesting for deployment situations where there are no other options. I am involved in the work being done on Harbor Satellite which aims to use such a feature.
https://github.com/container-registry/harbor-satellite

This would still not solve the bootstrap problem, but I think that is a separate challenge. Right now the question is time for me. There are many ways to approach this problem and I need to think through how to best implement the p2p storage.

[srgvg]

FYI in the mean time I found out that the kubernetes distro we use - Talos - will be implementing a solution for this in the next release:

siderolabs/talos#9612