Is it possible to use clustering with file system as storage?

[gsharma85]

First all of thanks for creating this wonderful product. it has immense potential in self hosted registry space.

I work for cloud engineering team in my organization and we are looking at a few self hosted registry solutions. Our main requirement is light weight and scale especially on the read path. Clustering will help in this regard but that makes a dependency on AWS S3 as remote backend. I understand this might be one of the way to get state away from the front ends and prevent a few concurrency issues but it does increases the complexity of the overall system. So a couple of questions:

1. Can we make clustering work using file system as storage? Something where only one node act as writer and rest of the nodes in cluster runs as read replicas?
2. Can Minio S3 can be used in place of AWS S3?
   3 For a single zot instance the role of boltDB is just for dedup, how much we are able to save using dedup in general? Any metrics around that.

Thanks for your help.

[rchincha]

@gsharma85 Thanks for trying out zot.

For clustering questions - 1 and 2 above, pls check if the following pointers suffice.

https://zotregistry.io/latest/articles/clustering/

https://github.com/project-zot/zot/blob/main/.github/workflows/cluster.yaml (we use minio here)
https://github.com/project-zot/zot/blob/main/examples/cluster/haproxy.cfg

For 3, dedupe savings really depend on your container images and their layers. Quoting from a deployment (several 10s of TBs, frequent dev container builds), we have seen about 50% savings. In some cases, it also saves client network bandwidth (download only if blob is present anywhere in the registry)

Hope this helps.

Also, pls consider joining our Slack channel https://cloud-native.slack.com/archives/C03EGRE4QGH

[rchincha]

@gsharma85 Note that #125 is targeted for an upcoming release.

[gsharma85]

Thanks @rchincha.

Clustering doc say you must use S3.
But if I use a shared file system storage with say three zot instances which we use only for say reading images via pull through from say dokcer hub, do we have a concurrency issue among replicas

[rchincha]

S3 or S3-compatible storage (minio is one)

More a consistency issue (tracked by issue #125)

OTOH, if you want to use local storage and only load-balance, you could setup a haproxy to write to one replica and make other replicas as sync mirrors (pointing to that replica) https://zotregistry.io/latest/articles/mirroring/ so that writes go to one replica and reads from all replicas. This solution is not perfect but could work for you.

[gsharma85]

Thanks for the quick reply @rchincha. I understood your point and you are right as the mirroring solution seems to be somewhat towards hacky side.

Actually we want to explore Zot in a pure pull through cache cluster scenario where we run multiple read only replicas that share same storage. Since we want to keep the architecture simple we want to use NFS as shared storage instead of remote backend like S3.

To further elaborate my question I am putting two diagram below and want to further ask which one will not have any concurrency issue while using the same shared storage. I don't want to have a separate storage for each instance or replica in my read only cache cluster. Also the only write use on the shared storage to consider is when a read replica pull through from a remote registry like docker hub,. If two read replicas pull through the same image from docker hub to shared storage could you please point any concurrency issue?

[rchincha]

@gsharma85 The mirror/sync feature was designed mainly for keep zot local to the client. IINM, your requirement is for a load-balanced (maybe HA) cache.

With the current zot capabilities, it will have to be something along the lines of #1529 (reply in thread). In reality, this may work out just fine as long as clients simply retry on failures.

[rchincha]

@gsharma85 Sorry, let me elaborate.

1. Firstly, sync/mirroring is a requirement for you since you want to run a cache. However, note that only on-demand sync works with dockerhub and not periodic polling - a limitation of dockerhub and not under zot's control

2. Due to current constraints within zot itself (libraries we use etc), "Solution 2" meaning S3 + sync is not supported. We are fixing this in an upcoming release.

3. That leaves us with Solution 1. One of the zot replicas stills needs to be a writer to a particular repo into NFS, and since on-demand needs to be supported due to 1., we must allow clients to hit any/all of the zot replicas. Since we cannot tell if an image has already been sync'ed or not a priori, we cannot simply assign a single replica to be a sync'er - all of them have to be sync'ers but still not step on each others' toes.

4. That leaves us currently with setting up a haproxy using https://www.haproxy.com/blog/path-based-routing-with-haproxy with a subset of repos handled by each replica exclusively. This gives a path to scale-out but not HA of course.

Ideally, 4. should be automatic, and we are working to address this in issue #125. Also note that clustering on bare-metal and kubernetes will be different and both have to be supported on zot - something we are designing for also btw.

[rchincha]

client A -> /v2/<repoA>/manifests/.... -> haproxy [repoA -> zotA] -> zotA (sync from dockerhub into repoA)
client B -> /v2/<repoB>/manifests/... -> haproxy [repoB -> zotB] -> zotB (sync from dockerhub into repoB)
... and so on.

[gsharma85]

Thanks @rchincha last post was very helpful and I am able to get better insight into how Zot works in clustering mode.

Taking a step further now:

If we say perform three settings with our cluster of 3 zot instances 1) disable sync/mirroring with other registries 2) make one zot instance primary/master (lets keep aside how we can achieve this) 3) let the push operations always go to master, pull can happen from any of the zot instance behind haproxy

1. With solution 1 where NFS was used as a shared storage (all images getting stored at a single location, any replica can get to any image) and with sync disabled and all push going through the master, we will not face a concurrency issue (replicas stepping over each other case) for push and furthermore pull can still happen from any zot instance now (without a need for a subset of repos handled by each replica exclusively)?

2. With solution 2 where S3 was used a remote backend, since we disabled sync/mirroring and writes happening through a single master, again no concurrency issue. But the question here is that if I relax the cluster and let every instance to handle push, just like pull behind haproxy, will we experience concurrency issue? Point is that if using S3 as remote backend also gives developers extra benefit of allowing even push getting load balanced across zot instances without concurrency issues?

After above clarification we will be in a much better shape to start a poc within our organization.

Thanks again for answering.

Gaurav

[rchincha]

1. above would work since you have a single writer and multiple readers. Do try and let us know how it went.

2. could have issues if the same image is being written to from multiple writers. S3 will be consistent for each API call, but the entire file content may not be - not different from multiple processes writing to same file on a local filesystem. If you can achieve separation/mutual exclusion, then don't see why this couldn't work. Also note that best if your metadata is on dynamodb for gc and dedupe in this case if you plan to enable those.

[gsharma85]

thanks @rchincha I believe all of my questions are answered. I will go ahead with design discussions and poc with wider team and come back with questions if any.