support google cloud storage

[carderne]

I had to add this to be able to use xpublish-tiles with Google Cloud Storage. Just using the library for the first time, apologies if this is misplaced!

After adding this, I was able to run with:

uv run xpublish-tiles --dataset "local://gs://bucket/repo::"

I don't really understand the local:// thing, so likely this is a bit of a kludge...

NB: the changes to the s3 branch are just a little clean-up to keep the two consistent.

[dcherian]

ooh that's an error. local refers to datasets stored locally in /tmp, most useful for datasets xpublish-tiles can generate. Can we instead have it trigger on s3:// and gcs:// instead of local://s3://?

[carderne]

With an s3:// or gcs:// prefix, how would we differentiate between zarr/icechunk?

Are the zarr:/// and icechunk:/// prefixes specific to Arraylake?

---

Aside: any plans to support some kind of config file? I guess this would require a bit of a manager to spawn an instance for each dataset. (Will move to a separate discussion if worthwhile.)

# stupid example, I don't know what all would need to be in here
[[datasets]]
name = "foo"
format = "icechunk"
path = "gcs://bucket/path"

[[datasets]]
name = "bar"
format = "zarr"
path = "zarr:///foo/bar"

[dcherian]

Right yes, we'll need zarr://s3:// and icechunk://s3://. ATM arraylake has no prefix, but we could add one like al://. Arraylake will figure out which object store to use, so we needn't handle s3:// in that branch.

as an aside, note that we use the cli stuff just for dev so it's quite messy.

I guess this would require a bit of a manager to spawn an instance for each dataset. (Will move to a separate discussion if worthwhile.)

yes. that's part of the xpublish design. The router simple handles the dataset handed to it.

[carderne]

Oh right of course, makes more sense now :)