Use cloudpathlib instead of fsspec?

[TomNicholas]

AFAIK the only filesystems we need to read from are local and cloud, so could we just use pathlib and cloudpathlib?

[TomNicholas]

In particular we could just call cloudpathlib.AnyPath

https://cloudpathlib.drivendata.org/stable/anypath-polymorphism/

[norlandrhagen]

This would be really cool @TomNicholas!

Seems like it can read over s3 into xarray:

from cloudpathlib import CloudPath
import xarray as xr 
cloudpath = CloudPath("s3://carbonplan-share/air_temp.nc")
ds = xr.open_dataset(cloudpath)

[norlandrhagen]

A little more exploration. It looks like SingleHDFToZarr works both for s3 and local.

from kerchunk.hdf import SingleHdf5ToZarr
import io 
from cloudpathlib import CloudPath
import xarray as xr 
# from cloudpathlib import AnyPath

cloudpath = CloudPath("s3://carbonplan-share/air_temp.nc")

with open(cloudpath, 'rb') as f:
  contents = f.read()
  refs = SingleHdf5ToZarr(io.BytesIO(contents)).translate()
refs

[TomNicholas]

Some more thoughts - one way to smooth this transition would be to replace all uses of UPath (which is based on fsspec) with cloudpathlib's AnyPath. They are both very similar - for example they both implement a .stat method, which is used in https://github.com/zarr-developers/VirtualiZarr/pull/187/files#r1678802398.

The snag here is that I don't think cloudpathlib supports https...

[TomNicholas]

The snag here is that I don't think cloudpathlib supports https...

I raised drivendataorg/cloudpathlib#455

[scottyhq]

I've just started using cloudpathlib and it provides a nice straightforward interface, however, it does seem to read entire remote files from object storage up front (https://cloudpathlib.drivendata.org/stable/caching/). As far as I can tell this is equivalent to fsspec 'filecache' machinery

Conceptually, pulling the full files is straightforward and has max compatibility with other libraries... but if most of the metadata is at the front of the file you could get everything you need by just transferring a small subset (ideally first 'n' bytes see https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084)

So this is another issue to track for range-reads in cloudpathlib: drivendataorg/cloudpathlib#9

[maxrjones]

I'm going to close this in favor of #476, since obstore does support partial and streaming reads