Cannot use zarr-python to open zarr stores with https url

[rly]

As I reported on Slack:

Dandiset 000719 contains NWB zarr stores. File sub-npI3_ses-20190421_behavior+ecephys_rechunk.nwb.zarr has this asset metadata
https://api.dandiarchive.org/api/dandisets/000719/versions/draft/assets/58de1d1c-278f-4278-953a-bf8790de4c69/
which lists the content URL:
https://dandiarchive.s3.amazonaws.com/zarr/d097af6b-8fd8-4d83-b649-fc6518e95d25/

This file has consolidated metadata.

The following code uses zarr-python 2.18.7 to open the Zarr store using that HTTPS URL but finds no keys:

import zarr
https_url = "https://dandiarchive.s3.amazonaws.com/zarr/d097af6b-8fd8-4d83-b649-fc6518e95d25/"
z = zarr.open(https_url, mode="r")
print(list(z.keys()))  # returns []

But there are many keys there. You can see that in the browser by going to a URL with the same base and a known file name, such as .zmetadata, e.g., https://dandiarchive.s3.amazonaws.com/zarr/d097af6b-8fd8-4d83-b649-fc6518e95d25/.zmetadata does return data

However, if I use the S3 URL for this Zarr store, I can see all the keys:

s3_form = https_url.replace("https://dandiarchive.s3.amazonaws.com/", "s3://dandiarchive")
z = zarr.open(s3_form, mode="r", storage_options={"anon": True})
print(list(z.keys()))  # returns ['acquisition', 'analysis', 'file_create_date', 'general', ...]

I believe I should be able to access a Zarr store using an HTTPS URL in general. I don't know why these would be any different.

This was also reported in #1745 but it seems like that discussion diverged.

@magland wrote:

This highlights the fact that you cannot use normal fetch http requests to read the tree structure of a Zarr directory in an S3 bucket - because there's no way to get a directory listing (unless the admin enables something that is highly not recommended).

What is this option that should not be enabled? Is there a way around that?

@satra wrote:

indeed for http we used to have our own api endpoint. i can't seem to find this any more, so checking with @AlmightyYakob.

There was no follow-up on this. Is there a special HTTP endpoint for Zarr stores?

@kabilar adds on Slack that with the HTTPS URL, the data is accessible if you know the keys. It just doesn't display the keys properly and lists 0 arrays/groups within a group:

In [3]: a=zarr.open("https://dandiarchive.s3.amazonaws.com/zarr/d097af6b-8fd8-4d83-b649-fc6518e95d25/",mode='r')

In [4]: a['acquisition']
Out[4]: <zarr.hierarchy.Group '/acquisition' read-only>

In [5]: a['acquisition']['ElectricalSeries']
Out[5]: <zarr.hierarchy.Group '/acquisition/ElectricalSeries' read-only>

In [6]: a['acquisition']['ElectricalSeries']['data']
Out[6]: <zarr.core.Array '/acquisition/ElectricalSeries/data' (349975807, 384) int16 read-only>

In [7]: print(list(a.keys()))
[]

In [32]: a['acquisition/ElectricalSeries'].info
Out[32]: 
Name        : /acquisition/ElectricalSeries
Type        : zarr.hierarchy.Group
Read-only   : True
Store type  : zarr.storage.FSStore
No. members : 0
No. arrays  : 0
No. groups  : 0

In [33]: a['acquisition/ElectricalSeries/data'].info
Out[33]: 
Name               : /acquisition/ElectricalSeries/data
Type               : zarr.core.Array
Data type          : int16
Shape              : (349975807, 384)
Chunk shape        : (262144, 32)
Order              : C
Read-only          : True
Filter [0]         : Zlib(level=4)
Compressor         : Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0)
Store type         : zarr.storage.FSStore
No. bytes          : 268781419776 (250.3G)
Chunks initialized : 0/16032

I can use the S3 URL fine. But the asset page lists the HTTP URL and not the S3 URL. If you cannot use programmatically access a zarr store using the HTTPS URL, then perhaps the S3 URL should be shared on the asset metadata page or it should be documented somewhere how to transform the URL.

There may also be a performance difference between the two. Early benchmarking results suggest that reading NWB HDF5 files using fsspec with HTTPS URLs is faster than with S3 URLs, and zarr-python uses fsspec under the hood.

[magland]

I think this is what you want @rly

import zarr
https_url = "https://dandiarchive.s3.amazonaws.com/zarr/d097af6b-8fd8-4d83-b649-fc6518e95d25/"
z = zarr.open_consolidated(https_url, mode="r")
print(list(z.keys()))

Use open_consolidated instead of open.

[rly]

Ah, that works. Thanks @magland ! If we want to use the HTTPS URL, can we only open zarr stores that have consolidated metadata?

[magland]

Ah, that works. Thanks @magland ! If we want to use the HTTPS URL, can we only open zarr stores that have consolidated metadata?

Yes, otherwise without .zmetadata it's impossible to get the keys.

[yarikoptic]

I wonder if there is a way to configure AWS S3 to provide simple http/https index as regular websites do, and then I believe (expect?) for zarr libraries to be able to open them, right?