Implement N5Store as a transformation layer over other stores

[alimanfoo]

The current work on an N5Store in zarr-developers/zarr-python#309 implements a storage class that does two things. First, it implements a transformation from the keys and values used in the zarr protocol into keys and values used in the N5 protocol. It then also implements storage of keys and values as files on a file system.

It would be possible to separate these concerns, such that the N5Store just implemented the protocol transformation, and then delegated the actual storage or retrieval of N5 keys and values from an inner store. This would mean that the N5Store class could be used with file systems or cloud stores or any form of key/value storage.

E.g., what is currently now:

store = zarr.N5Store('/path/to/store.n5')

...could become a shorthand for:

store = zarr.N5Store(zarr.DirectoryStore('/path/to/store.n5'))

...and other inner stores could be used, e.g.:

from gcsfs import GCSMap
store = zarr.N5Store(GCSMap('my-bucket', ...))

Internally, to make this possible, the N5Store class would replace all the logic for opening, reading or writing files with calls to inner.__getitem__(key) and inner.__setitem__(key, value) where inner is the inner store wrapped by the N5Store class.

[jakirkham]

Thanks for writing this up.

Would add that this is also true of NestedDirectoryStore, which N5Store currently relies on. If we change N5Store into a transformational layer, we may want to do something similar with NestedDirectoryStore. ( zarr-developers/zarr-python#395 )

[alimanfoo]

Good point, yes using "/" instead of "." as the separator within chunk keys could also be implemented as a protocol transformation over any other store. Would probably need a new class with an appropriate name for that, e.g., NestedChunkStore. So then NestedDirectoryStore(path) would become a shorthand for NestedChunkStore(DirectoryStore(path)).

…

On Tue, 26 Feb 2019, 02:53 jakirkham, ***@***.***> wrote: Thanks for writing this up. Would add that this is also true of NestedDirectoryStore, which N5Store currently relies on. If we change N5Store into a transformational layer, we may want to do something similar with NestedDirectoryStore. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/zarr-developers/zarr/issues/410#issuecomment-467273216>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QgW0v8n2Uujw2Qw7cYJpXlcXEDK_ks5vRKHFgaJpZM4bQeRV> .

[jakirkham]

As far as naming these classes that are not quite stores, what should we call them?

1. ...Store
2. ...Layer
3. ...Transformer
4. ?

[alimanfoo]

As far as naming these classes that are not quite stores, what should we call them?

1. ...Store
2. ...Layer
3. ...Transformer
4. ?

Good question. I guess ...StoreTransformer would be the most descriptive, but also the most verbose. I'd be happy with just ...Store too.

xref zarr-developers/zarr-specs#8

[d-v-b]

I transferred this issue from zarr-python to n5py.