Blosc support

[dhirschfeld]

Just thought I'd xref existing investigations into bringing blosc to the browser:
Blosc/c-blosc#238 (comment)

[gzuidhof]

Thanks for sharing :)

I would love to include support for Blosc (especially as it's the default compressor for Zarr), but will probably implement some of the compression schemes more readily implemented in JS first. If someone packages it up with a simple API it should be easy to fit in :)

[joshmoore]

During this weeks zarr-developers/community#1, @ryan-williams, @rabernat,
@WardF, and I discussed if chatting with @FrancescAlted and @wolfv wouldn't help. Any interest in either using the January 8th slot or setting up a separate call to discuss?

[oeway]

I started to use zarr.js, and getting Compression codec blosc is not supported error , would be really cool if the default codec can be supported!

[thewtex]

I have Blosc compiled to WebAssembly and tested here. Is there interest in packaging this into an npm package so zarr.js or zarr could use it? It is currently just used for decompression, but a compression interface could also be added. What interface is desired?

[gzuidhof]

Amazing, this is really exciting

I think the plan is to consolidate compression methods and filters into a separate repository/npm package in the future (like numcodecs in Python), but for now they live in the src/compression folder.

Here is the interface we use for compression and decompression, and this is an example implementation (Zlib) that wraps an external NPM package. As long as your package can be wrapped like this it should be easy to integrate :)

[thewtex]

@gzuidhof cool! If you want help adding the blosc support, please ping me once the package and interface is up. I do not have much typescript experience, so it will help to have the build configuration setup and interfaces defined.

Here is the interface we use for compression and decompression

Looks good. For blosc, it also needs, for decompression

• The underlying compression codec, typically identified as cname, e.g. zstd, lz4, a string.

And for compression:

• Compression level, i.e. clevel, an integer.
• Typesize, i.e. 2 for unsigned short. An integer.
• Shuffle, a compression option that applied to blosc that is independent of the underlying compression codec. This is a boolean, but is typically specified with an integer.

To stay in sync with numcodec, it may be helpful to configure the codec with a JSON object.

[gzuidhof]

As of v0.3.0 zarr.js supports Blosc compressed datasets (props to @manzt) :)