Skip to content

HDF5Plugins.jl: lz4, zstd, and bzip2 filter support #874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mkitti opened this issue Oct 31, 2021 · 3 comments
Closed

HDF5Plugins.jl: lz4, zstd, and bzip2 filter support #874

mkitti opened this issue Oct 31, 2021 · 3 comments

Comments

@mkitti
Copy link
Member

mkitti commented Oct 31, 2021

I started a package over at https://github.com/JuliaIO/HDF5Plugins.jl to implement a few additional compression plugins for HDF5. The plugins themselves are implemented in Julia similar to the Blosc plugin in this package. I've implemented the low level interface, and I'm starting to work on the high level interface.

Should we considering merging that code into HDF5.jl proper or should we consider spinning the Blosc plugin out (and perhaps upgrading it to Blosc 2)?
@musm
Copy link
Member

musm commented Oct 31, 2021 •
edited
Loading

Looks good. Yeah I think we should definitely merge these here to make using them simple. Take a look at https://github.com/JuliaIO/HDF5.jl/tree/master/src/filters and the change that made them in #847 which simplified filter support substantially.

(and perhaps upgrading it to Blosc 2)?

Agreed, I thought Blosc.jl was using the blosc2 library internally under the hood, but just checking it doesn't seem that that is true. I found this https://github.com/waralex/Blosc2.jl library which wraps Blosc2.jl.

@mkitti
Copy link
Member Author

mkitti commented Oct 31, 2021

Blosc2 is actually fairly recent. Release 2.0.0 was only on Jun 28, 2021.

One complication is that additional licensing is required to include the plugins since the original C code for the plugins (which I ported to Julia) come under various licenses.

Currently, I'm targeting the discrete package, HDF5Plugins.jl against the 0.15 branch with some consideration for the 0.16 branch. How about we keep it separate for the 0.15 branch, and consider merging it into the 0.16 branch?

@mkitti mkitti mentioned this issue Nov 1, 2021
Closed
@musm
Copy link
Member

musm commented Dec 17, 2021

closed by #880

@musm musm closed this as completed Dec 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add bzip2, lz4, and zstd filters from HDF5Plugins.jl JaneliaSciComp/HDF5.jl
2 participants
@musm @mkitti