You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The N5 API specifies the primitive operations needed to store large chunked n-dimensional tensors, and arbitrary meta-data in a hierarchy of groups similar to HDF5.
4
5
@@ -22,11 +23,11 @@ N5 group is not a single file but simply a directory on the file system. Meta-d
22
23
23
24
1. All directories of the file system are N5 groups.
24
25
2. A JSON file `attributes.json` in a directory contains arbitrary attributes. A group without attributes may not have an `attributes.json` file.
25
-
3. The version of this specification is 1.0.0 and is stored in the "n5" attribute of the root group "/".
26
+
3. The version of this specification is @n5-spec.version@ and is stored in the "n5" attribute of the root group "/".
26
27
4. A dataset is a group with the mandatory attributes:
* compression as a struct with the mandatory attribute type that specifies the compression scheme, currently available are:
31
32
* raw (no parameters),
32
33
* bzip2 with parameters
@@ -38,13 +39,13 @@ N5 group is not a single file but simply a directory on the file system. Meta-d
38
39
* xz with parameters
39
40
* preset (integer, default 6).
40
41
41
-
Custom compression schemes with arbitrary parameters can be added using [compression annotations](#extensible-compression-schemes), e.g. [N5 Blosc](https://github.com/saalfeldlab/n5-blosc).
42
+
Custom compression schemes with arbitrary parameters can be added using [compression annotations](#extensible-compression-schemes), e.g. [N5 Blosc](https://github.com/saalfeldlab/n5-blosc) and [N5 ZStandard](https://github.com/JaneliaSciComp/n5-zstandard/).
42
43
5. Chunks are stored in a directory hierarchy that enumerates their positive integer position in the chunk grid (e.g. `0/4/1/7` for chunk grid position p=(0, 4, 1, 7)).
43
44
6. Datasets are sparse, i.e. there is no guarantee that all chunks of a dataset exist.
44
45
7. Chunks cannot be larger than 2GB (2<sup>31</sup>Bytes).
45
46
8. All chunks of a chunked dataset have the same size except for end-chunks that may be smaller, therefore
46
47
9. Chunks are stored in the following binary format:
*[ mode == varlength ? number of elements (uint32 big endian) ]
@@ -134,4 +135,3 @@ Custom compression schemes can be implemented using the annotation discovery mec
134
135
HDF5 is a great format that provides a wealth of conveniences that I do not want to miss. It's inefficiency for parallel writing, however, limit its applicability for handling of very large n-dimensional data.
135
136
136
137
N5 uses the native filesystem of the target platform and JSON files to specify basic and custom meta-data as attributes. It aims at preserving the convenience of HDF5 where possible but doesn't try too hard to be a full replacement.
137
-
Please do not take this project too seriously, we will see where it will get us and report back when more data is available.
0 commit comments