Add flag to specify number of frames per-chunk in intermediate hdf5 data #367
Conversation
yousefmoazzam
force-pushed
the
write-hdf5-chunked
branch
from
e9b8443
to
3bde4b9
Compare
|
To address the problem of if the value given to the If other behaviour is preferred, feel free to mention, I'll open this for review now. |
| @@ -118,6 +118,12 @@ def check(yaml_config: Path, in_data_file: Path = None): | |||
| default=514, | |||
| help="Port on the host the syslog server is running on", | |||
| ) | |||
| @click.option( | |||
| "--frames-per-chunk", | |||
There was a problem hiding this comment.
@ptim0626 @yousefmoazzam
I wonder if we need more granular control over the chunk sizes here? For instance we would like to use a smaller chunk than the whole frame, e.g. (1, 150, 150)? Something like a tuple to pass instead of the number of frames. The first value in that tuple would be linked to the slicing dimension and the other two of non-slicing dimensions?
Could that help with more versatile optimisation around the better write performance? And also the read speed in Dawn?
Currently the run fails on Wilson if frames-per-chunk > 1
There was a problem hiding this comment.
Considering performance, instead of decreasing the chunk size, I think increasing the chunk size would actually help.
(1) with compression, applying the filter to a single bigger chunk is usually quicker than multiple smaller chunks because of the overhead of threading for some filters llike Blosc
(2) larger chunks mean fewer metadata is required (the size of B-tree for looking up where the chunks are is smaller)
Choosing an optimal chunk size depends how we access the data, also a balance between read and write performance. I would say having one frame per chunk as the smallest unit with the option of increasing it via --frames-per-chunk is a good balance of performance and usability (does the general users know the implication of using a customised chunk shape?)
The recommended 1 MiB chunk size is about the default raw data chunk cache (rdcc) in hdf5, which is default to be 1 MiB. As long as we ensure we set the size of rdcc to be at least one chunk, it doesn't matter much for the chunk size, if it fits our access pattern. Even so, I don't think it impacts performance a lot, as most of the time (I guess) we just write/read the data once (so caching is not used).
There was a problem hiding this comment.
Thanks @ptim0626 , it makes sense. The general user won't be playing with --frames-per-chunk I reckon so it would be nice to fix a default size that fits our needs best.
|
Actually by enabling |
Setting the `--frames-per-chunk` flag to 0 will write the intermediate data as contiguous, and any value > 0 will be the number of frames (projections or sinograms) per-chunk. Thus, this preserves the ability to turn chunking on/off when writing intermediate data, but also adds the ability to choose the number of frames per-chunk.
The method function now is standalone and has no reliance on the framework's global variable `FRAMES_PER_CHUNK` to know the number of frames per-chunk to write. Instead, the associated method wrapper passes the framework's global variable `FRAMES_PER_CHUNK` through to the method function.
If the value provided for the `--frames-per-chunk` flag is larger than the length of the data's slicing dimension, then the number of frames per-chunk will be set to 1 (and will be logged in the verbose logfile). This is to avoid `h5py` errors with trying to create a chunk whose shape exceeds the boundaries of the data being written.
yousefmoazzam
force-pushed
the
write-hdf5-chunked
branch
from
e195f27
to
aef83c9
Compare
Built on top of work done in #271
This adds the
--frames-per-chunkflag which enables the ability to:A value of 0 will turn chunking off (ie, write the hdf5 data as contiguous), and values > 0 will be the number of frames per-chunk.
Acceptance criteria checklist: