Repack Nwb Files

[pauladkisson]

Fixes #892

Depends on

• Wrap data in set_data_io with a DataChunkIterator to support overriding hdf5 dataset backend configurations hdmf-dev/hdmf#1172
• added link_data --> clear_cache relationship to support repacking zarr nwbfiles hdmf-dev/hdmf-zarr#215
• [Bug]: Error when pixel_mask PlaneSegmentation is exported hdmf-dev/hdmf-zarr#272
• [Bug]: pixel mask gets duplicated every time it is written hdmf-dev/hdmf-zarr#278
• [Bug]: compound Dtypes do not support custom chunking and compression with Zarr. hdmf-dev/hdmf#1296

[pauladkisson]

What are you thinking about how the API should look like for how to use this repack helper?

@CodyCBakerPhD, the code now reflects my vision for the API: repack_nwbfile takes an on-disk nwbfile and export path, configures the backend, and exports the nwbfile. Users can optionally specify the template (existing or default) and any manual changes to the backend config.

The code then progresses along 2 paths:

• existing: where backend info is read directly from the nwbfile
• default: where default backend info is obtained for each neurodata object in the nwbfile

lmk what you think!

[h-mayorquin]

use math and not np.prod as that will overflow in most cases and screw up your buffer case.

Are these changes related to repacking?

[pauladkisson]

use math and not np.prod as that will overflow in most cases and screw up your buffer case.

Ok, will do.

Are these changes related to repacking?

Yes, the DataChunkIterator solves this issue: hdmf-dev/hdmf#1170

[h-mayorquin]

Ah, thanks, OK. so hdmf does not support re-setting the chunking without copying so we do this through an iterator.

Mmm, you are solving your problem and this is great but in this PR as well as the one that you did for detecting compound data types I kind of feel that hdmf is kicking its complexity up to us for something that it should provide.

I guess that we are faster : P

[pauladkisson]

Speaking of which, I ran into another issue with hdmf-zarr when it comes to compound dtypes -- see here: hdmf-dev/hdmf-zarr#272

[pauladkisson]

Could you please add a chapter in the user guide?

Done!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.61%. Comparing base (d09abd9) to head (1f1c629).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1003      +/-   ##
==========================================
+ Coverage   90.31%   90.61%   +0.30%     
==========================================
  Files         138      138              
  Lines        8876     9000     +124     
==========================================
+ Hits         8016     8155     +139     
+ Misses        860      845      -15     

Flag	Coverage Δ
unittests	90.61% <100.00%> (+0.30%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/neuroconv/tools/nwb_helpers/__init__.py	100.00% <100.00%> (ø)
...roconv/tools/nwb_helpers/_backend_configuration.py	100.00% <100.00%> (ø)
...nwb_helpers/_configuration_models/_base_backend.py	100.00% <100.00%> (ø)
..._helpers/_configuration_models/_base_dataset_io.py	98.70% <100.00%> (+0.20%)	⬆️
..._helpers/_configuration_models/_hdf5_dataset_io.py	85.41% <100.00%> (+16.84%)	⬆️
..._helpers/_configuration_models/_zarr_dataset_io.py	96.96% <100.00%> (+13.33%)	⬆️
.../neuroconv/tools/nwb_helpers/_configure_backend.py	93.47% <100.00%> (+2.56%)	⬆️
...roconv/tools/nwb_helpers/_dataset_configuration.py	95.68% <100.00%> (+2.01%)	⬆️
...nv/tools/nwb_helpers/_metadata_and_file_helpers.py	92.52% <100.00%> (+3.42%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.