Create `transfer` module to package spool files into `tar` and upload to DANDI #51

kabilar · 2025-05-13T23:03:43Z

Context

Based on a discussion with Satra, Anastasia, and Aaron (and previously with Emine, Elizabeth, and Wenze), we have decided to combine the spool files into tar.gz files and upload them to https://dandiarchive.org. From there, the data can be downloaded, unpackaged, and converted into an OME-Zarr archive.

Updates

Create transfer module that combines a few gigabytes of spool dat files into a tar file, uploads the tar file to DANDI, deletes the tar file, and repeats until all dat files have been uploaded. This process is employed so that not much local storage is consumed.

Test upload to the LINC staging server. See example command:

linc-convert lsm transfer --input-dir './multicolor_section_run11__y06_z01_HR' --dandiset-url 'https://staging.lincbrain.org/dandiset/000012' --dandi-instance 'linc-staging' --subject 'm1' --output-dir '.' --max-size-gb 1

Benchmark transfer.py module
Add tests
- Add DANDI_API_key secret to GitHub repository
- Create https://lincbrain.org/dandiset/000051/draft dataset to privately store the test dataset

cc @satra @ayendiki

kabilar · 2025-05-15T19:17:41Z

Below are benchmarking results comparing spool .dat conversion to .tar vs .tar.gz. Tested a few times on my laptop (M2 Pro, 32 GB RAM, with many other processes running).

For ~1 GB, it takes ~300 seconds to package and compress to .tar.gz vs <2 seconds to package to .tar. The compression ratio is ~2.4 (1030 MB to 434 MB).

Based on these numbers, I would suggest that we just package into a tarbal without gzip compression and upload. In total, it would take 3.4 days to package the entire 216 TB plus the upload time of 25 days (assuming 100 MBps).

kabilar · 2025-05-15T20:33:34Z

Hi @calvinchai, have you done similar benchmarking for the OME-Zarr conversion (and/or stitching)? If so, could you point me to those findings? Thanks.

… lsm-transfer

…sm-transfer

kabilar · 2025-05-17T17:30:01Z

Tests now pass. See latest run. Not sure why they are failing in this pull request.

Copilot

Pull Request Overview

This PR introduces a new transfer module to batch .dat files into tar archives and optionally upload them to DANDI, updates dependencies and exports, and adds CI/test support.

Adds linc_convert.modalities.lsm.transfer for archiving and uploading spool files.
Extends pyproject.toml extras to include dandi and updates __init__.py.
Introduces test_transfer in tests/test_lsm.py and configures DANDI_API_KEY in CI.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_lsm.py	Add `test_transfer` to verify archive creation and file integrity
pyproject.toml	Add `dandi` to `lsm` and `all` extras
linc_convert/modalities/lsm/transfer.py	New transfer logic for batching, archiving, and uploading
linc_convert/modalities/lsm/init.py	Export `transfer` in `__all__`
.github/workflows/ci.yml	Trigger CI on push, PR, dispatch, and set `DANDI_API_KEY` env

Comments suppressed due to low confidence (2)

linc_convert/modalities/lsm/transfer.py:94

There's no test covering the upload=True branch. Consider adding a test with a mocked dandi.upload.upload call to verify upload logic and cleanup.

if upload:

tests/test_lsm.py:37

The test invokes dandi.download.download which performs network I/O. Mock dandi.download.download to avoid external dependencies and make tests deterministic.

transfer.dandi_transfer(input_dir=input_dir,

linc_convert/modalities/lsm/transfer.py

tests/test_lsm.py

pyproject.toml

.github/workflows/ci.yml

kabilar · 2025-05-27T22:10:48Z

Hi @balbasty @calvinchai, following up to see if you have feedback on this pull request. Thanks.

calvinchai · 2025-05-28T19:56:07Z

Hi @calvinchai, have you done similar benchmarking for the OME-Zarr conversion (and/or stitching)? If so, could you point me to those findings? Thanks.

I may have that on paper. Let me get back to you on that tomorrow.

calvinchai · 2025-05-28T19:56:54Z

Hi @balbasty @calvinchai, following up to see if you have feedback on this pull request. Thanks.

My apologee, missed this notification. Taking a look now

balbasty · 2025-05-29T19:22:09Z

linc_convert/modalities/lsm/transfer.py

+    Parameters
+    ----------
+    input_dir : str
+        Directory containing .dat files to upload


An alternative could be for the input to be a list of files, rather than doing the glob ourselves in the function. Dat files can still be filtered using a command-line glob: linc-convert lsm transfer path/to/dir/*.dat. And it would allows taring non-dat files if needed.

But I understand that the directory-based interface might be easier to use for Emin et al.

linc_convert/modalities/lsm/transfer.py

balbasty · 2025-05-29T19:25:00Z

linc_convert/modalities/lsm/transfer.py

+    dandi_instance : str
+        DANDI server (e.g. linc, dandi)
+    output_dir : str, optional
+        Directory to save the Dandiset directory (default: '.')


We delete everyting in the end, right? Should we just use a tempfile.TemporaryDirectory instead of asking the user for a location?

balbasty · 2025-05-29T19:28:25Z

linc_convert/modalities/lsm/transfer.py

+            print(f"Uploading {archive_path}.")
+            dandi.upload.upload([dandiset_directory],
+                                dandi_instance=dandi_instance,
+                                )


Is this robust enough? Would it be useful to wrap this into a loop with try/except and allow for multiple tries?

Also, maybe we should have an option to restart a transfer where it was left off (say that the person stops it with ctrl+c). Not sure how to implement this.

Is this robust enough? Would it be useful to wrap this into a loop with try/except and allow for multiple tries?

Good idea. I have added a loop to continually attempt the upload until successful.

Also, maybe we should have an option to restart a transfer where it was left off (say that the person stops it with ctrl+c). Not sure how to implement this.

I could create a manifest file that is stored locally to track the files that have been added to a tar and the tars that have been uploaded.

pyproject.toml

calvinchai · 2025-07-01T14:17:24Z

I was wondering if it is possible to do something like
linc-convert ...... --output dandi:// so we have a unified output format

calvinchai · 2025-07-01T18:30:38Z

I just checked using dandi-cli in the zarr 3 branch, and no luck. becuause hdmf-zarr requires zarr-python < 3.0,

kabilar added 2 commits May 13, 2025 17:50

Create module to package spool files into tar and upload to DANDI

a1f81aa

Remove print statement

8114a91

kabilar marked this pull request as draft May 13, 2025 23:03

Update init

07aa8e5

kabilar added 2 commits May 15, 2025 14:26

Update default value

ad3dd76

Revert gzip usage

05284de

kabilar added 22 commits May 16, 2025 11:09

Add API key to GitHub Action

ffce115

Add test for transfer.py module

9f143aa

Merge branch 'main' of https://github.com/lincbrain/linc-convert into…

41dc406

… lsm-transfer

Add manual trigger for CI

a6938de

Add dandi to dependencies

fe7043d

Fix pyproject file

53e3397

Add upload flag

b27dfa7

Fix linc-convert call

3b26e22

Try different order in github action

b67cbd2

Revert order change

692ef28

Update subprocess call

8029ed3

Fix subprocess call

638946c

Wait until subprocess finishes

83e1bad

Only remove the tar if uploading

b438146

Fix CI

40c7827

Fix subprocess call

bd38d6c

Define API key explicitly

bfe7ab2

Fix CI

0475c78

[WIP] Add print statement

b7746e2

Revert CI change

b91ce99

Update subprocess call

9191f15

Add print statement for subprocess call

2f7792b

kabilar added 12 commits May 17, 2025 11:04

Add print statements

5fc47fc

Add print statement

e03f0aa

Update print statements

b0917af

Change path handling

5879286

Update print statement

7102116

Fix path handling

a3cadb3

Update print statement

e6fd12a

Remove extra os.remove call

13006c2

Revert changes

143e0b0

Remove print statements

331e510

Remove extra imports

ddf7ad0

Merge branch 'main' of https://github.com/kabilar/linc-convert into l…

8fb8c3c

…sm-transfer

kabilar marked this pull request as ready for review May 17, 2025 17:31

kabilar requested review from balbasty, calvinchai and Copilot May 17, 2025 17:31

Copilot AI reviewed May 17, 2025

View reviewed changes

linc_convert/modalities/lsm/transfer.py Show resolved Hide resolved

tests/test_lsm.py Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

.github/workflows/ci.yml Show resolved Hide resolved

kabilar changed the title ~~Create transfer module to package spool files into tar.gz and upload to DANDI~~ Create transfer module to package spool files into tar and upload to DANDI May 17, 2025

balbasty reviewed May 29, 2025

View reviewed changes

kabilar added 3 commits June 12, 2025 13:27

Parse dandi instance from dandiset url

46c5bef

Pin dandi to minimum version

367ba0b

Add a loop to continually attempt the data upload until successful

d044e40

calvinchai approved these changes Jun 30, 2025

View reviewed changes

calvinchai mentioned this pull request Aug 4, 2025

Standardize isolated dandi usage (tests + runtime) to avoid dependency conflicts #58

Open

6 tasks

Create transfer module to package spool files into tar and upload to DANDI #51

Are you sure you want to change the base?

Create transfer module to package spool files into tar and upload to DANDI #51

Uh oh!

Conversation

kabilar commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Updates

Uh oh!

kabilar commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kabilar commented May 15, 2025

Uh oh!

kabilar commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kabilar commented May 27, 2025

Uh oh!

calvinchai commented May 28, 2025

Uh oh!

calvinchai commented May 28, 2025

Uh oh!

balbasty May 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

balbasty May 29, 2025

Choose a reason for hiding this comment

Uh oh!

balbasty May 29, 2025

Choose a reason for hiding this comment

Uh oh!

balbasty May 29, 2025

Choose a reason for hiding this comment

Uh oh!

kabilar Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

kabilar Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

calvinchai commented Jul 1, 2025

Uh oh!

calvinchai commented Jul 1, 2025

Uh oh!

Uh oh!

Create `transfer` module to package spool files into `tar` and upload to DANDI #51

Create `transfer` module to package spool files into `tar` and upload to DANDI #51

kabilar commented May 13, 2025 •

edited

Loading

kabilar commented May 15, 2025 •

edited

Loading

kabilar commented May 17, 2025 •

edited

Loading