[ENH] Formalize presence of optional scratch/ folder

[michellewang]

As per @yarikoptic's suggestion, this PR proposes formalizing an optional scratch/ subdirectory for storing temporary/working files. Nipoppy projects have a scratch/ directory, and it would be nice if it were added to the BIDS specification as well since we are moving towards aligning the Nipoppy project layout with the BIDS "study" dataset type.

I based this PR off of the changes made in #2104 for adding the docs/ directory, though I notice that #1962 has slightly different changes for the logs/ directory. I'm not sure what the differences in the schema changes are and if it would make sense to add the additional changes from #1962. @yarikoptic what do you think?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.71%. Comparing base (6bbd951) to head (895999c).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2184   +/-   ##
=======================================
  Coverage   82.71%   82.71%           
=======================================
  Files          20       20           
  Lines        1608     1608           
=======================================
  Hits         1330     1330           
  Misses        278      278           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[yarikoptic]

heh heh -- just a curse of duplication! I, frankly, missed the fact that in https://github.com/bids-standard/bids-specification/blob/master/src/schema/rules/directories.yaml we describe directories separately for each of the dataset types, so I need to complement it now with the one for

• [ENH] Add "study" DatasetType to organize a collection of source and derivative datasets #1972

uff -- filed

• [SCHEMA] Add "directories" description for "study" DatasetType and add "docs" where was missing #2185

IMHO scratch/ is good for all dataset types, so just list it for all.

[effigies]

I oppose adding scratch directories to raw and derivative datasets. Those dataset types should be archive-ready, and a dumping ground of intermediate files does not belong in either, IMO.

[yarikoptic]

Hi @effigies . Indeed I might have not thought it fully through!
The intent was more of an instruction to user where uniformly to have such a temporary location. But since it is not intended to be shipped with the dataset, it should not contain anything there during upload! So, oddly, I tried to standardize the location which must not contain any files, but for BIDS anything which is not prescribed by BIDS, is not legit BIDS... unless people do add it to .bidsignore, when it becomes ignored and still not legit.

So, even though I would have loved BIDS to prescribe as many locations as possible in a generic fashion, may be this "intermediate results" folder then should be "workflow specific" and instructed outside of the BIDS specification context.

[michellewang]

@yarikoptic are you saying that scratch should not be in the schema even for the study dataset type? In that case in Nipoppy we will probably end up putting it in .bidsignore

[yarikoptic]

yes. I think it should be added to all ignores out there - .bidsignore and .gitignore for a good measure!