[ENH][BEP028] Specification update for BEP028 BIDS-Prov #2099

bclenet · 2025-04-10T15:04:58Z

This is a work in progress PR proposing a specification update for BEP028 BIDS-Prov.

BIDS specification changes
- [ ] being proofread
BIDS schema changes
- [ ] validator error : /prov/* NOT_INCLUDED
- [ ] validator error : /prov/*.json SIDECAR_WITHOUT_DATAFILE
- [ ] validator error : derivative files are listed as NOT_INCLUDED / ALL_FILENAME_RULES_HAVE_ISSUES /FILENAME_MISMATCH / ENTITY_WITH_NO_LABEL
BIDS examples

for more information, see https://pre-commit.ci

src/schema/rules/files/common/tables.yaml

yarikoptic

some initial notes

yarikoptic · 2025-09-02T17:08:48Z

src/modality-agnostic-files/provenance.md

+!!! bug
+    TODO: Environment not currently defined in the context


what to be done about that?

should be added.

Is there an IRI from PROV-O or from any other ontology we should map with Enviroment ?

Environment is a prov:Entity

yarikoptic · 2025-09-02T17:13:11Z

src/modality-agnostic-files/provenance.md

+!!! example "`Activity`, `Environment`, `Software` naming examples"
+    - `bids:ds001734:prov#conversion-xfMMbHK1`: a conversion `Activity` described inside the `ds001734` dataset;
+    - `bids::prov#fedora-uldfv058`: a Fedora based `Environment` described inside the current dataset.
+    - `bids:derivatives:prov#fmriprep-r4kzzMt8`: the fMRIPrep `Software` described inside the `derivatives` dataset.


I feel we would need some very explicit statement about uniqueness of Ids, and enforce that in validator somehow.

Suggested change

- `bids:derivatives:prov#fmriprep-r4kzzMt8`: the fMRIPrep `Software` described inside the `derivatives` dataset.

- `bids:derivatives:prov#fmriprep-r4kzzMt8`: the fMRIPrep `Software` described inside the `derivatives` dataset.

An `Id` must be unique, as to identify an instance and thus REQUIRED to be different across entries and different instances MUST have different `Id`s.

see my suggestion above.

I added a sentence in the Consistency and uniqueness of Ids section, as suggested by Satra. Please let me know if it is ok for both of you.

yarikoptic · 2025-09-02T17:14:01Z

src/modality-agnostic-files/provenance.md

+
+!!! example
+    The following graph represents examples of links between provenance records. In this example, `Entities` *sub-001_brainmask.nii* and *sub-001_T1w.nii* represent files used by `Activity` *Brain extraction* to generate another file represented by `Entity` *sub-001_T1w_preproc.nii*. `Activities` *Brain extraction* and *Move to MNI* were associated with the `Software` *FSL* and used `Environment` *Linux* as a software environment.
+    ![](../images/provenance_definitions_graph.png)


is there a way to ship an original graph or ideally a collection of prov files + script to produce such a graph?

For now, the bids_prov.merge and bids_prov.visualize python modules are available in https://github.com/bids-standard/BEP028_BIDSprov/, to produce such a graph from a collection of prov files.
Would you like to have the prov files for this minimal example inside bids examples ?

IMHO prov files should be provided in this repo. makefile could be provided to generate them invoking code from that package, or even provide that code here (may be later).

@yarikoptic I just discussed this with Boris. I am not sure I follow here, can you detail a bit? I mean typically examples have been saved in BIDS-examples. Is there a specific reason here to instead have the prov files in the repo of the BIDS standard? If so, could you describe a bit more how this would look like.

src/modality-agnostic-files/provenance.md

yarikoptic · 2025-09-02T17:23:06Z

src/modality-agnostic-files/provenance.md

+   }
+) }}
+
+The `dataset_description.json` file of a BIDS-Derivatives dataset MUST include the following key to describe provenance:


this is from the "common-principles", so better be referred to it?

Do you suggest to remove all metadata tables from the Provenance at dataset level section because these metadata are already listed in src/modality-agnostic-files/dataset-description.json ?

satra

looks very nice @bclenet - left some comments which should be very lightweight to address.

src/modality-agnostic-files/dataset-description.md

src/modality-agnostic-files/provenance.md

satra · 2025-09-03T00:37:28Z

src/modality-agnostic-files/provenance.md

+            {
+                "Id": "bids::sub-01/func/sub-01_task-tonecounting_bold.nii",
+                "Label": "sub-01_task-tonecounting_bold.nii",
+                "AtLocation": "sub-01/func/sub-01_task-tonecounting_bold.nii",


any reason not to use the same uri as Id here?

For the Label, I guess we want to use the name of the file (without directory path) so that is more human readable.
About AtLocation, we describe this metadata as:

For input files, this is the relative path to the file on disk.

Should we be more specific for this definition ?

i meant to use the bids uri here as well.

src/modality-agnostic-files/provenance.md

satra · 2025-09-03T00:44:52Z

src/modality-agnostic-files/provenance.md

+!!! example "`Activity`, `Environment`, `Software` naming examples"
+    - `bids:ds001734:prov#conversion-xfMMbHK1`: a conversion `Activity` described inside the `ds001734` dataset;
+    - `bids::prov#fedora-uldfv058`: a Fedora based `Environment` described inside the current dataset.
+    - `bids:derivatives:prov#fmriprep-r4kzzMt8`: the fMRIPrep `Software` described inside the `derivatives` dataset.


see my suggestion above.

satra · 2025-09-03T00:45:45Z

src/modality-agnostic-files/provenance.md

+
+## Minimal example
+
+Here is a comprehensive example that considers the following dataset:


for these examples, do consider the changes about study-level organization and what that means for prov.

Hi @satra! I discussed this with Boris. Can you give us some more insights on what you mean by "study-level organization". Is this a change in the BIDS spec? Or do you mean a re-organization of the files within a BIDS folder for a given study?

here is the relevant doc: https://bids-specification.readthedocs.io/en/stable/common-principles.html#study-dataset this was merged with the recent release.

satra · 2025-09-03T00:47:43Z

src/modality-agnostic-files/provenance.md

+The following provenance record is defined in `prov/prov-dcm2niix_soft.json`, so its `Id` SHOULD start with `bids:<dataset>:prov#` or `bids::prov#`.
+
+```JSON
+{
+    "Software": [
+        {
+            "Id": "bids::prov#dcm2niix-70ug8pl5",
+            "Label": "dcm2niix",
+            "Version": "v1.1.3"
+        }
+    ]
+}
+```


not sure i'm following this one. does the Id really need to match the filename, or would that simply be a good convention?

i say this because in reproschema we make the id match the filename so the file can be accessed through a http server. here it doesn't seem necessary and also because the suffix-id would be unique.

I just rephrased this ; is it ok ?

We simply give an example illustrating the Consistency and uniqueness of Ids section here. There is indeed no correlation between the file prov/prov-dcm2niix_soft.json and the fact that the Id starts with bids:<dataset>:prov#

yarikoptic

Next batch of comments and recommendations

src/common-principles.md

yarikoptic · 2025-09-03T12:59:37Z

src/modality-agnostic-files/dataset-description.md

- https://github.com/bids-standard/bids-specification/blob/master/macros_doc.md
-->
-{{ MACROS___make_subobject_table("metadata.GeneratedBy.items") }}
-


All these removals is text and headings leave examples "hanging". If all of this is to move to the provenance section, surgery might be needed to become more thorough and have explicit reference to that section.

There is a link to the Provenance section in the description of GeneratedBy to cover the fact that this table was moved to the Provenance section.

yarikoptic · 2025-09-03T13:03:49Z

src/modality-agnostic-files/provenance.md

+
+### Principles for encoding provenance in BIDS
+
+-   Provenance information SHOULD be included in a BIDS dataset when possible.


I believe that GeneratedBy is just MAY, so here it then should also be such to be consistent

Suggested change

- Provenance information SHOULD be included in a BIDS dataset when possible.

- Provenance information MAY be included in a BIDS dataset when possible.

GeneratedBy is RECOMMENDED in BIDS datasets and REQUIRED in BIDS derivative datasets.

yarikoptic · 2025-09-03T13:05:35Z

src/modality-agnostic-files/provenance.md

+
+-   Provenance information SHOULD be included in a BIDS dataset when possible.
+-   If provenance records are included, these MUST be described using the conventions detailed by this specification.
+-   Provenance records MAY be used to reflect the provenance of a full dataset and/or of specific files at any level of the BIDS hierarchy.


I think it is not a matter of possibility or enforcement, rather a factual statement

Suggested change

- Provenance records MAY be used to reflect the provenance of a full dataset and/or of specific files at any level of the BIDS hierarchy.

- Provenance records reflect the provenance of a full dataset and/or of specific files at any level of the BIDS hierarchy.

Here, I think we want to say that provenance records may have different scopes. Would this sentence be better ?

Provenance records MAY describe the provenance of a full dataset or of specific files at any level of the BIDS hierarchy.

src/modality-agnostic-files/provenance.md

yarikoptic · 2025-09-03T13:20:01Z

src/modality-agnostic-files/provenance.md

+
+!!! example
+    The following graph represents examples of links between provenance records. In this example, `Entities` *sub-001_brainmask.nii* and *sub-001_T1w.nii* represent files used by `Activity` *Brain extraction* to generate another file represented by `Entity` *sub-001_T1w_preproc.nii*. `Activities` *Brain extraction* and *Move to MNI* were associated with the `Software` *FSL* and used `Environment` *Linux* as a software environment.
+    ![](../images/provenance_definitions_graph.png)


IMHO prov files should be provided in this repo. makefile could be provided to generate them invoking code from that package, or even provide that code here (may be later).

src/modality-agnostic-files/provenance.md

yarikoptic · 2025-09-03T13:45:30Z

src/modality-agnostic-files/provenance.md

+## Provenance at dataset level
+
+Provenance metadata MAY be stored inside the `dataset_description.json` of any BIDS dataset (or BIDS-Derivatives dataset) it applies to.
+This metadata describes the provenance of the whole dataset.


Similarly to SidecarGeneratedBy vs GeneratedBy, we have ambiguity as to define the scope, ie - does it apply to all files? What if file had it's own records? We could instruct the same need for a copy but it might be too hard to keep consent etc...

We should spell it out one way or another here

I'm not sure to fully understand what you mean.

But we can leave Provenance at dataset level as a (minimal) option to describe the provenance of the dataset as an Entity.
Of course this most probably contains partial information, but this does not prevent more precise descriptions from being included inside sidecars.

bclenet · 2025-09-11T09:46:17Z

Hi @effigies ,

Here are quick questions about the schema modifications in this PR.
I tested the BIDS examples listed in the description of this PR against a BIDS validator using the modified schema, but I still encounter errors.

Provenance files (.json) inside the prov directory are considered as sidecars, leading to SIDECAR_WITHOUT_DATAFILE error files
Provenance files (.json) inside the prov directory are NOT_INCLUDED according to the validator, although

I think this is due to the fact that the src/schema/rules/files/common/modality_agnostic.yaml file I've added has no impact for now.

Also, files inside derivative datasets lead to NOT_INCLUDED / ALL_FILENAME_RULES_HAVE_ISSUES / FILENAME_MISMATCH / ENTITY_WITH_NO_LABEL errors because their names do not conform to BIDS naming. It seems to be contradictory to this part of the spec

Could you please give your thoughts about these ? Let me know if I should post these anywhere else.

Thanks,

satra · 2025-09-14T17:11:49Z

src/images/provenance_definitions_graph.png

@bclenet - wondering if these could be created with mermaid markdown? this should not hold things up, but may be easier to embed and render if possible.

satra · 2025-09-14T17:16:14Z

src/modality-agnostic-files/provenance.md

+        ]
+    }
+    ```
+    This snippet is an extract of the following comprehensive example: [Provenance records for fMRI preprocessing using `fMRIPrep`](https://github.com/bclenet/bids-examples/tree/BEP028_fmriprep/provenance_fmriprep).


if we are going to point to a separate repo, we should note that it will have to be maintained after the PR is merged. perhaps something to think about and consider whether these examples are transitioned into bids examples repo and linked here.

bclenet and others added 30 commits March 18, 2025 11:06

Adding dir at dataset level

3b64a2b

Adding dir at any level

45f6af4

Adding rule for dir at dataset level

4c315ce

Adding objects: JSONLD extension + top level dir

b490c91

Adding object : prov entity

f2a67a6

Add suffixes in object.suffixes

d9bb7b0

GeneratedByProv and SidecarGeneratedBy metadata

64e2c65

prov entitiy and provenance json/jsonld files

acc6af0

provenance json/jsonld files inside new rules file

34fafe1

Merge branch 'master' into BEP028_schema

f73b668

[SCHEMA] removing provenance subdir level description

a1fbb8a

Yaml lint

6622d08

[SPEC] first draft for BEP028 BIDS-Prov

b05b950

[pre-commit.ci] auto fixes from pre-commit.com hooks

9ae1649

for more information, see https://pre-commit.ci

Adapting markdown to mkdocs

f572174

[pre-commit.ci] auto fixes from pre-commit.com hooks

ee0d30d

for more information, see https://pre-commit.ci

Broken paragraph links

b619de1

Citing BIRS Provenance + removing 'BIDS-Prov' from spec

74722b1

Into JSON only provenance files

34411ba

[pre-commit.ci] auto fixes from pre-commit.com hooks

248d2b8

for more information, see https://pre-commit.ci

Into JSON only provenance files

99faa51

Into JSON only provenance files

ac7aff4

Merge branch 'master' into BEP028_schema

6289722

Merge branch 'spec-refac' into BEP028_spec

3fef3dc

Flags for markdown code blocks

03026d1

Merge branch 'master' into BEP028_spec

a3dd018

Typo in provenance.md

0202806

Remark check

58055b7

Removing non latin content

a3cef66

Merge branch 'master' into BEP028_spec

c9a65b7

bclenet added 8 commits July 1, 2025 15:35

Markdown lint

be27631

Merge branch 'master' into BEP028_spec

05022aa

BIDS-Prov citation + Provenance example links

ab7872d

Correcting typos

535705c

Links from snippets to bids-examples

8cf521f

Links from snippets to bids-examples

2c43698

Provenance graph for definitions

d978907

Provenance example graph with neuro context

d4dabb4

effigies added this to the 1.11.0 milestone Aug 27, 2025

yarikoptic reviewed Sep 2, 2025

View reviewed changes

src/schema/rules/files/common/tables.yaml Outdated Show resolved Hide resolved

yarikoptic requested changes Sep 2, 2025

View reviewed changes

satra reviewed Sep 3, 2025

View reviewed changes

bclenet added 6 commits September 3, 2025 10:11

Merge branch 'master' into BEP028_spec

5eb1ec3

Remove unrelated changes

9381bf4

Typo and replace CAN by MAY

5c83813

Examples at the begining of subsections

891730b

Uniqueness of ids

5181dd0

Minimal example > consistency of Ids

c20912c

yarikoptic requested changes Sep 3, 2025

View reviewed changes

bclenet added 6 commits September 3, 2025 16:15

Issue with dataset naming

ff4746a

Simplification in definitions

9f6eeae

Deidentification

2f05124

Merge branch 'master' into BEP028_spec

ca61869

Removing prefix all

7345757

Remove unrelated changes

a3f5238

effigies added the BEP label Sep 4, 2025

Sidecargeneratedby precision

242062c

satra reviewed Sep 14, 2025

View reviewed changes

		!!! bug
		TODO: Environment not currently defined in the context


		## Minimal example

		Here is a comprehensive example that considers the following dataset:


		### Principles for encoding provenance in BIDS

		- Provenance information SHOULD be included in a BIDS dataset when possible.

	- Provenance information SHOULD be included in a BIDS dataset when possible.
	- Provenance information MAY be included in a BIDS dataset when possible.

	- Provenance records MAY be used to reflect the provenance of a full dataset and/or of specific files at any level of the BIDS hierarchy.
	- Provenance records reflect the provenance of a full dataset and/or of specific files at any level of the BIDS hierarchy.

[ENH][BEP028] Specification update for BEP028 BIDS-Prov #2099

Are you sure you want to change the base?

[ENH][BEP028] Specification update for BEP028 BIDS-Prov #2099

Uh oh!

Conversation

bclenet commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

yarikoptic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmaumet Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

satra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bclenet Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmaumet Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bclenet Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bclenet commented Apr 10, 2025 •

edited

Loading

cmaumet Sep 9, 2025 •

edited

Loading

bclenet Sep 3, 2025 •

edited

Loading

cmaumet Sep 9, 2025 •

edited

Loading

bclenet Sep 3, 2025 •

edited

Loading