Provenance Convention Metadata

• UUID: 0d4ef942-0f74-4187-ae72-016ddf083fe1
• Name: provenance:
• Schema URL: "https://raw.githubusercontent.com/BioImageTools/zarr-convention-provenance/refs/tags/v0.2.0/schema.json"
• Spec URL: "https://github.com/BioImageTools/zarr-convention-provenance/blob/v0.2.0/README.md"
• Scope: Array, Group
• Extension Maturity Classification: Proposal
• Owner: @BioImageTools/zarr-convention-mantainers

Description

This convention records the processing steps that produced or modified a Zarr node: which software ran, with which parameters, on which inputs, writing which outputs, and when. Properties use the provenance: prefix at the root attributes level, following the Zarr Conventions Specification.

Version 0.1 contained only what every surveyed producer already records (see Survey). Version 0.2 adds optional, nice-to-have ontology terms for the kind of operation a step performed and the type and format of the data it read and wrote, and one optional step field: environment. The convention says nothing about the image itself: axes, transforms, pyramids and labels belong to OME-Zarr, and licensing to the license convention. Every object here allows extra keys, so tools and later versions can add more without breaking validation.

It merges two drafts from the November 2026 BioImageTools hackathon: this repository and BioinfoTongLI/bioimage-analysis-spec v0.1.0, whose step list, data references and survey it adopts.

• Examples:
  • Minimal: one step, software only
  • Format conversion, TensorSwitch (Janelia)
  • Format conversion, BioImage Archive
  • Sparse staging, MIA harness (Janelia)
  • Segmentation, Fractal task run from Nextflow
  • A plate image after five steps: collection, conversion, illumination correction, montage, channel alignment
  • The same as a Zarr v2 .zattrs

Motivation

• Provenance travels with the data. Images are copied, converted and re-hosted; a side file or database row is lost on the way, a node's own attributes are not.
• One place to look. Today each tool writes its own key (_software, tensorswitch, mia, additional_metadata, processing.json). A reader should not need to know the tool to learn what happened.
• Multi-step history. A label image or table is usually several steps deep (conversion, preprocessing, segmentation, measurement). An ordered list of steps keeps that history.
• Reproducibility. "Which software, version and parameters made this?" is the question behind a paper, a re-run, or a check that two datasets were processed the same way.

Convention Registration

The convention must be registered in zarr_conventions:

{
  "zarr_conventions": [
    {
      "uuid": "0d4ef942-0f74-4187-ae72-016ddf083fe1",
      "schema_url": "https://raw.githubusercontent.com/BioImageTools/zarr-convention-provenance/refs/tags/v0.2.0/schema.json",
      "spec_url": "https://github.com/BioImageTools/zarr-convention-provenance/blob/v0.2.0/README.md",
      "name": "provenance:",
      "description": "The processing steps that produced or modified this node"
    }
  ]
}

Applicable To

• Group
• Array

Properties

All properties are placed at the root attributes level using the key-prefixed pattern.

Convention metadata name: provenance:

Field Name	Type	Description
provenance:steps	[Step Object]	REQUIRED. Steps that created or modified this node, in execution order. At least one.

Example:

{
  "attributes": {
    "zarr_conventions": [{ "name": "provenance:", "uuid": "0d4ef942-0f74-4187-ae72-016ddf083fe1" }],
    "provenance:steps": [
      {
        "name": "Threshold Segmentation",
        "category": "Segmentation",
        "software": { "name": "fractal-tasks-core", "version": "2.1.0" },
        "category_terms": [
          { "uri": "http://purl.obolibrary.org/obo/OBI_0003918", "label": "image segmentation", "ontology": "obi" }
        ],
        "parameters": { "segmentation_method": { "method": "Otsu" } },
        "inputs": [
          {
            "path": "B/03/0",
            "name": "image",
            "type": { "uri": "http://edamontology.org/data_2968", "label": "Image", "ontology": "edam" }
          }
        ],
        "outputs": [
          {
            "path": "B/03/0/labels/nuclei",
            "name": "label image",
            "type": { "uri": "http://purl.obolibrary.org/obo/OBI_0003360", "label": "image segmentation map", "ontology": "obi" }
          }
        ],
        "started": "2026-11-04T11:55:00+01:00",
        "ended": "2026-11-04T11:55:02+01:00"
      }
    ]
  }
}

provenance:steps

• Placement. Written on the node a step created or modified. A tool that modifies a node SHOULD append its step to the existing list rather than replace it.
• Order. Steps are listed in the order they ran; the last is the most recent.
• Upstream steps. A node MAY repeat the steps that produced its inputs, or a reader can follow inputs to the input node and read its own provenance:steps.
• Zarr v2. The attributes object is the content of .zattrs; provenance:steps sits beside multiscales or image-label. The schema validates either a zarr.json document or a bare attributes object.

Step Object

One execution of one step: a conversion, a filter, a segmentation, a measurement.

Field Name	Type	Description
software	Software Object	REQUIRED. The software that performed the step.
name	string	What the step did, e.g. Image conversion.
category	string	The kind of processing, e.g. Segmentation, Conversion.
category_terms	[Ontology Term Object]	The kind of processing as ontology terms, e.g. image segmentation.
description	string	Free text about the step.
parameters	object	Parameters the software ran with. Free-form.
inputs	[Data Reference Object]	Data the step read.
outputs	[Data Reference Object]	Data the step wrote, if not only this node.
environment	Data Reference Object	What the software ran in: a container image, lockfile or environment file, with its checksum.
started	string	When the step started: ISO 8601 date-time with time zone.
ended	string	When the step ended: ISO 8601 date-time with time zone.

category groups steps across tools, so that a reader can find, say, every segmentation in a history without knowing each tool's step names. It is free text. category_terms names the same kind of processing with terms from published ontologies, possibly at several levels of detail (for example a general EDAM term and a more specific OBI term), so that readers can match steps without agreeing on the free text.

A step MAY list the node it wrote among its outputs, to carry that node's identifier in a catalogue or its checksum. A node that is checked or otherwise modified after it was written gets a further step appended. A step records what happened to the data, not how the computation went: a step that failed and wrote nothing is not part of the node's history, and a check that is recorded ran and passed. Exit codes, logs and failed attempts belong to the workflow engine.

parameters SHOULD be recorded as the software received them, with defaults applied where the software can tell, so that the step can be repeated. A command-line tool MAY record its arguments as a list under one key (see the BioImage Archive example).

Software Object

Field Name	Type	Description
name	string	REQUIRED. Name of the software, package or tool.
version	string	REQUIRED. Released version, e.g. 2.1.0. Use the commit if there is no release.
entry_point	string	The part of the software that ran: a module, command or function.
uri	string	Where the software can be found: repository or package page.

One package often holds many tools: entry_point says which of them ran, e.g. the Python module mypackage.tasks.segment_nuclei, the command mytool convert, or a function name.

Data Reference Object

At least one of path, uri or id is REQUIRED.

Field Name	Type	Description
path	string	Path of a node relative to the root of this Zarr hierarchy, e.g. B/03/0.
uri	string	URI of data outside this hierarchy, or where a node of it was when read.
id	string	Identifier of the data in a catalogue or archive, e.g. a UUID or accession.
name	string	Human-readable name or role of the data, e.g. image, label image.
type	Ontology Term Object	What kind of data this is, e.g. an image or a segmentation map.
format	Ontology Term Object	The file or storage format, e.g. Zarr or OME-TIFF.
checksum	string	<algorithm>:<lowercase hex digest>, e.g. sha256:9f86d0….

A node of this hierarchy is identified by its path, which stays right when the hierarchy is copied or moved. It MAY also carry a uri: the absolute location it had when the step ran, e.g. s3://bucket/plate.zarr/B/03/0, for a reader to trace where the data was. That uri is a record, not a reference: once the hierarchy has moved it can point to an old location, so a reader SHOULD resolve the node from its path.

Code or model files a step loaded at run time that its environment does not pin, such as reader plugins, task scripts or trained weights, are inputs like any other: a uri pinned to a commit or release, a checksum when known, and a name that says what it is (snippet, model). The output depends on them exactly as it depends on the data.

Ontology Term Object

A term from a published ontology, so that readers can find, for example, every segmentation step or every label image without knowing each tool's own step names. Terms can be looked up in the EMBL-EBI Ontology Lookup Service (OLS).

Ontology terms are nice-to-have: category_terms, type and format are all optional, and so is every field of a term. A writer that only knows a label can record just the label and add the URI later.

Field Name	Type	Description
uri	string	Full URI of the term, e.g. http://edamontology.org/operation_3443.
label	string	The term's label in its ontology, e.g. Image analysis.
ontology	string	The ontology's id in OLS, e.g. edam, obi, ncit.

When an identifier is given, it should be the full URI rather than a compact identifier (CURIE), because prefixes are not used consistently: OLS itself shows the same EDAM term as EDAM:operation_3434 and EDAM_operation:3434. If uri is present it must be a valid URI.

Any ontology may be used. These are suggested:

Where	Ontology and branch	Examples
Step category_terms	EDAM operation, then OBI or NCIT	Image analysis, Data formatting, Feature extraction, image segmentation (OBI), Cell Segmentation (NCIT), Image Registration (NCIT)
Step category_terms, deconvolution	FBbi computational method	3D-deconvolution, blind deconvolution, Wiener filter
Data reference type	EDAM data, then OBI	Image, image segmentation map (OBI)
Data reference format	EDAM format	Zarr, OME-TIFF, TIFF

A step MAY list several category_terms, from one ontology or several, for example to give both a general and a more specific term. Their order carries no meaning. A cell segmentation step could record:

"category": "Segmentation",
"category_terms": [
  { "uri": "http://edamontology.org/operation_3443", "label": "Image analysis", "ontology": "edam" },
  { "uri": "http://purl.obolibrary.org/obo/OBI_0003918", "label": "image segmentation", "ontology": "obi" },
  { "uri": "http://purl.obolibrary.org/obo/NCIT_C224053", "label": "Cell Segmentation", "ontology": "ncit" }
]

Extending

Every object allows additional properties. Tools MAY add their own keys, preferably prefixed (e.g. fractal:task_name inside a step), without breaking validation.

Versioning

The version is part of the schema and spec URLs and follows semantic versioning: patch for wording, minor for new optional fields, major for incompatible changes. While the major version is 0, minor versions may also be incompatible.

• 0.2.0: optional step category and software entry_point; a data reference's uri may record where a node of this hierarchy was when read; optional ontology terms: step category_terms, data reference type and format; optional step environment.
• 0.1.0: first version.

Survey of existing practice

Version 0.1 keeps the concepts every source below records, or that all but one record and the last can trivially add. Software name and version are required because every source has them.

Concept	BioImage Archive	TensorSwitch	MIA harness	Allen processing.json	Fractal / Nextflow	In v0.1
Software name and version	conversion_function	_software	harness_commit	code.name, code.version	yes	software (required)
Parameters	conversion_config	tensorswitch	plan	code.parameters	yes	parameters
Inputs	input UUID	lmvd.source_path	derived_from	code.input_data	yes	inputs
Software URL		_software.url		code.url	yes	software.uri
Step name	provenance			process_type, name	yes	name
Start and end time			verification.time	start_date_time, end_date_time	yes	started, ended
Outputs	(the node itself)	(the node itself)	artifact	output_path	yes	outputs

Added in 0.2

• environment: every surveyed producer runs in something it can name (a container image, a conda or pixi lockfile, a Nextflow profile); a data reference with a checksum covers all of them without a new object type.
• Code loaded at run time that the environment does not pin (reader plugins, Fractal tasks, trained models: the working group's "runtime modules, DL models") needs no field of its own: it is an input with a pinned uri.

Deferred

Recorded by some sources or asked for by the working group, but not yet agreed. Each is a candidate for a minor version, and can be written today as an extra key.

• Commit separate from version (MIA, Allen, Fractal).
• Workflow the step ran in: name, version, uri (MIA, Allen, Fractal).
• Execution id: UUID or workflow-engine task id (BIA, Fractal).
• Compute: CPU, memory, GPU model and count (Allen, Fractal; working group).
• Manual steps, e.g. proofreading, with the BioImage Archive MIFA model as vocabulary (working group).
• Scripts, logs and fuller records, e.g. a Workflow Run RO-Crate (working group).
• Node identity: a recipe hash over inputs and parameters (MIA).
• Verification outcome and detail: a status or quality flag on a check step, and which checks ran and what each found (MIA). Kept out of 0.2 because a recorded step is one that ran and passed; a failed check should block publication or is a quality flag, which is a different concept from provenance.

Out of scope: license (license convention), people and funding (dataset-level metadata such as Allen biodata-schema), image geometry (OME-Zarr).

Known Implementations

Libraries and Tools

• fractal-multistep-nextflow: Nextflow pipeline chaining fractal-tasks-core tasks on OME-Zarr. Status: planned. Maintainer: @BioinfoTongLI.

Datasets Using This Convention

None yet.

If you implement or use this convention, please add your implementation to this list by submitting a pull request.

Acknowledgements

Follows the zarr-conventions template, itself based on the STAC extensions template. The step model and survey come from BioinfoTongLI/bioimage-analysis-spec; the survey draws on provenance written by the BioImage Archive, Janelia (TensorSwitch, MIA), the Allen Institute and the Fractal / Nextflow examples. Related: the STAC processing extension.