[WIP] Use JSON/TOML template for defining openPMD metadata in a config file #1277

franzpoeschel · 2022-05-17T15:56:54Z

Not relevant for next release

Until now: Simulations specify their metadata in-code via API calls
With this PR: In some workflows (e.g. experiments) there is no omniscient simulation, but metadata is instead input by the experimentors via configuration files, using the API is not a good workflow for that

Idea: We already have a JSON backend, use an openPMD-conforming JSON dataset to define only metadata. With this, the configuration file will be just another openPMD dataset.
Then, add some functionality to initialize an empty Series from such a metadata file.

TODO:

https://github.com/franzpoeschel/openPMD-api/compare/topic-json-short-modes..topic-json-template

franzpoeschel · 2022-05-18T12:28:26Z

An openPMD dataset in TOML:

[platform_byte_widths]
USHORT = 2
ULONG = 8
BOOL = 1
CLONG_DOUBLE = 32
LONGLONG = 8
CFLOAT = 8
CHAR = 1
DOUBLE = 8
CDOUBLE = 16
SHORT = 2
UCHAR = 1
FLOAT = 4
INT = 4
ULONGLONG = 8
UINT = 4
LONG = 8
LONG_DOUBLE = 16

[data]

[data.0]

[data.0.meshes]

[data.0.meshes.E]

[data.0.meshes.E.x]
datatype = "FLOAT"
data = [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

[data.0.meshes.E.x.attributes]

[data.0.meshes.E.x.attributes.unitSI]
value = 1.0
datatype = "DOUBLE"

[data.0.meshes.E.x.attributes.position]
value = [0.0]
datatype = "VEC_DOUBLE"

[data.0.meshes.E.attributes]

[data.0.meshes.E.attributes.timeOffset]
value = 0.0
datatype = "FLOAT"

[data.0.meshes.E.attributes.gridUnitSI]
value = 1.0
datatype = "DOUBLE"

[data.0.meshes.E.attributes.gridSpacing]
value = [1.0]
datatype = "VEC_DOUBLE"

[data.0.meshes.E.attributes.gridGlobalOffset]
value = [0.0]
datatype = "VEC_DOUBLE"

[data.0.meshes.E.attributes.unitDimension]
value = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
datatype = "ARR_DBL_7"

[data.0.meshes.E.attributes.geometry]
value = "cartesian"
datatype = "STRING"

[data.0.meshes.E.attributes.dataOrder]
value = "C"
datatype = "STRING"

[data.0.meshes.E.attributes.axisLabels]
value = ["x"]
datatype = "VEC_STRING"

[data.0.attributes]

[data.0.attributes.timeUnitSI]
value = 1.0
datatype = "DOUBLE"

[data.0.attributes.time]
value = 0.0
datatype = "DOUBLE"

[data.0.attributes.dt]
value = 1.0
datatype = "DOUBLE"

[attributes]

[attributes.softwareVersion]
value = "0.15.0-dev"
datatype = "STRING"

[attributes.software]
value = "openPMD-api"
datatype = "STRING"

[attributes.openPMDextension]
value = 0
datatype = "UINT"

[attributes.meshesPath]
value = "meshes/"
datatype = "STRING"

[attributes.iterationFormat]
value = "many_iterations_%T"
datatype = "STRING"

[attributes.iterationEncoding]
value = "fileBased"
datatype = "STRING"

[attributes.openPMD]
value = "1.1.0"
datatype = "STRING"

[attributes.date]
value = "2022-05-18 12:20:23 +0000"
datatype = "STRING"

[attributes.basePath]
value = "/data/%T/"
datatype = "STRING"

franzpoeschel · 2022-05-19T12:01:00Z

This is now a simplified TOML openPMD template, created by {"json":{"mode": "template"}}:

[data]

[data.meshes]

[data.meshes.temperature]
extent = [5, 5]
datatype = "FLOAT"

[data.meshes.temperature.attributes]
timeOffset = 0.0
# Explicit datatype can still be used if needed
unitSI = {"value" = 1.0, "datatype" = "FLOAT"}
position = [0.0]
gridUnitSI = 1.0
gridSpacing = [1.0]
gridGlobalOffset = [0.0]
unitDimension = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
geometry = "cartesian"
dataOrder = "C"
axisLabels = ["x"]

[data.attributes]
timeUnitSI = 1.0
snapshot = 0
time = 0.0
dt = 1.0

[attributes]
softwareVersion = "0.15.0-dev"
software = "openPMD-api"
openPMDextension = 0
meshesPath = "meshes/"
iterationFormat = "/data"
iterationEncoding = "variableBased"
openPMD = "1.1.0"
date = "2022-05-19 11:55:07 +0000"
basePath = "/data"

Differences to regular JSON/TOML openPMD datasets:

Platform byte width table is missing
Attributes don't explicitly store their datatypes, datatypes are dynamically (and a bit heuristically) restored from what is there.
No actual datasets can be written, instead just the extent is stored.

Template mode is also available in json:

{
  "attributes": {
    "basePath": "/data",
    "date": "2022-05-19 12:00:09 +0000",
    "iterationEncoding": "variableBased",
    "iterationFormat": "/data",
    "meshesPath": "meshes/",
    "openPMD": "1.1.0",
    "openPMDextension": 0,
    "software": "openPMD-api",
    "softwareVersion": "0.15.0-dev"
  },
  "data": {
    "attributes": {
      "dt": 1,
      "snapshot": 0,
      "time": 0,
      "timeUnitSI": 1
    },
    "meshes": {
      "temperature": {
        "attributes": {
          "axisLabels": [
            "x"
          ],
          "dataOrder": "C",
          "geometry": "cartesian",
          "gridGlobalOffset": [
            0
          ],
          "gridSpacing": [
            1
          ],
          "gridUnitSI": 1,
          "position": [
            0
          ],
          "timeOffset": 0,
          "unitDimension": [
            0,
            0,
            0,
            0,
            0,
            0,
            0
          ],
          "unitSI": 1
        },
        "datatype": "FLOAT",
        "extent": [
          5,
          5
        ]
      }
    }
  }
}

franzpoeschel · 2022-05-19T15:27:50Z

Longer example:

[data]

[data.particles]

[data.particles.e]

[data.particles.e.positionOffset]

[data.particles.e.positionOffset.z]

[data.particles.e.positionOffset.z.attributes]
value = 3.14
unitSI = 1.0
shape = [5, 5]

[data.particles.e.positionOffset.y]

[data.particles.e.positionOffset.y.attributes]
value = 3.14
unitSI = 1.0
shape = [5, 5]

[data.particles.e.positionOffset.x]

[data.particles.e.positionOffset.x.attributes]
value = 3.14
unitSI = 1.0
shape = [5, 5]

[data.particles.e.positionOffset.attributes]
unitDimension = [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
timeOffset = 0.0

[data.particles.e.position]

[data.particles.e.position.z]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.position.z.attributes]
unitSI = 1.0

[data.particles.e.position.y]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.position.y.attributes]
unitSI = 1.0

[data.particles.e.position.x]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.position.x.attributes]
unitSI = 1.0

[data.particles.e.position.attributes]
unitDimension = [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
timeOffset = 0.0

[data.particles.e.particlePatches]

[data.particles.e.particlePatches.numParticlesOffset]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.numParticlesOffset.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.numParticles]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.numParticles.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.offset]

[data.particles.e.particlePatches.offset.z]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.offset.z.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.offset.y]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.offset.y.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.offset.x]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.offset.x.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.offset.attributes]
unitDimension = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

[data.particles.e.particlePatches.extent]

[data.particles.e.particlePatches.extent.z]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.extent.z.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.extent.y]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.extent.y.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.extent.x]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.extent.x.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.extent.attributes]
unitDimension = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

[data.meshes]

[data.meshes.temperature]
extent = [5, 5]
datatype = "FLOAT"

[data.meshes.temperature.attributes]
timeOffset = 0.0
unitSI = 1.0
position = [0.0]
gridUnitSI = 1.0
gridSpacing = [1.0]
gridGlobalOffset = [0.0]
unitDimension = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
geometry = "cartesian"
dataOrder = "C"
axisLabels = ["x"]

[data.meshes.E]

[data.meshes.E.z]
extent = [5, 5]
datatype = "FLOAT"

[data.meshes.E.z.attributes]
unitSI = 1.0
position = [0.0]

[data.meshes.E.y]
extent = [5, 5]
datatype = "FLOAT"

[data.meshes.E.y.attributes]
unitSI = 1.0
position = [0.0]

[data.meshes.E.x]
extent = [5, 5]
datatype = "FLOAT"

[data.meshes.E.x.attributes]
unitSI = 1.0
position = [0.0]

[data.meshes.E.attributes]
timeOffset = 0.0
gridUnitSI = 1.0
gridSpacing = [1.0]
gridGlobalOffset = [0.0]
unitDimension = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
geometry = "cartesian"
dataOrder = "C"
axisLabels = ["x"]

[data.attributes]
timeUnitSI = 1.0
snapshot = 0
time = 0.0
dt = 1.0

[attributes]
softwareVersion = "0.15.0-dev"
particlesPath = "particles/"
software = "openPMD-api"
openPMDextension = 0
meshesPath = "meshes/"
iterationFormat = "/data"
iterationEncoding = "variableBased"
openPMD = "1.1.0"
date = "2022-05-19 15:26:37 +0000"
basePath = "/data"

franzpoeschel · 2022-07-21T16:38:54Z

Notes for myself on the recent reodering of commits:

5 3ee509de (HEAD -> topic-json-template, origin/topic-json-template) Properly deal with undefined datasets
2 06da2d58 Make JSON and TOML look like two different backends
5 960ab21a Initialize Dataset definitions from template
5 b88bae67 Initialize Series attributes from template
3 6302a33c Fix NVHPC Toml11 open mode
2 d825008b Fix precision-losing type conversion
4 da960a23 Enable .toml tests in generic tests
4 0398b86f Extend example
3 7332996e Windows compatibility
x 85527799 Add and use Attribute::getOptional<T>()
1 64cde966 Template mode: Fill with zero upon read
1 fa483843 Write/read shorthand attributes without explicit datatype
3 bd8da013 CI fixes
1 d802d2ac Don't write platform datatype size table in template mode
2 cba71f7f Use .toml as filename extension
2 b019a7d1 TOML as alternative backend for JSON backend
1 4b25de8c Select template mode via JSON param
1 8ef4753f Add template mode to JSON backend

franzpoeschel · 2022-09-15T09:23:37Z

src/RecordComponent.cpp

+        //     throw error::WrongAPIUsage(
+        //         "[RecordComponent] Must set specific datatype (Use "
+        //         "resetDataset call).");
+        // }


Note that this check was inactive up to now, since RecordComponentData::RecordComponentData initialized that field with Datatype::CHAR. Using an optional would make these things more obvious and avoid such pitfalls.
To be done in a different PR though

#1316 now uses std::optional

for more information, see https://pre-commit.ci

Reading the chunk table requires NOT using template mode, otherwise the string just consists of '\0' bytes.

Has the same implementation as JSON anyway, and it makes tests run into timeouts otherwise.

franzpoeschel added backend: JSON api: new additions to the API labels May 17, 2022

franzpoeschel force-pushed the topic-json-template branch from f10fc90 to c63c06a Compare May 18, 2022 12:26

franzpoeschel force-pushed the topic-json-template branch 3 times, most recently from 0d475a5 to 1a23a03 Compare May 19, 2022 11:54

franzpoeschel force-pushed the topic-json-template branch from 1a23a03 to 656e740 Compare May 19, 2022 12:33

franzpoeschel mentioned this pull request May 19, 2022

Add Attribute::getOptional<T>() and use to add some more dynamic datatype conversions at read time #1278

Merged

franzpoeschel force-pushed the topic-json-template branch 5 times, most recently from 475be7b to ee8bdf1 Compare May 23, 2022 11:14

franzpoeschel changed the title ~~Use JSON/TOML template for defining openPMD metadata in a config file~~ [WIP] Use JSON/TOML template for defining openPMD metadata in a config file May 23, 2022

franzpoeschel force-pushed the topic-json-template branch 3 times, most recently from d32fff3 to 376bc2a Compare July 5, 2022 09:23

franzpoeschel force-pushed the topic-json-template branch 2 times, most recently from 3ee509d to a662865 Compare July 21, 2022 16:35

franzpoeschel force-pushed the topic-json-template branch 2 times, most recently from 1db63f6 to 55b72f8 Compare July 29, 2022 09:00

franzpoeschel force-pushed the topic-json-template branch 2 times, most recently from b07a2a8 to a41c2c6 Compare August 17, 2022 09:21

franzpoeschel commented Sep 15, 2022

View reviewed changes

franzpoeschel mentioned this pull request Oct 4, 2022

RecordComponent: Properly handle uninitialized datasets #1316

Merged

6 tasks

franzpoeschel force-pushed the topic-json-template branch from a41c2c6 to c6a369c Compare October 12, 2022 09:45

franzpoeschel and others added 19 commits August 5, 2024 11:38

Introduce attribute mode

a62841e

Add example 14_toml_template.cpp

8a8e794

Use Datatype::UNDEFINED to indicate no dataset definition in template

f24ae79

Extend example

59c93c9

Test short attribute mode

5a7ed2e

Copy datatypeToString to JSON implementation

c25743f

Fix after rebase: Init JSON config in parallel mode

c8dd534

Fix after rebase: Don't erase JSON datasets when writing

baf1bdc

openpmd-pipe: use short modes for test

0721e6e

Less intrusive warnings, allow disabling them

fd4a00a

TOML: Use short modes by default

c35c2e4

Python formatting

e2fa15e

[pre-commit.ci] auto fixes from pre-commit.com hooks

deba1e9

for more information, see https://pre-commit.ci

Documentation

bd357fe

Short mode in default in openPMD >= 2.

a75060e

Short value by default in TOML

3f95d77

Store the openPMD version information in the IOHandler

1e948b2

Fixes

dcb16d0

Adapt test to recent rebase

a4a0771

Reading the chunk table requires NOT using template mode, otherwise the string just consists of '\0' bytes.

franzpoeschel force-pushed the topic-json-template branch from b44f0a9 to df04b3e Compare August 5, 2024 09:39

franzpoeschel added 9 commits August 5, 2024 11:43

toml11 4.0 compatibility

20d6502

Initialize Series attributes and datasets from template

94a8b9f

Further testing (@todo: cleanup)

21a7158

Extend example

1d9257c

Only opt-into tests for TOML

aeea63c

Has the same implementation as JSON anyway, and it makes tests run into timeouts otherwise.

Adapt this to changed SCALAR API

d5b437d

Avoid CI warnings

c1e2110

Fix after rebasing

b8e0537

Ensure that E.z is there

074a93d

franzpoeschel force-pushed the topic-json-template branch from df04b3e to 074a93d Compare August 5, 2024 09:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Use JSON/TOML template for defining openPMD metadata in a config file #1277

[WIP] Use JSON/TOML template for defining openPMD metadata in a config file #1277

franzpoeschel commented May 17, 2022 •

edited

Loading

franzpoeschel commented May 18, 2022 •

edited

Loading

franzpoeschel commented May 19, 2022 •

edited

Loading

franzpoeschel commented May 19, 2022

franzpoeschel commented Jul 21, 2022

franzpoeschel Sep 15, 2022

franzpoeschel May 9, 2023

[WIP] Use JSON/TOML template for defining openPMD metadata in a config file #1277

Are you sure you want to change the base?

[WIP] Use JSON/TOML template for defining openPMD metadata in a config file #1277

Conversation

franzpoeschel commented May 17, 2022 • edited Loading

franzpoeschel commented May 18, 2022 • edited Loading

franzpoeschel commented May 19, 2022 • edited Loading

franzpoeschel commented May 19, 2022

franzpoeschel commented Jul 21, 2022

franzpoeschel Sep 15, 2022

Choose a reason for hiding this comment

franzpoeschel May 9, 2023

Choose a reason for hiding this comment

franzpoeschel commented May 17, 2022 •

edited

Loading

franzpoeschel commented May 18, 2022 •

edited

Loading

franzpoeschel commented May 19, 2022 •

edited

Loading