[WIP] Use JSON/TOML template for defining openPMD metadata in a config file

CMakeLists.txt

@@ -469,6 +469,7 @@ set(CORE_SOURCE
         src/auxiliary/Filesystem.cpp
         src/auxiliary/JSON.cpp
         src/auxiliary/Mpi.cpp
+        src/auxiliary/TemplateFile.cpp
         src/backend/Attributable.cpp
         src/backend/BaseRecordComponent.cpp
         src/backend/MeshRecordComponent.cpp
@@ -771,6 +772,7 @@ set(openPMD_EXAMPLE_NAMES
     10_streaming_read
     12_span_write
     13_write_dynamic_configuration
+    14_toml_template
 )
 set(openPMD_PYTHON_EXAMPLE_NAMES
     2_read_serial
@@ -1383,6 +1385,9 @@ if(openPMD_BUILD_TESTING)
                             ${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe       \
                             --infile ../samples/git-sample/thetaMode/data_%T.bp    \
                             --outfile ../samples/git-sample/thetaMode/data%T.json  \
+                            --outconfig '                                          \
+                                json.attribute.mode = \"short\"                  \n\
+                                json.dataset.mode = \"template_no_warn\"'          \
                         "
                     WORKING_DIRECTORY ${openPMD_RUNTIME_OUTPUT_DIRECTORY}
                 )

docs/source/backends/json.rst

@@ -38,20 +38,46 @@ when working with the JSON backend.
 Datasets and groups have the same namespace, meaning that there may not be a subgroup
 and a dataset with the same name contained in one group.
 
-Any **openPMD dataset** is a JSON object with three keys:
+Datasets
+........
 
- * ``attributes``: Attributes associated with the dataset. May be ``null`` or not present if no attributes are associated with the dataset.
- * ``datatype``: A string describing the type of the stored data.
- * ``data`` A nested array storing the actual data in row-major manner.
+Datasets can be stored in two modes, either as actual datasets or as dataset templates.
+The mode is selected by the :ref:`JSON/TOML parameter<backendconfig>` ``json.dataset.mode`` (resp. ``toml.dataset.mode``) with possible values ``["dataset", "template"]`` (default: ``"dataset"``).
+
+Stored as an actual dataset, an **openPMD dataset** is a JSON object with three JSON keys:
+
+ * ``datatype`` (required): A string describing the type of the stored data.
+ * ``data`` (required): A nested array storing the actual data in row-major manner.
    The data needs to be consistent with the fields ``datatype`` and ``extent``.
    Checking whether this key points to an array can be (and is internally) used to distinguish groups from datasets.
+ * ``attributes``: Attributes associated with the dataset. May be ``null`` or not present if no attributes are associated with the dataset.
+
+Stored as a **dataset template**, an openPMD dataset is represented by three JSON keys:
+
+* ``datatype`` (required): As above.
+* ``extent`` (required): A list of integers, describing the extent of the dataset.
+* ``attributes``: As above.
 
-**Attributes** are stored as a JSON object with a key for each attribute.
+This mode stores only the dataset metadata.
+Chunk load/store operations are ignored.
+
+Attributes
+..........
+
+In order to avoid name clashes, attributes are generally stored within a separate subgroup ``attributes``.
+
+Attributes can be stored in two formats.
+The format is selected by the :ref:`JSON/TOML parameter<backendconfig>` ``json.attribute.mode`` (resp. ``toml.attribute.mode``) with possible values ``["long", "short"]`` (default: ``"long"`` for JSON in openPMD 1.*, ``"short"`` otherwise, i.e. generally in openPMD 2.*, but always in TOML).
+
+Attributes in **long format** store the datatype explicitly, by representing attributes as JSON objects.
 Every such attribute is itself a JSON object with two keys:
 
  * ``datatype``: A string describing the type of the value.
  * ``value``: The actual value of type ``datatype``.
 
+Attributes in **short format** are stored as just the simple value corresponding with the attribute.
+Since JSON/TOML values are pretty-printed into a human-readable format, byte-level type details can be lost when reading those values again later on (e.g. the distinction between different integer types).
+
 TOML File Format
 ----------------
 

docs/source/details/backendconfig.rst

@@ -104,6 +104,8 @@ The key ``rank_table`` allows specifying the creation of a **rank table**, used
 Configuration Structure per Backend
 -----------------------------------
 
+Please refer to the respective backends' documentations for further information on their configuration.
+
 .. _backendconfig-adios2:
 
 ADIOS2
@@ -231,8 +233,21 @@ The parameters eligible for being passed to flush calls may be configured global
 
 .. _backendconfig-other:
 
-Other backends
-^^^^^^^^^^^^^^
+JSON/TOML
+^^^^^^^^^
 
-Do currently not read the configuration string.
-Please refer to the respective backends' documentations for further information on their configuration.
+A full configuration of the JSON backend:
+
+.. literalinclude:: json.json
+   :language: json
+
+The TOML backend is configured analogously, replacing the ``"json"`` key with ``"toml"``.
+
+All keys found under ``hdf5.dataset`` are applicable globally as well as per dataset.
+Explanation of the single keys:
+
+* ``json.dataset.mode`` / ``toml.dataset.mode``: One of ``"dataset"`` (default) or ``"template"``.
+  In "dataset" mode, the dataset will be written as an n-dimensional (recursive) array, padded with nulls (JSON) or zeroes (TOML) for missing values.
+  In "template" mode, only the dataset metadata (type, extent and attributes) are stored and no chunks can be written or read.
+* ``json.attribute.mode`` / ``toml.attribute.mode``: One of ``"long"`` (default in openPMD 1.*) or ``"short"`` (default in openPMD 2.*).
+  The long format explicitly encodes the attribute type in the dataset on disk, the short format only writes the actual attribute as a JSON/TOML value, requiring readers to recover the type.

docs/source/details/json.json

@@ -0,0 +1,10 @@
+{
+  "json": {
+    "dataset": {
+      "mode": "template"
+    },
+    "attribute": {
+      "mode": "short"
+    }
+  }
+}

examples/14_toml_template.cpp

@@ -0,0 +1,129 @@
+#include <openPMD/auxiliary/TemplateFile.hpp>
+#include <openPMD/openPMD.hpp>
+
+std::string backendEnding()
+{
+    auto extensions = openPMD::getFileExtensions();
+    if (auto it = std::find(extensions.begin(), extensions.end(), "toml");
+        it != extensions.end())
+    {
+        return *it;
+    }
+    else
+    {
+        // Fallback for buggy old NVidia compiler
+        return "json";
+    }
+}
+
+void write()
+{
+    std::string config = R"(
+{
+  "iteration_encoding": "variable_based",
+  "json": {
+    "dataset": {"mode": "template"},
+    "attribute": {"mode": "short"}
+  },
+  "toml": {
+    "dataset": {"mode": "template"},
+    "attribute": {"mode": "short"}
+  }
+}
+)";
+
+    openPMD::Series writeTemplate(
+        "../samples/tomlTemplate." + backendEnding(),
+        openPMD::Access::CREATE,
+        config);
+    auto iteration = writeTemplate.writeIterations()[0];
+
+    openPMD::Dataset ds{openPMD::Datatype::FLOAT, {5, 5}};
+
+    auto temperature =
+        iteration.meshes["temperature"][openPMD::RecordComponent::SCALAR];
+    temperature.resetDataset(ds);
+
+    auto E = iteration.meshes["E"];
+    E["x"].resetDataset(ds);
+    E["y"].resetDataset(ds);
+    /*
+     * Don't specify datatype and extent for this one to indicate that this
+     * information is not yet known.
+     */
+    E["z"].resetDataset({openPMD::Datatype::UNDEFINED});
+
+    ds.extent = {10};
+
+    auto electrons = iteration.particles["e"];
+    electrons["position"]["x"].resetDataset(ds);
+    electrons["position"]["y"].resetDataset(ds);
+    electrons["position"]["z"].resetDataset(ds);
+
+    electrons["positionOffset"]["x"].resetDataset(ds);
+    electrons["positionOffset"]["y"].resetDataset(ds);
+    electrons["positionOffset"]["z"].resetDataset(ds);
+    electrons["positionOffset"]["x"].makeConstant(3.14);
+    electrons["positionOffset"]["y"].makeConstant(3.14);
+    electrons["positionOffset"]["z"].makeConstant(3.14);
+
+    ds.dtype = openPMD::determineDatatype<uint64_t>();
+    electrons.particlePatches["numParticles"][openPMD::RecordComponent::SCALAR]
+        .resetDataset(ds);
+    electrons
+        .particlePatches["numParticlesOffset"][openPMD::RecordComponent::SCALAR]
+        .resetDataset(ds);
+    electrons.particlePatches["offset"]["x"].resetDataset(ds);
+    electrons.particlePatches["offset"]["y"].resetDataset(ds);
+    electrons.particlePatches["offset"]["z"].resetDataset(ds);
+    electrons.particlePatches["extent"]["x"].resetDataset(ds);
+    electrons.particlePatches["extent"]["y"].resetDataset(ds);
+    electrons.particlePatches["extent"]["z"].resetDataset(ds);
+}
+
+void read()
+{
+    /*
+     * The config is entirely optional, these things are also detected
+     * automatically when reading
+     */
+
+    // std::string config = R"(
+    // {
+    //   "iteration_encoding": "variable_based",
+    //   "toml": {
+    //     "dataset": {"mode": "template"},
+    //     "attribute": {"mode": "short"}
+    //   }
+    // }
+    // )";
+
+    openPMD::Series read(
+        "../samples/tomlTemplate." + backendEnding(),
+        openPMD::Access::READ_LINEAR);
+    read.readIterations(); // @todo change to read.parseBase()
+
+    std::string jsonConfig = R"(
+{
+  "iteration_encoding": "variable_based",
+  "json": {
+    "dataset": {"mode": "template"},
+    "attribute": {"mode": "short"}
+  }
+}
+)";
+    openPMD::Series cloned(
+        "../samples/jsonTemplate.json", openPMD::Access::CREATE, jsonConfig);
+    openPMD::auxiliary::initializeFromTemplate(cloned, read, 0);
+    // Have to define the dataset for E/z as it is not defined in the template
+    // @todo check that the dataset is defined only upon destruction, not at
+    // flushing already
+    cloned.writeIterations()[0].meshes["E"].at("z").resetDataset(
+        {openPMD::Datatype::INT});
+}
+
+int main()
+{
+    write();
+    read();
+}

include/openPMD/Dataset.hpp

@@ -44,7 +44,7 @@ class Dataset
         JOINED_DIMENSION = std::numeric_limits<std::uint64_t>::max()
     };
 
-    Dataset(Datatype, Extent, std::string options = "{}");
+    Dataset(Datatype, Extent = {1}, std::string options = "{}");
 
     /**
      * @brief Constructor that sets the datatype to undefined.

include/openPMD/Error.hpp

@@ -109,6 +109,12 @@ namespace error
     public:
         NoSuchAttribute(std::string attributeName);
     };
+
+    class IllegalInOpenPMDStandard : public Error
+    {
+    public:
+        IllegalInOpenPMDStandard(std::string what);
+    };
 } // namespace error
 
 /**

include/openPMD/IO/AbstractIOHandler.hpp

@@ -186,9 +186,11 @@ class AbstractIOHandler
 {
     friend class Series;
     friend class ADIOS2IOHandlerImpl;
+    friend class JSONIOHandlerImpl;
     friend class detail::ADIOS2File;
 
 private:
+    std::string m_openPMDVersion;
     IterationEncoding m_encoding = IterationEncoding::groupBased;
 
     void setIterationEncoding(IterationEncoding encoding)

include/openPMD/IO/JSON/JSONIOHandler.hpp

@@ -23,6 +23,7 @@
 
 #include "openPMD/IO/AbstractIOHandler.hpp"
 #include "openPMD/IO/JSON/JSONIOHandlerImpl.hpp"
+#include "openPMD/auxiliary/JSON_internal.hpp"
 
 #if openPMD_HAVE_MPI
 #include <mpi.h>

include/openPMD/IO/JSON/JSONIOHandlerImpl.hpp

@@ -180,6 +180,8 @@ class JSONIOHandlerImpl : public AbstractIOHandlerImpl
         std::string originalExtension);
 #endif
 
+    void init(openPMD::json::TracingJSON config);
+
     ~JSONIOHandlerImpl() override;
 
     void
@@ -265,8 +267,69 @@ class JSONIOHandlerImpl : public AbstractIOHandlerImpl
      */
     FileFormat m_fileFormat{};
 
+    std::string backendConfigKey() const;
+
+    /*
+     * First return value: The location of the JSON value (either "json" or
+     * "toml") Second return value: The value that was maybe found at this place
+     */
+    std::pair<std::string, std::optional<openPMD::json::TracingJSON>>
+    getBackendConfig(openPMD::json::TracingJSON &) const;
+
     std::string m_originalExtension;
 
+    enum class SpecificationVia
+    {
+        DefaultValue,
+        Manually
+    };
+
+    /////////////////////
+    // Dataset IO mode //
+    /////////////////////
+
+    enum class IOMode
+    {
+        Dataset,
+        Template
+    };
+
+    IOMode m_mode = IOMode::Dataset;
+    SpecificationVia m_IOModeSpecificationVia = SpecificationVia::DefaultValue;
+    bool m_printedSkippedWriteWarningAlready = false;
+
+    struct DatasetMode
+    {
+        IOMode m_IOMode;
+        SpecificationVia m_specificationVia;
+        bool m_skipWarnings;
+
+        template <typename A, typename B, typename C>
+        operator std::tuple<A, B, C>()
+        {
+            return std::tuple<A, B, C>{
+                m_IOMode, m_specificationVia, m_skipWarnings};
+        }
+    };
+    DatasetMode retrieveDatasetMode(openPMD::json::TracingJSON &config) const;
+
+    ///////////////////////
+    // Attribute IO mode //
+    ///////////////////////
+
+    enum class AttributeMode
+    {
+        Short,
+        Long
+    };
+
+    AttributeMode m_attributeMode = AttributeMode::Long;
+    SpecificationVia m_attributeModeSpecificationVia =
+        SpecificationVia::DefaultValue;
+
+    std::pair<AttributeMode, SpecificationVia>
+    retrieveAttributeMode(openPMD::json::TracingJSON &config) const;
+
     // HELPER FUNCTIONS
 
     // will use the IOHandler to retrieve the correct directory.
@@ -313,7 +376,7 @@ class JSONIOHandlerImpl : public AbstractIOHandlerImpl
     // essentially: m_i = \prod_{j=0}^{i-1} extent_j
     static Extent getMultiplicators(Extent const &extent);
 
-    static Extent getExtent(nlohmann::json &j);
+    static std::pair<Extent, IOMode> getExtent(nlohmann::json &j);
 
     // remove single '/' in the beginning and end of a string
     static std::string removeSlashes(std::string);
@@ -371,7 +434,7 @@ class JSONIOHandlerImpl : public AbstractIOHandlerImpl
 
     // check whether the json reference contains a valid dataset
     template <typename Param>
-    void verifyDataset(Param const &parameters, nlohmann::json &);
+    IOMode verifyDataset(Param const &parameters, nlohmann::json &);
 
     static nlohmann::json platformSpecifics();