ADIOS2 schema 2022_07_26, based on ADIOS2 modifiable attributes (#1310)

* Deactivate 2021 ADIOS2 schema
* Move ADIOS2FilePosition::GD to GroupOrDataset
* Fix changesOverSteps parameter
* Reimplement CurrentStep() for engines without support
* Main implementation: Third schema in ADIOS2
* Remove old schema from tests and use new schema in tests
* Ensure support for changing constant record components
* Update CI

Use apple-clang for testing the new layout

* Turn this into a feature of schema 0
* Documentation
* Mark all parent paths, too
* Add test for group_table option
* Update documentation docs/source/backends/adios2.rst
* Move ADIOS2 version macros to their own header

.github/workflows/linux.yml

@@ -80,7 +80,7 @@ jobs:
         find . -name *.bp.dir | xargs -n1 -P1 -I {} rm -rf {}
         ctest --test-dir build --output-on-failure
 
-  clang7_nopy_ompi_h5_ad2_newLayout:
+  clang7_nopy_ompi_h5_ad2:
     runs-on: ubuntu-20.04
     if: github.event.pull_request.draft == false
     steps:
@@ -94,7 +94,7 @@ jobs:
         sudo apt-get install clang-7 gfortran libopenmpi-dev python3
         sudo .github/workflows/dependencies/install_spack
     - name: Build
-      env: {CC: clang-7, CXX: clang++-7, CXXFLAGS: -Werror, OPENPMD2_ADIOS2_SCHEMA: 20210209}
+      env: {CC: clang-7, CXX: clang++-7, CXXFLAGS: -Werror}
       run: |
         eval $(spack env activate --sh .github/ci/spack-envs/clang7_nopy_ompi_h5_ad2/)
         spack install
@@ -148,7 +148,7 @@ jobs:
         ctest --test-dir build --output-on-failure
 
   # ADIOS2 v2.7.1
-  clang8_py38_mpich_h5_ad2_newLayout:
+  clang8_py38_mpich_h5_ad2:
     runs-on: ubuntu-20.04
     if: github.event.pull_request.draft == false
     steps:
@@ -162,7 +162,7 @@ jobs:
         sudo apt-get install clang-8 gfortran libmpich-dev python3
         sudo .github/workflows/dependencies/install_spack
     - name: Build
-      env: {CC: clang-8, CXX: clang++-8, CXXFLAGS: -Werror, OPENPMD2_ADIOS2_SCHEMA: 20210209}
+      env: {CC: clang-8, CXX: clang++-8, CXXFLAGS: -Werror}
       run: |
         cmake --version
         mpiexec --version

.github/workflows/macos.yml

@@ -70,6 +70,9 @@ jobs:
           -DPython_EXECUTABLE=$(which python3)
         cmake --build build --parallel 3
         ctest --test-dir build --verbose
+        rm -rf build/samples
+        share/openPMD/download_samples.sh build
+        OPENPMD2_ADIOS2_USE_GROUP_TABLE=1 ctest --test-dir build --verbose
 
 # TODO: apple_conda_ompi_all (similar to conda_ompi_all on Linux)
 #   both OpenMPI and MPICH cause startup (MPI_Init) issues on GitHub Actions

CMakeLists.txt

@@ -478,7 +478,6 @@ set(IO_SOURCE
         src/IO/JSON/JSONFilePosition.cpp
         src/IO/ADIOS/ADIOS2IOHandler.cpp
         src/IO/ADIOS/ADIOS2Auxiliary.cpp
-        src/IO/ADIOS/ADIOS2PreloadAttributes.cpp
         src/IO/InvalidatableFile.cpp)
 
 # library

docs/source/backends/adios2.rst

@@ -82,7 +82,7 @@ environment variable                  default    description
 ``OPENPMD_ADIOS2_HAVE_METADATA_FILE`` ``1``      Online creation of the adios journal file (``1``: yes, ``0``: no).
 ``OPENPMD_ADIOS2_NUM_SUBSTREAMS``     ``0``      Number of files to be created, 0 indicates maximum number possible.
 ``OPENPMD_ADIOS2_ENGINE``             ``File``   `ADIOS2 engine <https://adios2.readthedocs.io/en/latest/engines/engines.html>`_
-``OPENPMD2_ADIOS2_SCHEMA``            ``0``      ADIOS2 schema version (see below)
+``OPENPMD2_ADIOS2_USE_GROUP_TABLE``   ``0``      Use group table (see below)
 ``OPENPMD_ADIOS2_STATS_LEVEL``        ``0``      whether to generate statistics for variables in ADIOS2. (``1``: yes, ``0``: no).
 ``OPENPMD_ADIOS2_ASYNC_WRITE``        ``0``      ADIOS2 BP5 engine: 1 means setting "AsyncWrite" in ADIOS2 to "on". Flushes will go to the buffer by default (see ``preferred_flush_target``).
 ``OPENPMD_ADIOS2_BP5_BufferChunkMB``  ``0``      ADIOS2 BP5 engine: applies when using either EveryoneWrites or EveryoneWritesSerial aggregation
@@ -135,14 +135,32 @@ The preferred backend usually depends on the system's native software stack.
 For fine-tuning at extreme scale or for exotic systems, please refer to the ADIOS2 manual and talk to your filesystem admins and the ADIOS2 authors.
 Be aware that extreme-scale I/O is a research topic after all.
 
-Experimental new ADIOS2 schema
-------------------------------
+Experimental group table feature
+--------------------------------
 
-The experimental new ADIOS2 schema is deprecated and **will be removed soon**. It used to be activated via the JSON parameter ``adios2.schema = 20210209`` or via the environment variable ``export OPENPMD2_ADIOS2_SCHEMA=20210209``.
+We are experimenting with a feature that will make the structure of an ADIOS2 file more explicit.
+Currently, the hierarchical structure of an openPMD dataset in ADIOS2 is recovered implicitly by inspecting variables and attributes found in the ADIOS2 file.
+Inspecting attributes is necessary since not every openPMD group necessarily contains an (array) dataset.
+The downside of this approach is that ADIOS2 attributes do not properly interact with ADIOS2 steps, resulting in many problems and workarounds when parsing an ADIOS2 dataset.
+An attribute, once defined, cannot be deleted, implying that the ADIOS2 backend will recover groups that might not actually be logically present in the current step.
 
-**Do no longer use these options**, the created datasets will no longer be read by the openPMD-api.
+As a result of this behavior, support for ADIOS2 steps is currently restricted.
 
-An alternative data layout with less intrusive changes and similar features is `currently in development <https://github.com/openPMD/openPMD-api/pull/1310>`__.
+For full support of ADIOS2 steps, we introduce a group table that makes use of modifiable attributes in ADIOS2 v2.9, i.e. attributes that can have different values across steps.
+
+An openPMD group ``<group>`` is present if:
+
+1. The integer attribute ``__openPMD_groups/<group>`` exists
+2. and:
+
+  a. the file is either accessed in random-access mode
+  b. or the current value of said attribute is equivalent to the current step index.
+
+This feature can be activated via the JSON/TOML key ``adios2.use_group_table = true`` or via the environment variable ``OPENPMD2_ADIOS2_USE_GROUP_TABLE=1``.
+It is fully backward-compatible with the old layout of openPMD in ADIOS2 and mostly forward-compatible (except the support for steps).
+
+The variable-based encoding of openPMD automatically activates the group table feature.
+The group table feature automatically activates the use of ADIOS2 steps (which until version 0.15 was an opt-in feature via ``adios2.engine.usesteps = true``).
 
 Memory usage
 ------------

include/openPMD/IO/ADIOS/ADIOS2Auxiliary.hpp

@@ -21,7 +21,10 @@
 
 #pragma once
 
+#include "openPMD/Error.hpp"
+#include "openPMD/IO/ADIOS/macros.hpp"
 #include "openPMD/config.hpp"
+
 #if openPMD_HAVE_ADIOS2
 #include "openPMD/Dataset.hpp"
 #include "openPMD/Datatype.hpp"
@@ -33,9 +36,17 @@
 #include <stdexcept>
 #include <utility>
 #include <vector>
+#endif
 
 namespace openPMD
 {
+enum class GroupOrDataset
+{
+    GROUP,
+    DATASET
+};
+
+#if openPMD_HAVE_ADIOS2
 namespace detail
 {
     // ADIOS2 does not natively support boolean values
@@ -119,6 +130,45 @@ namespace detail
         std::string const &attributeName,
         bool verbose,
         VariableOrAttribute voa = VariableOrAttribute::Attribute);
+
+    inline bool readOnly(adios2::Mode mode)
+    {
+        switch (mode)
+        {
+        case adios2::Mode::Append:
+        case adios2::Mode::Write:
+            return false;
+        case adios2::Mode::Read:
+#if openPMD_HAS_ADIOS_2_8
+        case adios2::Mode::ReadRandomAccess:
+#endif
+            return true;
+        case adios2::Mode::Undefined:
+        case adios2::Mode::Sync:
+        case adios2::Mode::Deferred:
+            break;
+        }
+        throw error::Internal("Control flow error: No ADIOS2 open mode.");
+    }
+    inline bool writeOnly(adios2::Mode mode)
+    {
+        switch (mode)
+        {
+        case adios2::Mode::Append:
+        case adios2::Mode::Write:
+            return true;
+        case adios2::Mode::Read:
+#if openPMD_HAS_ADIOS_2_8
+        case adios2::Mode::ReadRandomAccess:
+#endif
+            return false;
+        case adios2::Mode::Undefined:
+        case adios2::Mode::Sync:
+        case adios2::Mode::Deferred:
+            break;
+        }
+        throw error::Internal("Control flow error: No ADIOS2 open mode.");
+    }
 } // namespace detail
 
 /**
@@ -277,6 +327,5 @@ auto switchAdios2VariableType(Datatype dt, Args &&...args)
             std::to_string(static_cast<int>(dt)));
     }
 }
-} // namespace openPMD
-
 #endif // openPMD_HAVE_ADIOS2
+} // namespace openPMD

include/openPMD/IO/ADIOS/ADIOS2FilePosition.hpp

@@ -20,6 +20,7 @@
  */
 #pragma once
 
+#include "openPMD/IO/ADIOS/ADIOS2Auxiliary.hpp"
 #include "openPMD/IO/AbstractFilePosition.hpp"
 #include <string>
 #include <utility>
@@ -28,27 +29,21 @@ namespace openPMD
 {
 struct ADIOS2FilePosition : public AbstractFilePosition
 {
-    enum class GD
-    {
-        GROUP,
-        DATASET
-    };
-
-    ADIOS2FilePosition(std::string s, GD groupOrDataset)
+    ADIOS2FilePosition(std::string s, GroupOrDataset groupOrDataset)
         : location{std::move(s)}, gd{groupOrDataset}
     {}
 
-    explicit ADIOS2FilePosition(GD groupOrDataset)
+    explicit ADIOS2FilePosition(GroupOrDataset groupOrDataset)
         : ADIOS2FilePosition{"/", groupOrDataset}
     {}
 
-    ADIOS2FilePosition() : ADIOS2FilePosition{GD::GROUP}
+    ADIOS2FilePosition() : ADIOS2FilePosition{GroupOrDataset::GROUP}
     {}
 
     /**
      * Convention: Starts with slash '/', ends without.
      */
     std::string location;
-    GD gd;
+    GroupOrDataset gd;
 }; // ADIOS2FilePosition
 } // namespace openPMD