Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Roadmap: Wishlist of new features #1332

Open
3 of 12 tasks
franzpoeschel opened this issue Nov 15, 2022 · 0 comments
Open
3 of 12 tasks

Roadmap: Wishlist of new features #1332

franzpoeschel opened this issue Nov 15, 2022 · 0 comments

Comments

@franzpoeschel
Copy link
Contributor

franzpoeschel commented Nov 15, 2022

A wishlist of features after the upcoming releases:

Near-Term

Mid-Term

  • Support for joined arrays in backends other than ADIOS2
  • async for iteration in s.read_iterations() for Python
  • More flexible reads in C++17 A Variant-Based LoadChunk #1372 and Python
  • MPI-wise logging of IO actions
  • Performance optimization: Long-running simulations (many iterations, reading and writing)
  • Specify default attributes not upon construction, but upon closing, clean up the logic for specifying defaults, constructors and destructors of our object model
    • Context: crashing simulations
    • If in read, a standard attribute is missing, then warn and add a reasonable default (e.g., a 3-value axisLabel for a 3D-mesh)
  • Support for PIConGPU-style dataset-specific JSON/TOML configuration
    https://picongpu.readthedocs.io/en/0.6.0/usage/plugins/openPMD.html#cfg-file, also for iteration-specific configuration, e.g. for InitialBufferSize per file
  • Maybe Flag for writing attributes only from rank 0
  • Python docstrings docstrings in python version / Python documentation #1328
  • Maybe SoA <-> AoS flexibility (affects the standard)
    • Probably requires struct-type fields
  • Project structure: Separate MPI headers from serial headers
    With this change: Provide openPMD-api via Linux package managers
  • Compression and plugins in HDF5
  • HDF5 hardlinks + maybe as a fallback softlinks Generalize Record Definition openPMD-standard#283

Long-Term / Ideas

  • Synchronous mode: Avoid UB for store and load chunks

    • In both C++ and Python, we would like to avoid that the user can interact with allocated but non-flushed (UB) data.
    • For this, we could rename storeChunk / loadChunk to ...Async(), which returns a std::future (C++) or asyncio.Future (Python).
      • We need to keep track which of these in-flight objects we created and will set them to valid on series.flush().
      • If a future is awaited before flush was called, we throw a runtime exception, which allows to recover in interactive use. Futures also allow us to check if the futures are valid w/o having to catch exceptions.
    • The existing APIs would be sync.
  • Maybe Use ADIOS2 group feature in reading (-> faster parsing)

  • Maybe Chunk distribution algorithms

  • Maybe Async I/O (especially Python)

@ax3l ax3l added this to the 0.16.0 milestone Dec 8, 2022
@ax3l ax3l pinned this issue Dec 8, 2022
@ax3l ax3l changed the title After 0.15: Wishlist of new features Roadmap: Wishlist of new features Aug 8, 2023
@ax3l ax3l modified the milestones: 0.16.0, 0.17.0 Jun 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants