support for loading into xarray?

[BenWibking]

Is there support for loading openPMD (with ADIOS2/BP backend) into xarray arrays?

The machinery that is needed is to allow lazy loading of chunks that can then be processed with, e.g., xarray's Dask backend: https://docs.xarray.dev/en/latest/internals/how-to-add-new-backend.html#how-to-support-lazy-loading

[pordyna]

If this is merged pydata/xarray#4659, then you could go around through dask, or just use dask directly. I think it has some xarray support.

[BenWibking]

Is there already support for lazy loading into Dask? From this issue (#1500), it seems like that is also a work-in-progress. Is that the case?

[pordyna]

I think the issue you mentioned is about processing metadata only by on one worker? If I understand lazy loading correctly, then it should be already implemented. With dask the reading from disk is delayed until the actual compute and only the actually used chunks are being read. The chunk distribution for reading is not automatically optimized and is what availableChunks returns. At least for meshes, the chunking can be overridden, see #1477.

[ax3l]

Loading via DASK as suggested in @pordyna's answer is a great idea. We support both DASK ndarrays and data frames in openPMD-api: https://openpmd-api.readthedocs.io/en/latest/analysis/dask.html

Alternatively, you could go for smaller data sets via numpy (our default loadChunk/load_chunk) and our lazy pandas support for particles:

• https://openpmd-api.readthedocs.io/en/latest/analysis/pandas.html
• https://docs.xarray.dev/en/latest/user-guide/pandas.html

Please also do not hesitate to add your working recipe to our docs in the same sections. We follow a batteries-included approach and import/export to xarray is definitely desired :)

Currently, we list: Numpy (default), ParaView, Pandas, DASK, RAPIDS (cupy & cudf) and further contributed integrations. We also use openPMD-api with PyTorch (e.g., WarpX openPMD output: https://warpx.readthedocs.io/en/latest/usage/workflows/ml_dataset_training.html), should document that here as well...