use span to allocate mesh data by default #6123
Conversation
this makes it possible to aggregate data for later flush
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
guj
changed the title
ax3l
added
Performance optimization
component: openPMD
| * if isBTD=true, advice to use ADIOS Put() instead of PDW for better performance. | ||
| * in ADIOS, the action will be PerformDataWrite | ||
| * if isBTD=true, in ADIOS, the action will be PerformPut | ||
| * because no action is taken for the span tasks. |
There was a problem hiding this comment.
Is this generally true? You only add spans for fields right now, what if particles are involved?
There was a problem hiding this comment.
You are right BTD particles will be calling PerformPut. Right now particles is not the I/O performance bottleneck for BTD. It will happen when you have enormous amount of particles. I will definitely revisit this issue with another pull request.
| }); | ||
| auto span = dynamicMemoryView.currentBuffer(); | ||
| amrex::Gpu::dtoh_memcpy_async(span.data(), fab.dataPtr(icomp), local_box.numPts()*sizeof(amrex::Real)); | ||
| } else |
There was a problem hiding this comment.
Do we need this else branch for D2H? I thought openPMD-api has a fallback implementation for HDF5...?
There was a problem hiding this comment.
Turns out no. We do need it.
| auto span = dynamicMemoryView.currentBuffer(); | ||
| std::memcpy(span.data(), fab.dataPtr(icomp), local_box.numPts()*sizeof(amrex::Real)); | ||
| } | ||
| else |
There was a problem hiding this comment.
Same as above: I thought openPMD-api has a fallback implementation for Span for backends that do not support it?
There was a problem hiding this comment.
I added this because the first round of CPU span failed pull request (it uses HDF5) so the fallback does not work with collecting a few buffers (during a snapshot) and then flush out. The fallback works if flushes out immediately.
So we can aggregate (mainly for BTD) data for a later flush.
introduced an optional but related entry in the input file:
<diag>.buffer_flush_limit_btdby default it is 5.New implementation of #3200 but so far only for fields/meshes. Particles will be a follow-up PR.