Performance: Flush IOHandler only once, not for each Iteration

[franzpoeschel]

This might address a performance regression seen by @guj.

Until now, Series::flushFileBased() and Series::flushGorVBased() flushed the IOHandler for every iteration. Since flushing has a constant overhead, calling Series::flush() has a linear complexity along with the number of Iterations, even if only a single Iteration has modifications.

Fixing this was not entirely trivial, since in file-based encoding, some frontend object must unset the written flag, since they must be written anew for each file. Before this PR, this could easily be done synchronously in the frontend, but since all Iterations are now flushed at once, this must now be done asynchronously in the backend as a backend task.

Also speed up the common case of parsing flush options: No options at all.

• Check if this addresses the performance regressions seen by @guj
• Check this against PIConGPU, the new written logic might behave weird in more complex setups

[franzpoeschel]

I just tested this against @guj's data on Perlmutter and it clearly improves the performance.

[guj]

I just tested this against @guj's data on Perlmutter and it clearly improves the performance.

Thanks Franz. I just checked out this branch, and on Mac it did not show any difference however I do see on Perlmutter it saved about 10% (20 seconds) . On my Mac times observed are pretty much the same as before.

[franzpoeschel]

No difference at all? I observed a clear system-independent performance regression for issue380_f /pscratch/sd/j/junmin/perlmutter/Jan2024/Test/issue380/bp5-f/defaultBP5-rank-ews_diags_f/diag1/openpmd_ bp f: The IOHandler is flushed for each Iteration even if no IOtasks are enqueued. The PR fuses all flushes into a single one, thus getting rid of this overhead.

You can further improve performance by using deferred iteration parsing:

Series series = Series(fileName, readMode, "defer_iteration_parsing = true");
// ...
    auto ex = series.iterations[i].open().meshes["E"]["x"];

And even further by closing the Iterations after using them:

   series.iterations[i].close();

Even without these improvements, I see a performance improvement far above just 20 seconds for going through 4000 Iterations.

[guj]

Ah, I was running issue380.cpp

Yes, issue380_f has significant improvement after your fix!!

No difference at all? I observed a clear system-independent performance regression for issue380_f /pscratch/sd/j/junmin/perlmutter/Jan2024/Test/issue380/bp5-f/defaultBP5-rank-ews_diags_f/diag1/openpmd_ bp f: The IOHandler is flushed for each Iteration even if no IOtasks are enqueued. The PR fuses all flushes into a single one, thus getting rid of this overhead.

You can further improve performance by using deferred iteration parsing:

Series series = Series(fileName, readMode, "defer_iteration_parsing = true");
// ...
    auto ex = series.iterations[i].open().meshes["E"]["x"];

And even further by closing the Iterations after using them:

   series.iterations[i].close();

Even without these improvements, I see a performance improvement far above just 20 seconds for going through 4000 Iterations.

[franzpoeschel]

Great to hear!

Ah, I was running issue380.cpp

Does that one have noticeable performance issues that we need to look into?

[guj]

Great to hear!

Ah, I was running issue380.cpp

Does that one have noticeable performance issues that we need to look into?
This fix basically made the time gap between issue380.cpp and issue380_f.cpp disappear. So would be great to merge it.

[franzpoeschel]

Note: This change makes the test stricter, since Series::flush() unlike the destructor will not swallow exceptions.