Opt-in support for unknown ADIOS2 engines

[franzpoeschel]

The ADIOS2 backend has a set of known and supported ADIOS2 engines. The consequence is that new, unknown or unsupported engines cannot be used without changing the code of the ADIOS2 backend.
This PR adds the backend key adios2.engine.treat_unsupported_engine_like in order to allow users to try alternative backends at their own risk.

e.g.:

[adios2.engine]
type = "dataman"
treat_unsupported_engine_like = "sst"

TODO:

• Documentation

Notes to myself for documenting advanced features not supported by this:
QueueLimit
filename extension handling: default extensions, mandatory extensions in BP3 and SST
Span API
PerformDataWrite

[guj]

Is this PR intending to allow new additional adios engine types (other than the ones defined in the list)?

It seems to me that the function fileSuffix() still throws error when engine type (from either definitions) is not in the defined list.

[franzpoeschel]

Is this PR intending to allow new additional adios engine types (other than the ones defined in the list)?

Yes. The idea is that new engines which we don't support (like the currently experimental Campaign engine, or other engines like Dataman) can be experimentally used by telling the ADIOS2 backend to treat the engine like one of the supported engines.
When ADIOS2 v2.11 comes out and brings something new, this means that we ideally don't need to publish a new release for supporting it.

It seems to me that the function fileSuffix() still throws error when engine type (from either definitions) is not in the defined list.

The logic of fileSuffix() is skipped when an unsupported engine is in use:

std::string ADIOS2IOHandlerImpl::fileSuffix(bool verbose) const
{
    // ... some defines ....

    if (m_realEngineType.has_value())
    {
        // unknown engine type, use whatever ending the user specified
        return m_userSpecifiedExtension;
    }

    // ....

So this should not be a problem

[guj]

Thanks Franz. As the engine type is also able to be defined as env variable "OPENPMD_ADIOS_ENGINE", should there be an equivalent env for "treat_unsupported_engine_like" as well? I understand the json config takes priority over env variables.

[guj]

I guess a boolean flag would also do? "e.g. accept_unspported_engine=true"?

[guj]

If true, can always return ".bp" at realEngineType(). The actual value of the treat_supported_engine expected to be one of the known adios engines, and exactly which one is not relavent I think.

[franzpoeschel]

I guess a boolean flag would also do? "e.g. accept_unspported_engine=true"?

That will not be enough because there is no "standard" way how an engine should be treated. Especially, most engines differ slightly in how steps should be treated (staging engines require steps, BP5 requires steps with normal Read mode, but cannot deal with steps in ReadRandomAccess, the Campaign engine fails when using steps in any way, other engines are mostly a mixture).

I don't think that providing one generic implementation for an unknown future engine will work, it seems better to use the existing implementations and let users try which works best.

If true, can always return ".bp" at realEngineType(). The actual value of the treat_supported_engine expected to be one of the known adios engines, and exactly which one is not relavent I think.

realEngineType() will return the Engine type that is actually passed on to ADIOS2. m_engineType determines the behavior of the ADIOS2 backend. Especially BP5 behaves slightly different from the other BP engines.
I don't see a reason to put extra work into restricting the options here.

[guj]

Sure. Thanks

[franzpoeschel]

Thanks Franz. As the engine type is also able to be defined as env variable "OPENPMD_ADIOS_ENGINE", should there be an equivalent env for "treat_unsupported_engine_like" as well? I understand the json config takes priority over env variables.

I can add that, yeah

[ax3l]

Awesome, LGTM :)