ULFM: Support for intercomm [i]agree and [i]shrink

[dalcinl]

This PR is a bit of cheating, as it implements ULFM for intercomm [i]agree and [i]shrink using the non-ft implementation of allreduce. There was already a precedent for such approach, and it is arguably better for the thing to work in the non-failure path rather than segfaulting bad as currently happening.

Refs. #12260 and mpi4py tests.

[dalcinl]

@bosilca @abouteiller I need your review here. I did my best to write self-contained commits to ease review. The topmost commit is a log of consistency fixes for error branches.

I have doubts about how ompi_comm_request_t should be used, in particular whether ompi_comm_request_return() should be called in case of errors. The FT code had a lot of these request return calls, but some where missing and I added them. HOWEVER, in the comm_cid code, ompi_comm_request_return() is generally not used after a requests have been started and put in the progress loop. The asymmetry is disturbing, but I'm afraid I don't know well enough how the OMPI progress engine works to figure out what's the right pattern to follow.

If things are done wrong, the comm request freelist may endup with duplicated entries, and that would explain why I was getting double-release errors in debug mode from OBJ_RELEASE.
Something weird is still going on when using MPI_Comm_shrink (only with the blocking variant) on an intercomm and OMPI_MCA_mpi_ft_enable=1. Sometimes I get a deadlock at the nextcid machinery. Was this the function giving us trouble with high-stress spawn?

[rhc54]

Sometimes I get a deadlock at the nextcid machinery. Was this the function giving us trouble with high-stress spawn?

Yep, that's the culprit.

[dalcinl]

mpi4py failure... this one is unrelated to this PR, and already reported in #12367

testAlltoallInter (test_util_pkl5.TestMPIWorld.testAlltoallInter) ... python: pml_ob1_comm.c:51: mca_pml_ob1_comm_proc_destruct: Assertion `NULL == proc->frags_cant_match' failed.

More disturbing, I re-ran the testsuite with re-enabled ULFM tests again, and now I get a failure. Not sure whether this is a consequence of applying @abouteiller suggestion or something else. I'll have to test this locally before proceeding, therefore I'm marking this PR as draft.

The failure I got is not reproducible, everything is now fine after rerunning again. I pulling this PR out of the draft state.

[dalcinl]

@abouteiller I've implemented your suggestion where you asked and ported it to another place. Could you please quickly re-review so that I can move this PR forward?

[abouteiller]

I got the following error running the ulfm-testing test harness, I'll need a couple more days to investigate this

 4  /home/bouteill/ompi/master.debug/lib/libmpi.so.0(mca_pml_ob1_revoke_comm+0x409) [0x7ff4e61d7d30]
 5  /home/bouteill/ompi/master.debug/lib/libmpi.so.0(+0x7df76) [0x7ff4e5eb8f76]
 6  /home/bouteill/ompi/master.debug/lib/libmpi.so.0(+0x7a60e) [0x7ff4e5eb560e]
 7  /home/bouteill/ompi/master.debug/lib/libopen-pal.so.0(+0xda43f) [0x7ff4e5baa43f]
 8  /lib64/libevent_core-2.1.so.7(+0x21b98) [0x7ff4e55a4b98]
 9  /lib64/libevent_core-2.1.so.7(event_base_loop+0x577) [0x7ff4e55a67b7]
10  /home/bouteill/ompi/master.debug/lib/libopen-pal.so.0(+0x263be) [0x7ff4e5af63be]
11  /home/bouteill/ompi/master.debug/lib/libopen-pal.so.0(opal_progress+0xa7) [0x7ff4e5af6477]
12  /home/bouteill/ompi/master.debug/lib/libmpi.so.0(+0x6f471) [0x7ff4e5eaa471]
13  /home/bouteill/ompi/master.debug/lib/libmpi.so.0(ompi_comm_nextcid+0x7a) [0x7ff4e5eabeb2]
14  /home/bouteill/ompi/master.debug/lib/libmpi.so.0(ompi_intercomm_create+0x3b6) [0x7ff4e5ea6de6]
15  /home/bouteill/ompi/master.debug/lib/libmpi.so.0(PMPI_Intercomm_create+0x13c) [0x7ff4e5f50428]

[dalcinl]

Oh boy... that looks like the intercomm create issue that have been popping up elsewhere.

[dalcinl]

@bosilca @jsquyres If a barrier-like synchronization helps fixing the intercomm issues, why don't just add the workaround and investigate it separately? Otherwise the problem will keep hunting everyone ad eternum. Or there is someone working actively on fixing this issue? Would a PR keeping testing working for others be accepted and merged quickly?

[abouteiller]

Oh boy... that looks like the intercomm create issue that have been popping up elsewhere.

Yes, that error is also present in main, not specific to this PR. I'll try to dig into it a bit to see what is the root cause.

The intercom creation error has also happened --with-ft=no and without fault injection?

[dalcinl]

The intercom creation error has also happened --with-ft=no and without fault injection?

When you ask about --with-ft=no, do you mean the configure option? In all of my CI testing, I never specify --with-ft=... to configure, so the default should be in place. Regarding the second part of your question, all my testing so far is WITHOUT fault injection. The main purpose of this PR is to enable testing of the ULFM APIs when no fault occurs.

[dalcinl]

@abouteiller I fixed up your commit to keel history clean, then I rebased to latest main. My CI testing stills fails[link], but IMHO these failures are unrelated to this PR.

@bosilca I'm not sure what's the status of the intercommunicator deadlock issues. I'm thinking of manually adding intercomm.Barrier() everywhere after creation in mpi4py testsuite, otherwise that issue will keep us from making progress with other things like this PR.

[dalcinl]

After adding a Barrier() right after creating the intercomm, now mpi4py tests all pass with the changes in this PR
https://github.com/mpi4py/mpi4py-testing/actions/runs/8466320394

[hppritcha]

@dalcinl you may want to rebase this PR on main now. I tested it (using modified test_ulfm.py) and it seems to work consistently now.

[hppritcha]

I also checked current main without the commits in this PR and test_ulfm.py does still fail. Passes for me with these commits added in.

[dalcinl]

@hppritcha Thanks. Indeed, after updating this PR, and after enabling all ULFM tests in mpi4py, everything is now OK. Full testsuite run here: https://github.com/mpi4py/mpi4py-testing/actions/runs/8734090801

@abouteiller @bosilca This PR is now ready. Can please you give it a final look and eventually merge?

[dalcinl]

@bosilca I've updated this PR as per your recommendations. Note however that I've added a few more ompi_comm_request_return(request) in places where I believe these cleanups were missing.

Please double check I didn't mess things up.

larger scale test failed, need further study

[bosilca]

Thanks @dalcinl for sticking with us on this.

[dalcinl]

@bosilca Unfortunately, I'll have to keep sticking on this...

The changes you objected about the calling order of OBJ_RELEASE(...) and ompi_comm_request_return(...) where actually all good and needed. The problem is that after two months of having done all that work, I had forgotten the gory details.

If you look at the implementation of ompi_comm_request_return() you will notice the line OBJ_RELEASE (request->context). Therefore, any use of the internal request context after ompi_comm_request_return() is a use-after-free bug. That was the reason I originally incorporated all these fixes in this PR. Most of the error paths are therefore still broken with the use-after-free bug.

[dalcinl]

I've opened #12480 as a follow up.

BTW, How do we get all these fixes into branch v5.0.x ?