pml/ob1: ensure RDMA fragments are released in the get -> send/recv f…

…allback

Under a number of circumstances it may be necessary to abandon an RDMA get in
ob1. In some cases it falls back to put but it may fall back to using send/recv.
If that happens then we may either crash or leak RDMA fragments because they
are still attached to the send request. Debug builds will crash due to a check
on rdma_frag when they are returned. This CL fixes the flaw by releasing any
rdma fragment when sceduling sends.

Signed-off-by: Nathan Hjelm <[email protected]>

ompi/mca/pml/ob1/pml_ob1_sendreq.c

@@ -22,6 +22,7 @@
  * Copyright (c) 2018-2019 Triad National Security, LLC. All rights
  *                         reserved.
  * Copyright (c) 2022      IBM Corporation.  All rights reserved.
+ * Copyright (c) 2024      Google, LLC. All rights reserved.
  * $COPYRIGHT$
  *
  * Additional copyrights may follow
@@ -1110,6 +1111,12 @@ mca_pml_ob1_send_request_schedule_once(mca_pml_ob1_send_request_t* sendreq)
 
     range = get_send_range(sendreq);
 
+    if (NULL != sendreq->rdma_frag) {
+        /* this request was first attempted with RDMA but is now using send/recv */
+        MCA_PML_OB1_RDMA_FRAG_RETURN(sendreq->rdma_frag);
+        sendreq->rdma_frag = NULL;
+    }
+
     while(range && (false == sendreq->req_throttle_sends ||
           sendreq->req_pipeline_depth < mca_pml_ob1.send_pipeline_depth)) {
         mca_pml_ob1_frag_hdr_t* hdr;