Generate bigcount interfaces for Fortran and C #12226

jtronge · 2024-01-10T21:16:33Z

This is a revision of PR #12033, removing all ABI-specific changes while leaving in support for bigcount functions and also adding bigcount support for mpi_f08. This is currently a draft since not all functions are currently implemented.

jsquyres

I haven't finished reading / understanding / reviewing both generate_bindings.py scripts yet, but since I'm so late for this and the clock is ticking on @jtronge's availability, I'm going to submit what I have so far.

jsquyres · 2024-01-30T20:12:49Z

ompi/mpi/fortran/use-mpi-f08/Makefile.am


 lib@OMPI_LIBMPI_NAME@_usempif08_la_SOURCES = \
        $(mpi_api_files) \
        mpi-f08.F90

 # These are generated; do not ship them
-nodist_lib@OMPI_LIBMPI_NAME@_usempif08_la_SOURCES =
+# nodist_lib@OMPI_LIBMPI_NAME@_usempif08_la_SOURCES =


Go ahead and delete this line; don't just comment it out.

jsquyres · 2024-01-30T20:13:41Z

ompi/mpi/fortran/use-mpi-f08/Makefile.am


 # JMS Somehow this variable substitution isn't quite working, and I
 # don't have time to figure it out.  So just wholesale copy the file
 # list.  :-(
-#pmpi_api_files = $(mpi_api_files:%=profile/p%)
+#pmpi_api_files = $(mpi_api_files:%=p%)


If you can figure out why this substitution isn't working, go ahead and uncomment it. Otherwise, it would probably be ok to delete this whole commented-out block.

jsquyres · 2024-02-07T14:47:31Z

ompi/include/mpi.h.in

@@ -1389,7 +1389,7 @@ OMPI_DECLSPEC extern struct ompi_predefined_datatype_t ompi_mpi_ub;
 /*
 * MPI API
 */
-
+#ifndef OMPI_NO_MPI_PROTOTYPES


This is the only file in the whole tree where I see this macro used -- where does it come from?

If we're using it in our infrastructure, we prefer to always define boolean-like macros to be 0 or 1 (vs. defined or undefined). There's less of a chance for error that way.

This is left over from the standard ABI code; I can remove it for now.

IIRC the reason I didn't use a 0/1 macro there is because it would require adding the macro in mpicc or forcing the end user to add it. Both the standard ABI and the ompi ABI code were using mpi.h since some types defined there needed to be used in both versions, but the prototypes cannot be the same.

When we start working again on the standard ABI code, it might be better to refactor mpi.h out into multiple files to make this easier.

Ok, gotcha -- those are good reasons. If you end up keeping something like this, having a comment to explain the deviation from our norm (i.e., using #ifndef) would be good.

jsquyres · 2024-02-07T14:48:21Z

ompi/mpi/c/Makefile.am

@@ -41,6 +41,36 @@ endif

 headers = bindings.h

+prototype_sources = \


Probably worth putting a comment before this line explaining what this macro is for -- it may not be intuitive to the casual code reader that these are all the files with bigcount APIs.

jsquyres · 2024-02-07T14:49:26Z

ompi/mpi/c/Makefile.am

@@ -60,7 +90,7 @@ endif
 # List of all C files that have profile versions
 #


Might also be good to include in this comment before the interface_profile_sources that the ompi_FOO.c files have the ompi_ prefix because they were generated.

jsquyres · 2024-02-07T15:46:48Z

ompi/mpi/c/generate_bindings.py

+
+def main():
+    if len(sys.argv) < 2:
+        # Fix required for Python 3.6


What does this comment mean -- does this mean that this script does not work if Python is < v3.6?

This a fix for Python v3.6, since the argparse package has some limitations for that version. I have not tested this code below v3.6, unless the CI is using older versions.

jsquyres · 2024-02-07T15:52:09Z

ompi/mpi/c/generate_bindings.py

+        sys.exit(1)
+
+    parser = argparse.ArgumentParser(description='generate ABI header file and conversion code')
+    subparsers = parser.add_subparsers()


I'm a bit confused by this CLI interface. Running generate_bindings.py --help shows this:

$ ./ompi/mpi/c/generate_bindings.py --help usage: generate_bindings.py [-h] {header,source} ... generate ABI header file and conversion code positional arguments: {header,source} options: -h, --help show this help message and exit

Which doesn't give me a lot of information about how to use this script. In ompi/mpi/c/Makefile.am, I see it used like this:

$(PYTHON) $(srcdir)/generate_bindings.py source ompi $< > $@

I think that means it's falling down into the "source" sub-parser.

What's the header sub-parser for? Is that for future functionality or something?

Would it be possible to use more traditional --foo=bar kinds of CLI arguments, perchance?

I can try to make --help a little more descriptive about the sub-parsers. The source sub-parser is designed to generate the source code from a template, in this case for the ompi ABI. The header sub-parser is designed to auto-generate the prototype definitions, although it was only used previously for the standard ABI code (it could be useful for the ompi ABI as well).

The standard ABI functionality was completely removed from the Makefiles, but I've left the functionality in the script, that's why some of these arguments are still there but unused.

jsquyres · 2024-02-07T15:52:33Z

ompi/mpi/c/generate_bindings.py

+        print('ERROR: missing subparser argument (see --help)')
+        sys.exit(1)
+
+    parser = argparse.ArgumentParser(description='generate ABI header file and conversion code')


Is this generating ABI files, or BigCount stuff? I thought this PR was about BigCount and ABI was future...?

...after reading a bunch more of generate_bindings.py, I see that it looks like a lot of the ABI functionality is included, but it doesn't look like it's complete...? E.g., I don't see any other kind of ABI infrastructure (e.g., creating the other libraries, stacking the libraries and implementations like we talked about, etc.).

If we're too late in the game to fully dis-entangle ABI and bigcount into 2 wholly separate commits (with one almost certainly building upon the other), it would be good to explain that this commit has elements of stuff that will be used in an upcoming ABI commit and it wasn't separated because of lack of time blah blah blah. Otherwise, the reader (e.g., me) is left wondering why a "Bigcount" commit has a bunch of unused / half-done ABI stuff included.

Sorry I should have been more clear with my commit message. I removed all ABI-specific changes from the make-system, but the script still includes the unused ABI code. This seemed the best route to take, instead of completely removing the extra ABI code, which is not affecting how ompi is being built right now. I can remove it though.

jsquyres · 2024-02-07T15:54:24Z

ompi/mpi/c/generate_bindings.py

+    args = parser.parse_args()
+
+    # Always add the header
+    print('/* THIS FILE WAS AUTOGENERATED BY ompi/mpi/c/abi.py. DO NOT EDIT BY HAND. */')


Additionally, it might be better to have a traditional --outfile=FILENAME kind of CLI arg that indicates the name of the file that you want to write to. This leaves stdout available for info/verbose/debug kinds of output.

jsquyres · 2024-02-07T16:11:53Z

ompi/mpi/c/generate_bindings.py

+* Functions requiring a bigcount implementation should have type COUNT in
+  place of MPI_Count or int for each count parameter. Bigcount functions will
+  be generated automatically for any function that includes a COUNT type.
+"""


Please add a larger comment here at the top explaining the purpose of this script, and at least a high-level overview of what it does / how it works. This is a very large script; having an intro to it at the top would be most helpful to the reader (e.g., me).

jsquyres · 2024-02-07T16:15:30Z

Should #12033 be closed (if this PR wholly replaces it)?

jtronge · 2024-02-07T17:14:40Z

I think I'll try to remove the extra standard ABI code from the script, as well as the additional items you caught @jsquyres.

I don't think this code completely replaces PR #12033, especially since that one includes the refactor of libmpi into multiple libraries that we discussed.

hppritcha · 2024-02-08T21:23:34Z

config/ompi_setup_mpi_fortran.m4

+    AS_IF([test $OMPI_TRY_FORTRAN_BINDINGS -ge $OMPI_FORTRAN_USEMPIF08_BINDINGS],
+          [OMPI_FORTRAN_CHECK_TS([OMPI_FORTRAN_HAVE_TS=1])])
+
+    AC_SUBST(OMPI_MPI_SUBARRAYS_SUPPORTED)


I think lines 458 and 459 should b moved below line 472 since it is there that we may be setting OMPI_MPI_SUBARRAYS_SUPPORTED and OMPI_MPI_ASYNC_PROTECTS_NONBLOCKING to true.

hppritcha · 2024-02-12T17:29:45Z

docs/developers/bindings.rst

+The Fortran file ``api_f08_generated.F90`` contains all the internal subroutine
+definitions, each of which makes a call into corresponding C functions. Two
+different C files are generated: ``api_f08_ts_generated.c`` contains support
+for compilers with TS 29113 support, allowing the use of ``CFI_cdesc_t`` types;


might be nice to add a hyperlink here, e.g. https://fortranwiki.org/fortran/show/Fortran+2018

jtronge · 2024-02-14T22:44:16Z

/azp run

github-actions · 2024-04-01T21:15:56Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-01T21:36:33Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-01T21:37:14Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-03T15:42:53Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-16T21:18:09Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

e673867: Add int, char, and MPI_Info-related types

check_signed_off: does not contain a valid Signed-off-by line

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-17T16:35:50Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

fc151a8: Add DATATYPE, STATUS, and GREQUEST_ types*

check_signed_off: does not contain a valid Signed-off-by line

e673867: Add int, char, and MPI_Info-related types

check_signed_off: does not contain a valid Signed-off-by line

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-17T17:56:04Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

5127029: Add ELEMENT_COUNT for MPI_Get_elements_x

check_signed_off: does not contain a valid Signed-off-by line

fc151a8: Add DATATYPE, STATUS, and GREQUEST_ types*

check_signed_off: does not contain a valid Signed-off-by line

e673867: Add int, char, and MPI_Info-related types

check_signed_off: does not contain a valid Signed-off-by line

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-18T16:37:38Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

5127029: Add ELEMENT_COUNT for MPI_Get_elements_x

check_signed_off: does not contain a valid Signed-off-by line

fc151a8: Add DATATYPE, STATUS, and GREQUEST_ types*

check_signed_off: does not contain a valid Signed-off-by line

e673867: Add int, char, and MPI_Info-related types

check_signed_off: does not contain a valid Signed-off-by line

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-22T16:44:26Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

5127029: Add ELEMENT_COUNT for MPI_Get_elements_x

check_signed_off: does not contain a valid Signed-off-by line

fc151a8: Add DATATYPE, STATUS, and GREQUEST_ types*

check_signed_off: does not contain a valid Signed-off-by line

e673867: Add int, char, and MPI_Info-related types

check_signed_off: does not contain a valid Signed-off-by line

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-22T16:44:41Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

5127029: Add ELEMENT_COUNT for MPI_Get_elements_x

check_signed_off: does not contain a valid Signed-off-by line

fc151a8: Add DATATYPE, STATUS, and GREQUEST_ types*

check_signed_off: does not contain a valid Signed-off-by line

e673867: Add int, char, and MPI_Info-related types

check_signed_off: does not contain a valid Signed-off-by line

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-23T14:39:57Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

5127029: Add ELEMENT_COUNT for MPI_Get_elements_x

check_signed_off: does not contain a valid Signed-off-by line

fc151a8: Add DATATYPE, STATUS, and GREQUEST_ types*

check_signed_off: does not contain a valid Signed-off-by line

e673867: Add int, char, and MPI_Info-related types

check_signed_off: does not contain a valid Signed-off-by line

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-24T17:52:23Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

5127029: Add ELEMENT_COUNT for MPI_Get_elements_x

check_signed_off: does not contain a valid Signed-off-by line

fc151a8: Add DATATYPE, STATUS, and GREQUEST_ types*

check_signed_off: does not contain a valid Signed-off-by line

e673867: Add int, char, and MPI_Info-related types

check_signed_off: does not contain a valid Signed-off-by line

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2024-04-24T21:10:27Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

5127029: Add ELEMENT_COUNT for MPI_Get_elements_x

check_signed_off: does not contain a valid Signed-off-by line

fc151a8: Add DATATYPE, STATUS, and GREQUEST_ types*

check_signed_off: does not contain a valid Signed-off-by line

e673867: Add int, char, and MPI_Info-related types

check_signed_off: does not contain a valid Signed-off-by line

01fa44e: Add PARTITIONED_COUNT type

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Signed-off-by: Howard Pritchard <[email protected]>

Signed-off-by: Jake Tronge <[email protected]>

Signed-off-by: Howard Pritchard <[email protected]>

hppritcha · 2024-08-21T19:08:12Z

bot:ompi:retest

and fix a make distcheck problem detected by AWS jenkins ci Signed-off-by: Howard Pritchard <[email protected]>

Signed-off-by: Jake Tronge <[email protected]>

Signed-off-by: Howard Pritchard <[email protected]>

Signed-off-by: Jake Tronge <[email protected]>

defined and passing in CEA test harness Signed-off-by: Howard Pritchard <[email protected]>

Tested with CEA pcvs tool Signed-off-by: Jake Tronge <[email protected]>

Signed-off-by: Jake Tronge <[email protected]>

win_allocate win_allocate_shared win_create win_shared_query fix win_detach Signed-off-by: Howard Pritchard <[email protected]>

Signed-off-by: Jake Tronge <[email protected]>

Signed-off-by: Howard Pritchard <[email protected]>

Signed-off-by: Jake Tronge <[email protected]>

set back to 3.1 compliance for now Signed-off-by: Howard Pritchard <[email protected]>

Signed-off-by: Jake Tronge <[email protected]>

Signed-off-by: Howard Pritchard <[email protected]>

Signed-off-by: Jake Tronge <[email protected]>

github-actions bot added the Target: main label Jan 10, 2024

hppritcha requested a review from jsquyres January 10, 2024 21:18

jtronge force-pushed the bigcount branch from 1a9a108 to 37ada30 Compare January 11, 2024 15:29

jsquyres mentioned this pull request Jan 30, 2024

mpi_f08 configure tests failing for NAG #9795

Open

jsquyres reviewed Feb 7, 2024

View reviewed changes

jtronge mentioned this pull request Feb 7, 2024

Standard ABI generation code and library refactor. #12033

Draft

hppritcha reviewed Feb 8, 2024

View reviewed changes

hppritcha reviewed Feb 12, 2024

View reviewed changes

hppritcha mentioned this pull request Feb 14, 2024

COLL framework needs work to support MPI Bigcount #12336

Closed

jtronge force-pushed the bigcount branch from 2ae81f2 to cad1e4e Compare April 1, 2024 20:14

jtronge force-pushed the bigcount branch from 66909b4 to e00da70 Compare April 22, 2024 16:44

jsquyres mentioned this pull request Jul 13, 2024

Fortran: first try if type(*), dimension(..) works as-is #11591

Draft

jtronge force-pushed the bigcount branch from 48b2f53 to f49c835 Compare July 26, 2024 17:07

hppritcha and others added 6 commits August 19, 2024 15:14

forgot some files

8a366fc

Signed-off-by: Howard Pritchard <[email protected]>

mpi4py fix

b8553c3

Signed-off-by: Howard Pritchard <[email protected]>

mpi4py: more fixes

557f481

Signed-off-by: Howard Pritchard <[email protected]>

fix for mpi_type_create_resized

0a10f95

Signed-off-by: Howard Pritchard <[email protected]>

Cast function pointers for register_datatrep call

16e2e01

Signed-off-by: Jake Tronge <[email protected]>

mpi4py: more fixes

19d703b

Signed-off-by: Howard Pritchard <[email protected]>

hppritcha and others added 23 commits August 21, 2024 16:10

mpi4py: some more fixes

cc50e01

and fix a make distcheck problem detected by AWS jenkins ci Signed-off-by: Howard Pritchard <[email protected]>

Remove old TODOs

64aec2f

Signed-off-by: Jake Tronge <[email protected]>

Convert additional Fortran functions

4a93847

Signed-off-by: Jake Tronge <[email protected]>

remove more unneeded TODO comments

5b034db

Signed-off-by: Howard Pritchard <[email protected]>

Fix error in free_mem_ts.c.in

8ac5c75

Signed-off-by: Jake Tronge <[email protected]>

fortran bindings: get MPI_Win_shared_query_c

c75ee98

defined and passing in CEA test harness Signed-off-by: Howard Pritchard <[email protected]>

fortran bindings: convert MPI_Get_accumulate

9484376

Tested with CEA pcvs tool Signed-off-by: Jake Tronge <[email protected]>

fortran bindings: Convert MPI_Get_address and MPI_Get

37dccba

Signed-off-by: Jake Tronge <[email protected]>

fortran bindings: Fix C call for MPI_Get

50d480f

Signed-off-by: Jake Tronge <[email protected]>

fortran bindings: Convert MPI_Iallgather* functions

01dafa0

Signed-off-by: Jake Tronge <[email protected]>

mpi_f08: add some win functions

1c4d384

win_allocate win_allocate_shared win_create win_shared_query fix win_detach Signed-off-by: Howard Pritchard <[email protected]>

Convert iallreduce, ialltoall* functions for Fortran

45e42c8

Signed-off-by: Jake Tronge <[email protected]>

Convert MPI_{Ibcast, Ibsend, Iexscan, Igather} functions for Fortran

a5de974

Signed-off-by: Jake Tronge <[email protected]>

update pack/unpack related functions

2c5d0b2

Signed-off-by: Howard Pritchard <[email protected]>

Convert MPI_Igatherv and MPI_Imrecv for fortran

5e72dd4

Signed-off-by: Jake Tronge <[email protected]>

Convert Fortran MPI_Ineighbor_allgather* functions

4628e20

Signed-off-by: Jake Tronge <[email protected]>

VERSION

5650c84

set back to 3.1 compliance for now Signed-off-by: Howard Pritchard <[email protected]>

Convert MPI_Ineighbor_alltoall* functions for Fortran

2c05a74

Signed-off-by: Jake Tronge <[email protected]>

add support for several type constructors

58197dd

Signed-off-by: Howard Pritchard <[email protected]>

Convert Fortran MPI_Ireduce_* functions

b7fc3f6

Signed-off-by: Jake Tronge <[email protected]>

Convert MPI_Irsend and MPI_Iscan for Fortran

06771fb

Signed-off-by: Jake Tronge <[email protected]>

Convert MPI_Iscatter* functions for Fortran

c99da54

Signed-off-by: Jake Tronge <[email protected]>

Convert MPI_Isendrecv and MPI_Isendrecv_replace for fortran

b4e1405

Signed-off-by: Jake Tronge <[email protected]>

		@@ -41,6 +41,36 @@ endif

		headers = bindings.h

		prototype_sources = \

		@@ -60,7 +90,7 @@ endif
		# List of all C files that have profile versions
		#

Generate bigcount interfaces for Fortran and C #12226

Are you sure you want to change the base?

Generate bigcount interfaces for Fortran and C #12226

Conversation

jtronge commented Jan 10, 2024

jsquyres left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtronge Feb 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsquyres commented Feb 7, 2024

jtronge commented Feb 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtronge commented Feb 14, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 3, 2024

github-actions bot commented Apr 16, 2024

github-actions bot commented Apr 17, 2024

github-actions bot commented Apr 17, 2024

github-actions bot commented Apr 18, 2024

github-actions bot commented Apr 22, 2024

github-actions bot commented Apr 22, 2024

github-actions bot commented Apr 23, 2024

github-actions bot commented Apr 24, 2024

github-actions bot commented Apr 24, 2024

hppritcha commented Aug 21, 2024

jtronge Feb 7, 2024 •

edited

Loading