Python integration using F2PY

[arunoruto]

First and foremost, awesome work. I use MSTM with a Matlab wrapper, which generates the needed input file, executes the command, and parses the output file. Down the line, I switched to python and wanted to implement a similar approach, but then I found about f2py and was wondering if it would be possible to build an interface for python to bypass the creation of the input file and parsing of the output file.

I have zero experience with Fortran and was wondering if you could give me some hints about where I could inject/modify/expand the code with the f2py syntax to build an interface suitable for python. Extending it like this could also make it possible to use frameworks like numba or dask to make the parallelized execution on multiple threads or across an HPC cluster more flexible.

[dmckwski]

Thanks for the compliment! I have zero experience with python, so I don't know how helpful my suggestions would be.

What I assume this project would involve is peeling off from mstm the modules and subroutines that perform the calculations, putting them together in a block, and using f2py to link them to a python program that does the input/output and initialization. Almost all of the input and initialization processes in mstm are done within the "variable_list_operation" and "inputdata" subroutines located in the "inputinterface" module, and within the "program main" at the very end of the code.

Perhaps the simplest way forward would be to replace the fortran "program main" with a fortran subroutine. The current main parses the command-line arguments to open and read the input file. The file contents are then copied into a character array -- each element in array corresponding to a line in the input file -- and this array is sent to the inputdata subroutine to start the process. The main program also does some other things, like initialization of MPI and "new_run" and "loop_variable" implementation. The subroutine replacement would have a character array argument containing the input file data. It would initialize mpi and call "inputdata" using the character array. The new_run and loop_variable operations could be done within python.

Output would be more of a problem. Most all of the output is done within the "output_header", "print_run_variables", and "print_calculation_results", but there are "write" statements throughout the code, used to report on calculation progress, error alerts, etc. Using python to parse an output file is going to be the easiest way forward, I believe. I've written several mathematica codes which do this -- likely very similar to your matlab wrappers.

[arunoruto]

So in the past few days, I tried to make something work out, so I will share my progress.

The naive way was to invoke the f2py command and generate the needed module for python. I did this using the command f2py3.10 --verbose -c --f90flags='-O2 -fallow-argument-mismatch' -m mstm mstm-intrinsics.f90 mpidefs-serial.f90 mstm-v4.0.f90. The logs of the compilation attempts are long, so here is a link to pastebin with the complete log: https://pastebin.com/4GPwufTS

Relevant lines would be 288, 289, and 307:

...
routsign2map: Confused: function gkintegrate has externals ['qsub'] but no "use" statement.
sign2map: Confused: external qsub is not in lcb_map[].
...
getctype: No C-type found in "{'=': '(1.d0', 'attrspec': ['optional']}", assuming void.
...

I tried to comment out everything past line 2775, leaving only the surface_subroutines module and what comes before it. This results in a similar problem, but with a GCC log found here: https://pastebin.com/8KbEu17p

It seems that the external functions presented in the subroutine qng and used in gkintegrate. I tried different methods but wasn't able to compile it through. Maybe you could know what is wrong, or someone else has a solution.

[dmckwski]

What stands out to me is that subroutine gkintegrate is a recursive subroutine, i.e., it calls itself. The log makes it appear that the issue is the subroutine argument qsub, which refers to the external function that evaluates the integrand, yet the subroutine qng, which appears directly before gkintegrate and provides the basic driver for the numerical integrator, also uses the external-function-as-subroutine-argument protocol and has no problems with f2py. I don't know if the recursive feature is incompatible with python.

[arunoruto]

If I comment out the gkintegrate subroutine it does compile without problems (I have to comment out the flush subroutine in mstm-intrinsics.f90, but that's because I use gfortran on fedora I guess). So I guess the recursiveness of the subroutine is the problem. A similar error happens in this StackOverflow question, and the problem was solved by defining the output type. I am not sure if this is applicable here too.

EDIT: here is a case where someone is using recursive subroutines and it seems to work for them.