docs: Add explicit docs about --with-mpi-moduledir

As a follow on to 7338594, add some additional information about
the --with-mpi-moduledir configure CLI option, especially as it
pertains to making relocatable Open MPI installations.

Also, remove an errant space that crept into the mpifort wrapper
compiler flags in 7338594.

Signed-off-by: Jeff Squyres <[email protected]>

config/opal_setup_wrappers.m4

@@ -541,7 +541,7 @@ AC_DEFUN([OPAL_SETUP_WRAPPER_FINAL],[
        OMPI_WRAPPER_FCFLAGS='-I${includedir}'" ${wrapper_extra_fcflags} ${with_wrapper_fcflags}"
        AS_IF([test -n "${OMPI_FC_MODULE_FLAG}"],
              [dnl deal with some interesting expansion behavior in OPAL_APPEND
-              wrapper_tmp_arg="${OMPI_FC_MODULE_FLAG} ${OMPI_FORTRAN_MODULEDIR}"
+              wrapper_tmp_arg="${OMPI_FC_MODULE_FLAG}${OMPI_FORTRAN_MODULEDIR}"
               OPAL_APPEND([OMPI_WRAPPER_FCFLAGS], [${wrapper_tmp_arg}])])
        AC_SUBST([OMPI_WRAPPER_FCFLAGS])
        AC_MSG_RESULT([$OMPI_WRAPPER_EXTRA_FCFLAGS])
@@ -780,7 +780,7 @@ AC_DEFUN([OPAL_SETUP_WRAPPER_FINAL],[
        OSHMEM_WRAPPER_FCFLAGS='-I${includedir}'" ${wrapper_extra_fcflags} ${with_wrapper_fcflags}"
        AS_IF([test -n "${OMPI_FC_MODULE_FLAG}"],
              [dnl deal with some interesting expansion behavior in OPAL_APPEND
-              wrapper_tmp_arg="${OMPI_FC_MODULE_FLAG} ${OMPI_FORTRAN_MODULEDIR}"
+              wrapper_tmp_arg="${OMPI_FC_MODULE_FLAG}${OMPI_FORTRAN_MODULEDIR}"
               OPAL_APPEND([OSHMEM_WRAPPER_FCFLAGS], [${wrapper_tmp_arg}])])
        AC_SUBST([OSHMEM_WRAPPER_FCFLAGS])
        AC_MSG_RESULT([$OSHMEM_WRAPPER_EXTRA_FCFLAGS])

docs/installing-open-mpi/configure-cli-options/misc.rst

@@ -63,3 +63,35 @@ above categories that can be used with ``configure``:
   See the section on :ref:`customizing wrapper compiler behavior
   <label-customizing-wrapper-compiler>` to see how to alter the
   wrapper compiler behavior at run time.
+
+* ``--with-mpi-moduledir``: Specify the directory where the Fortran
+  MPI module files are installed.
+
+  For historical reasons, Open MPI's Fortran MPI modulefiles are
+  installed into ``$libdir`` by default.  This option lets you change
+  where they are installed; some users prefer Fortran module files
+  installed into ``$installdir``, for example.
+
+  .. note:: If you intend to make your Open MPI installation
+            relocatable :ref:`via the OPAL_PREFIX mechanism
+            <install-location-opal-prefix>`, you will want to ensure
+            to specify the module directory relative to the
+            ``prefix``.  For example:
+
+            .. code-block::
+
+               $ ./configure --prefix=/opt/openmpi --with-mpi-moduledir='${includedir}/modules`...
+
+            Note the additional shell quoting that is likely necessary
+            to prevent shell variable expansion, and the additional
+            ``${}`` around ``includedir`` that is necessary for Open MPI
+            to recognize that it is a special name that needs to be
+            expanded.
+
+            Finally, note that the Fortran module installation
+            directory is *not* one of the :ref:`recognized directories
+            that can be specified via environment variable at run time
+            <install-location-overriding-individual-directories>`.
+            Instead, to make the Fortran module directory relocatable,
+            it needs to be relative to one of the other recognized
+            directories.

docs/installing-open-mpi/installation-location.rst

@@ -162,6 +162,8 @@ variables to the new location where Open MPI now resides).
 
 There are three methods.
 
+.. _install-location-opal-prefix:
+
 Move an existing Open MPI installation to a new prefix
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -171,6 +173,41 @@ MPI.  For example, if Open MPI had initially been installed to
 ``/home/openmpi``, setting ``OPAL_PREFIX`` to ``/home/openmpi`` will
 enable Open MPI to function properly.
 
+.. note:: The ``OPAL_PREFIX`` mechanism relies on all installation
+          directories being specified as relative to the ``prefix``
+          directory specified during ``configure``.
+
+          For example, if Open MPI is configured the following way:
+
+          .. code-block::
+
+             $ ./configure --prefix=/opt/openmpi --libdir=/usr/lib ...
+
+          Then setting ``OPAL_PREFIX`` will not affect the run-time
+          implications of ``libdir``, since ``/usr/lib`` is not
+          specified as relative to ``/opt/openmpi``.
+
+          Instead of specifying absolute directories, you can make
+          them relative to other ``configure``-recognized directories.
+          For example:
+
+          .. code-block::
+
+             $ ./configure --prefix=/opt/openmpi --libdir='${exec_prefix}/x86_64/lib' ...
+
+          Note the additional shell quoting that is likely necessary
+          to prevent shell variable expansion, and the additional
+          ``${}`` around ``exec_prefix`` that is necessary for Open MPI
+          to recognize that it is a special name that needs to be
+          expanded.
+
+          The directory names recognized by Open MPI are listed in the
+          :ref:`Overriding individual directories
+          <install-location-overriding-individual-directories>`
+          section (below), without the ``OPAL_`` prefix, and in lower
+          case.  For example, the ``OPAL_SYSCONFDIR`` environment
+          variable corresponds to ``${sysconfdir}``.
+
 "Stage" an Open MPI installation in a temporary location
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -189,6 +226,8 @@ environment variable can be used; setting ``OPAL_DESTDIR`` to
 ``/var/rpm/build.1234`` will automatically prefix every directory such
 that Open MPI can function properly.
 
+.. _install-location-overriding-individual-directories:
+
 Overriding individual directories
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^