Chap_API_Server.tex

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Chapter: API Server
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Server-Specific Interfaces}
\label{chap:api_server}

The process that hosts the \ac{PMIx} server library interacts with that library in two distinct manners. First, \ac{PMIx} provides a set of \acp{API} by which the host can request specific services from its library. This includes:

\begin{compactitemize}
    \item collecting inventory to support scheduling algorithms,
    \item providing subsystems with an opportunity to precondition their resources for optimized application support,
    \item generating regular expressions,
    \item registering information to be passed to client processes, and
    \item requesting information on behalf of a remote process.
\end{compactitemize}

Note that the host always has access to all \ac{PMIx} client \acp{API} - the functions listed below are in addition to those available to a \ac{PMIx} client.

Second, the host can provide a set of callback functions by which the \ac{PMIx} server library can pass requests upward for servicing by the host. These include notifications of client connection and finalize, as well as requests by clients for information and/or services that the \ac{PMIx} server library does not itself provide.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Server Initialization and Finalization}
\label{chap:api_init:server}

Initialization and finalization routines for \ac{PMIx} servers.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_init}}
\declareapi{PMIx_server_init}

%%%%
\summary

Initialize the \ac{PMIx} server.

%%%%
\format

\copySignature{PMIx_server_init}{1.0}{
pmix_status_t \\
PMIx_server_init(pmix_server_module_t *module, \\
\hspace*{17\sigspace}pmix_info_t info[], size_t ninfo);
}

\begin{arglist}
\arginout{module}{\refapi{pmix_server_module_t} structure (handle)}
\argin{info}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (\code{size_t})}
\end{arglist}

\returnsimple

\reqattrstart
The following attributes are required to be supported by all \ac{PMIx} libraries:

\pasteAttributeItem{PMIX_SERVER_NSPACE}
\pasteAttributeItem{PMIX_SERVER_RANK}
\pasteAttributeItem{PMIX_SERVER_TMPDIR}
\pasteAttributeItem{PMIX_SYSTEM_TMPDIR}
\pasteAttributeItem{PMIX_SERVER_TOOL_SUPPORT}
\pasteAttributeItem{PMIX_SERVER_SYSTEM_SUPPORT}
\pasteAttributeItem{PMIX_SERVER_SESSION_SUPPORT}
\pasteAttributeItem{PMIX_SERVER_GATEWAY}
\pasteAttributeItem{PMIX_SERVER_SCHEDULER}

\reqattrend

\optattrstart
The following attributes are optional for implementers of \ac{PMIx} libraries:

\pasteAttributeItemBegin{PMIX_USOCK_DISABLE} If the library supports Unix socket connections, this attribute may be supported for disabling it.
\pasteAttributeItemEnd{}
\pasteAttributeItemBegin{PMIX_SOCKET_MODE} If the library supports socket connections, this attribute may be supported for setting the socket mode.
\pasteAttributeItemEnd{}
\pasteAttributeItem{PMIX_SINGLE_LISTENER}
\pasteAttributeItemBegin{PMIX_TCP_REPORT_URI} If the library supports TCP socket connections, this attribute may be supported for reporting the URI.
\pasteAttributeItemEnd{}
\pasteAttributeItemBegin{PMIX_TCP_IF_INCLUDE} If the library supports TCP socket connections, this attribute may be supported for specifying the interfaces to be used.
\pasteAttributeItemEnd{}
\pasteAttributeItemBegin{PMIX_TCP_IF_EXCLUDE} If the library supports TCP socket connections, this attribute may be supported for specifying the interfaces that are \textit{not} to be used.
\pasteAttributeItemEnd{}
\pasteAttributeItemBegin{PMIX_TCP_IPV4_PORT} If the library supports IPV4 connections, this attribute may be supported for specifying the port to be used.
\pasteAttributeItemEnd{}
\pasteAttributeItemBegin{PMIX_TCP_IPV6_PORT} If the library supports IPV6 connections, this attribute may be supported for specifying the port to be used.
\pasteAttributeItemEnd{}
\pasteAttributeItemBegin{PMIX_TCP_DISABLE_IPV4} If the library supports IPV4 connections, this attribute may be supported for disabling it.
\pasteAttributeItemEnd{}
\pasteAttributeItemBegin{PMIX_TCP_DISABLE_IPV6} If the library supports IPV6 connections, this attribute may be supported for disabling it.
\pasteAttributeItemEnd{}
\pasteAttributeItemBegin{PMIX_SERVER_REMOTE_CONNECTIONS} If the library supports connections from remote tools, this attribute may be supported for enabling or disabling it.
\pasteAttributeItemEnd{}
\pasteAttributeItem{PMIX_EXTERNAL_PROGRESS}
\pasteAttributeItem{PMIX_EVENT_BASE}
\pasteAttributeItem{PMIX_TOPOLOGY2}
\pasteAttributeItemBegin{PMIX_SERVER_SHARE_TOPOLOGY}The \ac{PMIx} server will
perform the necessary actions to scalably expose the description to the local
clients. This includes creating any required shared memory backing stores and/
or \ac{XML} representations, plus ensuring that all necessary key-value pairs
for clients to access the description are included in the job-level
information provided to each client. All required files are to be installed
under the effective \refattr{PMIX_SERVER_TMPDIR} directory. The \ac{PMIx}
server library is responsible for cleaning up any artifacts (e.g., shared
memory backing files or cached key-value pairs) at library finalize.
\pasteAttributeItemEnd{}
\pasteAttributeItem{PMIX_SERVER_ENABLE_MONITORING}
\pasteAttributeItem{PMIX_HOMOGENEOUS_SYSTEM}
\pasteAttributeItem{PMIX_SINGLETON}
\pasteAttributeItem{PMIX_IOF_LOCAL_OUTPUT}

\optattrend

%%%%
\descr

Initialize the \ac{PMIx} server support library, and provide a pointer to a \refapi{pmix_server_module_t} structure containing the caller's callback functions.
The array of \refstruct{pmix_info_t} structs is used to pass additional info that may be required by the server when initializing.
For example, it may include the \refattr{PMIX_SERVER_TOOL_SUPPORT} attribute, thereby indicating that the daemon is willing to accept connection requests from tools.

\advicermstart
Providing a value of \code{NULL} for the \refarg{module} argument is permitted, as is passing an empty \refarg{module} structure. Doing so indicates that the host environment will not provide support for multi-node operations such as \refapi{PMIx_Fence}, but does intend to support local clients access to information.
\advicermend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_finalize}}
\declareapi{PMIx_server_finalize}

%%%%
\summary

Finalize the PMIx server library.

%%%%
\format

\copySignature{PMIx_server_finalize}{1.0}{
pmix_status_t \\
PMIx_server_finalize(void);
}

\returnsimple

%%%%
\descr

Finalize the \ac{PMIx} server support library, terminating all connections to attached tools and any local clients.
All memory usage is released.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Server Initialization Attributes}
\label{chap:api_init:serverattrs}

These attributes are used to direct the configuration and operation of the \ac{PMIx} server library by passing them into \refapi{PMIx_server_init}.

%
\declareAttribute{PMIX_TOPOLOGY2}{"pmix.topo2"}{pmix_topology_t}{
Provide a pointer to an implementation-specific description of the local node
topology.
}
%
\declareAttribute{PMIX_SERVER_SHARE_TOPOLOGY}{"pmix.srvr.share"}{bool}{
The \ac{PMIx} server is to share its copy of the local node topology (whether given to it or self-discovered) with any clients.
}
%
\declareAttribute{PMIX_USOCK_DISABLE}{"pmix.usock.disable"}{bool}{
Disable legacy UNIX socket (usock) support.
}
%
\declareAttribute{PMIX_SOCKET_MODE}{"pmix.sockmode"}{uint32_t}{
POSIX \var{mode_t} (9 bits valid).
}
%
\declareAttribute{PMIX_SINGLE_LISTENER}{"pmix.sing.listnr"}{bool}{
Use only one rendezvous socket, letting priorities and/or environment parameters select the active transport.
}
%
\declareAttribute{PMIX_SERVER_TOOL_SUPPORT}{"pmix.srvr.tool"}{bool}{
The host \ac{RM} wants to declare itself as willing to accept tool connection requests.
}
%
\declareAttribute{PMIX_SERVER_REMOTE_CONNECTIONS}{"pmix.srvr.remote"}{bool}{
Allow connections from remote tools. Forces the PMIx server to not exclusively use loopback device.
}
%
\declareAttribute{PMIX_SERVER_SYSTEM_SUPPORT}{"pmix.srvr.sys"}{bool}{
The host \ac{RM} wants to declare itself as being the local system server for PMIx connection requests.
}
%
\declareAttribute{PMIX_SERVER_SESSION_SUPPORT}{"pmix.srvr.sess"}{bool}{
The host \ac{RM} wants to declare itself as being the local session server for PMIx connection requests.
}
%
\declareAttribute{PMIX_SERVER_START_TIME}{"pmix.srvr.strtime"}{char*}{
Time when the server started - i.e., when the server created it's rendezvous file (given in ctime string format).
}
%
\declareAttribute{PMIX_SERVER_TMPDIR}{"pmix.srvr.tmpdir"}{char*}{
Top-level temporary directory for all client processes connected to this server, and where the PMIx server will place its tool rendezvous point and contact information.
}
%
\declareAttribute{PMIX_SYSTEM_TMPDIR}{"pmix.sys.tmpdir"}{char*}{
Temporary directory for this system, and where a PMIx server that declares itself to be a system-level server will place a tool rendezvous point and contact information.
}
%
\declareAttribute{PMIX_SERVER_ENABLE_MONITORING}{"pmix.srv.monitor"}{bool}{
Enable PMIx internal monitoring by the PMIx server.
}
%
\declareAttribute{PMIX_SERVER_NSPACE}{"pmix.srv.nspace"}{char*}{
Name of the namespace to use for this PMIx server.
}
%
\declareAttribute{PMIX_SERVER_RANK}{"pmix.srv.rank"}{pmix_rank_t}{
Rank of this PMIx server.
}
%
\declareAttribute{PMIX_SERVER_GATEWAY}{"pmix.srv.gway"}{bool}{
Server is acting as a gateway for PMIx requests that cannot be serviced on backend nodes (e.g., logging to email).
}
%
\declareAttribute{PMIX_SERVER_SCHEDULER}{"pmix.srv.sched"}{bool}{
Server is supporting system scheduler and desires access to appropriate \ac{WLM}-supporting features. Indicates that the library is to be initialized for scheduler support.
}
%
\declareAttribute{PMIX_EXTERNAL_PROGRESS}{"pmix.evext"}{bool}{
The host shall progress the \ac{PMIx} library via calls to \refapi{PMIx_Progress}
}
%
\declareAttribute{PMIX_HOMOGENEOUS_SYSTEM}{"pmix.homo"}{bool}{
The nodes comprising the session are homogeneous - i.e., they each contain the same number of identical packages, fabric interfaces, \acp{GPU}, and other devices.
}
%
\declareAttributeProvisional{PMIX_SINGLETON}{"pmix.singleton"}{char*}{
String representation (nspace.rank) of proc ID for the singleton the server was started to support
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Server Support Functions}

The following \acp{API} allow the \ac{RM} daemon that hosts the \ac{PMIx} server library to request specific services from the \ac{PMIx} library.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_generate_regex}}
\declareapi{PMIx_generate_regex}

%%%%
\summary

Generate a compressed representation of the input string.

%%%%
\format

\copySignature{PMIx_generate_regex}{1.0}{
pmix_status_t \\
PMIx_generate_regex(const char *input, char **output);
}

\begin{arglist}
\argin{input}{String to process (string)}
\argout{output}{Compressed representation of \refarg{input} (array of bytes)}
\end{arglist}

\returnsimple

%%%%
\descr

Given a comma-separated list of \refarg{input} values, generate a reduced size representation of the input that can be passed down to the \ac{PMIx} server library's \refapi{PMIx_server_register_nspace} \ac{API} for parsing. The order of the individual values in the \refarg{input} string is preserved across the operation. The caller is responsible for releasing the returned data.

\label{regex:fmt}The precise compressed representations will be implementation specific. The regular expression itself is not required to be a printable string nor to obey typical string constraints (e.g., include a \code{NULL} terminator byte). However, all \ac{PMIx} implementations are required to include a colon-delimited \code{NULL}-terminated string at the beginning of the output representation that can be printed for diagnostic purposes and identifies the method used to generate the representation. The following identifiers are reserved by the \ac{PMIx} Standard:

\begin{itemize}
    \item \code{"raw:\textbackslash0"} - indicates that the expression following the identifier is simply the comma-delimited input string (no processing was performed).
    \item \code{"pmix:\textbackslash0"} - a \ac{PMIx}-unique regular expression represented as a \code{NULL}-terminated string following the identifier.
    \item \code{"blob:\textbackslash0"} - a \ac{PMIx}-unique regular expression that is not represented as a \code{NULL}-terminated string following the identifier. Additional implementation-specific metadata may follow the identifier along with the data itself. For example, a compressed binary array format based on the \emph{zlib} compression package, with the size encoded in the space immediately following the identifier.
\end{itemize}

Communicating the resulting output should be done by first packing the returned expression using the \refapi{PMIx_Data_pack}, declaring the input to be of type \refconst{PMIX_REGEX}, and then obtaining the resulting blob to be communicated using the \refmacro{PMIX_DATA_BUFFER_UNLOAD} macro. The reciprocal method can be used on the remote end prior to passing the regex into \refapi{PMIx_server_register_nspace}. The pack/unpack routines will ensure proper handling of the data based on the regex prefix.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_generate_ppn}}
\declareapi{PMIx_generate_ppn}

%%%%
\summary

Generate a compressed representation of the input identifying the processes on each node.

%%%%
\format

\copySignature{PMIx_generate_ppn}{1.0}{
pmix_status_t \\
PMIx_generate_ppn(const char *input, char **ppn);
}

\begin{arglist}
\argin{input}{String to process (string)}
\argout{ppn}{Compressed representation of \refarg{input} (array of bytes)}
\end{arglist}

\returnsimple

%%%%
\descr

The input shall consist of a semicolon-separated list of ranges representing the ranks of processes on each node of the job - e.g., \code{"1-4;2-5;8,10,11,12;6,7,9"}. Each field of the input must correspond to the node name provided at that position in the input to \refapi{PMIx_generate_regex}. Thus, in the example, ranks 1-4 would be located on the first node of the comma-separated list of names provided to \refapi{PMIx_generate_regex}, and ranks 2-5 would be on the second name in the list.

Rules governing the format of the returned regular expression are the same as those specified for \refapi{PMIx_generate_regex}, as detailed \hyperref[regex:fmt]{here}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_register_nspace}}
\declareapi{PMIx_server_register_nspace}

%%%%
\summary

Setup the data about a particular namespace.

%%%%
\format

\copySignature{PMIx_server_register_nspace}{1.0}{
pmix_status_t \\
PMIx_server_register_nspace(const pmix_nspace_t nspace, \\
\hspace*{28\sigspace}int nlocalprocs, \\
\hspace*{28\sigspace}pmix_info_t info[], size_t ninfo, \\
\hspace*{28\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{28\sigspace}void *cbdata);
}

\begin{arglist}
\argin{nspace}{Character array of maximum size \refconst{PMIX_MAX_NSLEN} containing the namespace identifier (string)}
\argin{nlocalprocs}{number of local processes (integer)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} to be executed upon completion of the operation. A \code{NULL} function reference indicates that the function is to be executed as a blocking operation (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\returnsimplenb

\returnstart
\begin{itemize}
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
\end{itemize}
\returnend

\reqattrstart
The following attributes are required to be supported by all \ac{PMIx} libraries:

\pasteAttributeItem{PMIX_REGISTER_NODATA}
\pasteAttributeItem{PMIX_SESSION_INFO_ARRAY}
\pasteAttributeItem{PMIX_JOB_INFO_ARRAY}
\pasteAttributeItem{PMIX_APP_INFO_ARRAY}
\pasteAttributeItem{PMIX_PROC_INFO_ARRAY}
\pasteAttributeItem{PMIX_NODE_INFO_ARRAY}

\divider

Host environments are required to provide a wide range of session-, job-, application-, node-, and process-realm information, and may choose to provide a similarly wide range of optional information. The information is broadly separated into categories based on the \emph{data realm} definitions explained in Section \ref{api:struct:attributes:retrieval}, and retrieved according to the rules detailed in Section \ref{chap:api_rsvd_keys:retrules}.

Session-realm information may be passed as individual \refstruct{pmix_info_t} entries, or as part of a \refstruct{pmix_data_array_t} using the \refattr{PMIX_SESSION_INFO_ARRAY} attribute. The list of data referenced in this way shall include:

\begin{itemize}
    \item \pasteAttributeItem{PMIX_UNIV_SIZE}
    \item \pasteAttributeItemBegin{PMIX_MAX_PROCS}Must be provided if \refattr{PMIX_UNIV_SIZE} is not given. Requires use of the \refattr{PMIX_SESSION_INFO} attribute to avoid ambiguity when retrieving it.
    \pasteAttributeItemEnd
    \item \pasteAttributeItem{PMIX_SESSION_ID}
\end{itemize}

plus the following optional information:

\begin{itemize}
    \item \pasteAttributeItemBegin{PMIX_CLUSTER_ID}As this information is not related to the namespace, it is best passed using the \refapi{PMIx_server_register_resources} \ac{API}.
    \pasteAttributeItemEnd
    \item \pasteAttributeItem{PMIX_ALLOCATED_NODELIST}
    \item \pasteAttributeItemBegin{PMIX_RM_NAME}As this information is not related to the namespace, it is best passed using the \refapi{PMIx_server_register_resources} \ac{API}.
    \pasteAttributeItemEnd
    \item \pasteAttributeItemBegin{PMIX_RM_VERSION}As this information is not related to the namespace, it is best passed using the \refapi{PMIx_server_register_resources} \ac{API}.
    \pasteAttributeItemEnd
    \item \pasteAttributeItemBegin{PMIX_SERVER_HOSTNAME}As this information is not related to the namespace, it is best passed using the \refapi{PMIx_server_register_resources} \ac{API}.
    \pasteAttributeItemEnd
\end{itemize}

Job-realm information may be passed as individual \refstruct{pmix_info_t} entries, or as part of a \refstruct{pmix_data_array_t} using the \refattr{PMIX_JOB_INFO_ARRAY} attribute. The list of data referenced in this way shall include:

\begin{itemize}
    \item \pasteAttributeItemBegin{PMIX_SERVER_NSPACE}Identifies the namespace of the \ac{PMIx} server itself
    \pasteAttributeItemEnd
    \item \pasteAttributeItemBegin{PMIX_SERVER_RANK}Identifies the rank of the \ac{PMIx} server itself.
    \pasteAttributeItemEnd
    \item \pasteAttributeItemBegin{PMIX_NSPACE}Identifies the namespace of the job being registered.
    \pasteAttributeItemEnd
    \item \pasteAttributeItem{PMIX_JOBID}
    \item \pasteAttributeItem{PMIX_JOB_SIZE}
    \item \pasteAttributeItemBegin{PMIX_MAX_PROCS}Retrieval of this attribute defaults to the job level unless an appropriate specification is given (e.g., \refattr{PMIX_SESSION_INFO}).
    \pasteAttributeItemEnd
    \item \pasteAttributeItem{PMIX_NODE_MAP}
    \item \pasteAttributeItem{PMIX_PROC_MAP}
\end{itemize}

plus the following optional information:

\begin{itemize}
    \item \pasteAttributeItem{PMIX_NPROC_OFFSET}
    \item \pasteAttributeItemBegin{PMIX_JOB_NUM_APPS}This is a required attribute if more than one application is included in the job.
    \pasteAttributeItemEnd
    \item \pasteAttributeItem{PMIX_MAPBY}
    \item \pasteAttributeItem{PMIX_RANKBY}
    \item \pasteAttributeItem{PMIX_BINDTO}
    \item \pasteAttributeItem{PMIX_HOSTNAME_KEEP_FQDN}
    \item \pasteAttributeItem{PMIX_ANL_MAP}
    \item \pasteAttributeItem{PMIX_TDIR_RMCLEAN}
    \item \pasteAttributeItem{PMIX_CRYPTO_KEY}
\end{itemize}

If more than one application is included in the namespace, then the host environment is also required to supply data consisting of the following items for each application in the job, passed as a \refstruct{pmix_data_array_t} using the \refattr{PMIX_APP_INFO_ARRAY} attribute:

\begin{itemize}
    \item \pasteAttributeItemBegin{PMIX_APPNUM}This attribute must appear at the beginning of the array.
    \pasteAttributeItemEnd
    \item \pasteAttributeItem{PMIX_APP_SIZE}
    \item \pasteAttributeItemBegin{PMIX_MAX_PROCS}Requires use of the \refattr{PMIX_APP_INFO} attribute to avoid ambiguity when retrieving it.
    \pasteAttributeItemEnd
    \item \pasteAttributeItem{PMIX_APPLDR}
    \item \pasteAttributeItemBegin{PMIX_WDIR}This attribute is required for all registrations, but may be provided as an individual \refstruct{pmix_info_t} entry if only one application is included in the namespace.
    \pasteAttributeItemEnd
    \item \pasteAttributeItemBegin{PMIX_APP_ARGV}This attribute is required for all registrations, but may be provided as an individual \refstruct{pmix_info_t} entry if only one application is included in the namespace.
    \pasteAttributeItemEnd
\end{itemize}

plus the following optional information:

\begin{itemize}
    \item \pasteAttributeItem{PMIX_PSET_NAMES}
    \item \pasteAttributeItemBegin{PMIX_APP_MAP_TYPE}This attribute may be provided as an individual \refstruct{pmix_info_t} entry if only one application is included in the namespace.
    \pasteAttributeItemEnd
    \item \pasteAttributeItemBegin{PMIX_APP_MAP_REGEX}This attribute may be provided as an individual \refstruct{pmix_info_t} entry if only one application is included in the namespace.
    \pasteAttributeItemEnd
\end{itemize}

The data may also include attributes provided by the host environment that identify the programming model (as specified by the user) being executed within the application. The \ac{PMIx} server library may utilize this information to customize the environment to fit that model (e.g., adding environmental variables specified by the corresponding standard for that model):

\begin{itemize}
    \item \pasteAttributeItem{PMIX_PROGRAMMING_MODEL}
    \item \pasteAttributeItem{PMIX_MODEL_LIBRARY_NAME}
    \item \pasteAttributeItem{PMIX_MODEL_LIBRARY_VERSION}
\end{itemize}


Node-realm information may be passed as individual \refstruct{pmix_info_t} entries if only one node will host processes from the job being registered, or as part of a \refstruct{pmix_data_array_t} using the \refattr{PMIX_NODE_INFO_ARRAY} attribute when multiple nodes are involved in the job. The list of data referenced in this way shall include:

\begin{itemize}
    \item \pasteAttributeItem{PMIX_NODEID}
    \item \pasteAttributeItemBegin{PMIX_HOSTNAME}As this information is not related to the namespace, it can be passed using the \refapi{PMIx_server_register_resources} \ac{API}. However, either it or the \refattr{PMIX_NODEID} must be included in the array to properly identify the node.
    \pasteAttributeItemEnd
    \item \pasteAttributeItemBegin{PMIX_HOSTNAME_ALIASES}As this information is not related to the namespace, it is best passed using the \refapi{PMIx_server_register_resources} \ac{API}.
    \pasteAttributeItemEnd
    \item \pasteAttributeItem{PMIX_LOCAL_SIZE}
    \item \pasteAttributeItem{PMIX_NODE_SIZE}
    \item \pasteAttributeItem{PMIX_LOCALLDR}
    \item \pasteAttributeItem{PMIX_LOCAL_PEERS}
    \item \pasteAttributeItem{PMIX_NODE_OVERSUBSCRIBED}
\end{itemize}

plus the following information for the server's own node:

\begin{itemize}
    \item \pasteAttributeItem{PMIX_TMPDIR}
    \item \pasteAttributeItem{PMIX_NSDIR}
    \item \pasteAttributeItem{PMIX_LOCAL_PROCS}
\end{itemize}

The data may also include the following optional information for the server's own node:

\begin{itemize}
    \item \pasteAttributeItem{PMIX_LOCAL_CPUSETS}
    \item \pasteAttributeItemBegin{PMIX_AVAIL_PHYS_MEMORY}As this information is not related to the namespace, it can be passed using the \refapi{PMIx_server_register_resources} \ac{API}.
    \pasteAttributeItemEnd
\end{itemize}

and the following optional information for other nodes:

\begin{itemize}
    \item \pasteAttributeItemBegin{PMIX_MAX_PROCS}Requires use of the \refattr{PMIX_NODE_INFO} attribute to avoid ambiguity when retrieving it.
    \pasteAttributeItemEnd
\end{itemize}

Process-realm information shall include the following data for each process in the job, passed as a \refstruct{pmix_data_array_t} using the \refattr{PMIX_PROC_INFO_ARRAY} attribute:

\begin{itemize}
    \item \pasteAttributeItem{PMIX_RANK}
    \item \pasteAttributeItemBegin{PMIX_APPNUM}This attribute may be omitted if only one application is present in the namespace.
    \pasteAttributeItemEnd
    \item \pasteAttributeItemBegin{PMIX_APP_RANK}This attribute may be omitted if only one application is present in the namespace.
    \pasteAttributeItemEnd
    \item \pasteAttributeItem{PMIX_GLOBAL_RANK}
    \item \pasteAttributeItem{PMIX_LOCAL_RANK}
    \item \pasteAttributeItem{PMIX_NODE_RANK}
    \item \pasteAttributeItem{PMIX_NODEID}
    \item \pasteAttributeItem{PMIX_REINCARNATION}
    \item \pasteAttributeItem{PMIX_SPAWNED}
\end{itemize}

plus the following information for processes that are local to the server:

\begin{itemize}
    \item \pasteAttributeItem{PMIX_LOCALITY_STRING}
    \item \pasteAttributeItem{PMIX_PROCDIR}
    \item \pasteAttributeItem{PMIX_PACKAGE_RANK}
\end{itemize}

and the following optional information - note that some of this information can be derived from information already provided by other attributes, but it may be included here for ease of retrieval by users:

\begin{itemize}
    \item \pasteAttributeItem{PMIX_HOSTNAME}
    \item \pasteAttributeItem{PMIX_CPUSET}
    \item \pasteAttributeItem{PMIX_CPUSET_BITMAP}
    \item \pasteAttributeItem{PMIX_DEVICE_DISTANCES}
\end{itemize}

\divider

Attributes not directly provided by the host environment may be derived by the \ac{PMIx} server library from other required information and included in the data made available to the server library's clients.
\reqattrend


%%%%
\descr

Pass job-related information to the \ac{PMIx} server library for distribution to local client processes.

\advicermstart
Host environments are required to execute this operation prior to starting any local application process within the given namespace.

The \ac{PMIx} server must register all namespaces that will participate in collective operations with local processes.
This means that the server must register a namespace even if it will not host any local processes from within that namespace if any local process of another namespace might at some point perform an operation involving one or more processes from the new namespace.
This is necessary so that the collective operation can identify the participants and know when it is locally complete.

The caller must also provide the number of local processes that will be launched within this namespace.
This is required for the \ac{PMIx} server library to correctly handle collectives as a collective operation call can occur before all the local processes have been started.

A \code{NULL} \refarg{cbfunc} reference indicates that the function is to be executed as a blocking operation.
\advicermend

\adviceuserstart
The number of local processes for any given namespace is generally fixed at the time of application launch. Calls to \refapi{PMIx_Spawn} result in processes launched in their own namespace, not that of their parent. However, it is possible for processes to \textit{migrate} to another node via a call to \refapi{PMIx_Job_control_nb}, thus resulting in a change to the number of local processes on both the initial node and the node to which the process moved. It is therefore critical that applications not migrate processes without first ensuring that \ac{PMIx}-based collective operations are not in progress, and that no such operations be initiated until process migration has completed.
\adviceuserend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Namespace registration attributes}
\label{api:struct:attributes:storage}

The following attributes are defined specifically for use with the \refapi{PMIx_server_register_nspace} \ac{API}:
%
\declareAttribute{PMIX_REGISTER_NODATA}{"pmix.reg.nodata"}{bool}{
Registration is for this namespace only, do not copy job data.
}

\vspace{\baselineskip}
The following attributes are used to assemble information according to its data realm (\refterm{session}, \refterm{job}, \refterm{application}, \refterm{node}, or \refterm{process} as defined in Section \ref{api:struct:attributes:retrieval}) for registration where ambiguity may exist - see \ref{chap:api_server:assemble} for examples of their use.

%
\declareAttribute{PMIX_SESSION_INFO_ARRAY}{"pmix.ssn.arr"}{pmix_data_array_t}{
Provide an array of \refstruct{pmix_info_t} containing session-realm  information. The \refattr{PMIX_SESSION_ID} attribute is required to be included in the array.
}
%
\declareAttribute{PMIX_JOB_INFO_ARRAY}{"pmix.job.arr"}{pmix_data_array_t}{
Provide an array of \refstruct{pmix_info_t} containing job-realm information. The \refattr{PMIX_SESSION_ID} attribute of the \refterm{session} containing the \refterm{job} is required to be included in the array whenever the \ac{PMIx} server library may host multiple sessions (e.g., when executing with a host \ac{RM} daemon). As information is registered one job (aka namespace) at a time via the \refapi{PMIx_server_register_nspace} \ac{API}, there is no requirement that the array contain either the \refattr{PMIX_NSPACE} or \refattr{PMIX_JOBID} attributes when used in that context (though either or both of them may be included). At least one of the job identifiers must be provided in all other contexts where the job being referenced is ambiguous.
}
%
\declareAttribute{PMIX_APP_INFO_ARRAY}{"pmix.app.arr"}{pmix_data_array_t}{
Provide an array of \refstruct{pmix_info_t} containing application-realm information. The \refattr{PMIX_NSPACE} or \refattr{PMIX_JOBID} attributes of the \refterm{job} containing the application, plus its \refattr{PMIX_APPNUM} attribute, must to be included in the array when the array is \textit{not} included as part of a call to \refapi{PMIx_server_register_nspace} - i.e., when the job containing the application is ambiguous. The job identification is otherwise optional.
}
%
\declareAttribute{PMIX_PROC_INFO_ARRAY}{"pmix.pdata"}{pmix_data_array_t}{
Provide an array of \refstruct{pmix_info_t} containing process-realm information. The \refattr{PMIX_RANK} and \refattr{PMIX_NSPACE} attributes, or the \refattr{PMIX_PROCID} attribute, are required to be included in the array when the array is not included as part of a call to \refapi{PMIx_server_register_nspace} - i.e., when the job containing the process is ambiguous. All three may be included if desired. When the array is included in some broader structure that identifies the job, then only the \refattr{PMIX_RANK} or the \refattr{PMIX_PROCID} attribute must be included (the others are optional).
}
%
\declareAttribute{PMIX_NODE_INFO_ARRAY}{"pmix.node.arr"}{pmix_data_array_t}{
Provide an array of \refstruct{pmix_info_t} containing node-realm information. At a minimum, either the \refattr{PMIX_NODEID} or \refattr{PMIX_HOSTNAME} attribute is required to be included in the array, though both may be included.
}

Note that these assemblages can be used hierarchically:

\begin{itemize}
\item a \refattr{PMIX_JOB_INFO_ARRAY} might contain multiple \refattr{PMIX_APP_INFO_ARRAY} elements, each describing values for a specific application within the job.
\item a \refattr{PMIX_JOB_INFO_ARRAY} could contain a \refattr{PMIX_NODE_INFO_ARRAY} for each node hosting processes from that job, each array describing job-level values for that node.
\item a \refattr{PMIX_SESSION_INFO_ARRAY} might contain multiple \refattr{PMIX_JOB_INFO_ARRAY} elements, each describing a job executing within the session. Each job array could, in turn, contain both application and node arrays, thus providing a complete picture of the active operations within the allocation.
\end{itemize}

\adviceimplstart
\ac{PMIx} implementations must be capable of properly parsing and storing any hierarchical depth of information arrays. The resulting stored values are must to be accessible via both \refapi{PMIx_Get} and \refapi{PMIx_Query_info_nb} \acp{API}, assuming appropriate directives are provided by the caller.
\adviceimplend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Assembling the registration information}
\label{chap:api_server:assemble}

The following description is not intended to represent the actual layout of information in a given \ac{PMIx} library. Instead, it is describes how information provided in the \refarg{info} parameter of the \refapi{PMIx_server_register_nspace} shall be organized for proper processing by a \ac{PMIx} server library. The ordering of the various information elements is arbitrary - they are presented in a top-down hierarchical form solely for clarity in reading.

\advicermstart
Creating the \refarg{info} array of data requires knowing in advance the number of elements required for the array. This can be difficult to compute and somewhat fragile in practice. One method for resolving the problem is to create a linked list of objects, each containing a single \refstruct{pmix_info_t} structure. Allocation and manipulation of the list can then be accomplished using existing standard methods. Upon completion, the final \refarg{info} array can be allocated based on the number of elements on the list, and then the values in the list object \refstruct{pmix_info_t} structures transferred to the corresponding array element utilizing the \refapi{PMIx_Info_xfer} \ac{API}.
\advicermend

\label{cptr:api_server:noderegex}A common building block used in several areas is the construction of a regular expression identifying the nodes involved in that area - e.g., the nodes in a \refterm{session} or \refterm{job}. \ac{PMIx} provides several tools to facilitate this operation, beginning by constructing an argv-like array of node names. This array is then passed to the \refapi{PMIx_generate_regex} function to create a regular expression parseable by the \ac{PMIx} server library, as shown below:

\cspecificstart
\begin{codepar}
char **nodes = NULL;
char *nodelist;
char *regex;
size_t n;
pmix_status_t rc;
pmix_info_t info;

/* loop over an array of nodes, adding each
 * name to the array */
for (n=0; n < num_nodes; n++) \{
    /* filter the nodes to ignore those not included
     * in the target range (session, job, etc.). In
     * this example, all nodes are accepted */
    PMIX_ARGV_APPEND(&nodes, node[n]->name);
\}

/* join into a comma-delimited string */
nodelist = PMIX_ARGV_JOIN(nodes, ',');

/* release the array */
PMIX_ARGV_FREE(nodes);

/* generate regex */
rc = PMIx_generate_regex(nodelist, &regex);

/* release list */
free(nodelist);

/* pass the regex as the value to the PMIX_NODE_MAP key */
PMIx_Info_load(&info, PMIX_NODE_MAP, regex, PMIX_REGEX);
/* release the regex */
free(regex);
\end{codepar}
\cspecificend

Changing the filter criteria allows the construction of node maps for any level of information. A description of the returned regular expression is provided \hyperref[regex:fmt]{here}.

\label{cptr:api_server:ppnregex}A similar method is used to construct the map of processes on each node from the namespace being registered. This may be done for each information level of interest (e.g., to identify the process map for the entire \refterm{job} or for each \refterm{application} in the job) by changing the search criteria. An example is shown below for the case of creating the process map for a \refterm{job}:

\cspecificstart
\begin{codepar}
char **ndppn;
char rank[30];
char **ppnarray = NULL;
char *ppn;
char *localranks;
char *regex;
size_t n, m;
pmix_status_t rc;
pmix_info_t info;

/* loop over an array of nodes */
for (n=0; n < num_nodes; n++) \{
    /* for each node, construct an array of ranks on that node */
    ndppn = NULL;
    for (m=0; m < node[n]->num_procs; m++) \{
        /* ignore processes that are not part of the target job */
        if (!PMIX_CHECK_NSPACE(targetjob,node[n]->proc[m].nspace)) \{
            continue;
        \}
        snprintf(rank, 30, "%d", node[n]->proc[m].rank);
        PMIX_ARGV_APPEND(&ndppn, rank);
    \}
    /* convert the array into a comma-delimited string of ranks */
    localranks = PMIX_ARGV_JOIN(ndppn, ',');
    /* release the local array */
    PMIX_ARGV_FREE(ndppn);
    /* add this node's contribution to the overall array */
    PMIX_ARGV_APPEND(&ppnarray, localranks);
    /* release the local list */
    free(localranks);
\}

/* join into a semicolon-delimited string */
ppn = PMIX_ARGV_JOIN(ppnarray, ';');

/* release the array */
PMIX_ARGV_FREE(ppnarray);

/* generate ppn regex */
rc = PMIx_generate_ppn(ppn, &regex);

/* release list */
free(ppn);

/* pass the regex as the value to the PMIX_PROC_MAP key */
PMIx_Info_load(&info, PMIX_PROC_MAP, regex, PMIX_REGEX);
/* release the regex */
free(regex);
\end{codepar}
\cspecificend

Note that the \refattr{PMIX_NODE_MAP} and \refattr{PMIX_PROC_MAP} attributes are linked in that the order of entries in the process map must match the ordering of nodes in the node map - i.e., there is no provision in the \ac{PMIx} process map regular expression generator/parser pair supporting an out-of-order node or a node that has no corresponding process map entry (e.g., a node with no processes on it). Armed with these tools, the registration \refarg{info} array can be constructed as follows:

\begin{itemize}
\item Session-level information includes all session-specific values. In many cases, only two values (~\refattr{PMIX_SESSION_ID} and \refattr{PMIX_UNIV_SIZE}) are included in the registration array. Since both of these values are session-specific, they can be specified independently - i.e., in their own \refstruct{pmix_info_t} elements of the \refarg{info} array. Alternatively, they can be provided as a \refstruct{pmix_data_array_t} array of \refstruct{pmix_info_t} using the \refattr{PMIX_SESSION_INFO_ARRAY} attribute and identifed by including the \refattr{PMIX_SESSION_ID} attribute in the array - this is required in cases where non-specific attributes (e.g., \refattr{PMIX_NUM_NODES} or \refattr{PMIX_NODE_MAP}~) are passed to describe aspects of the session. Note that the node map can include nodes not used by the job being registered as no corresponding process map is specified.

The \refarg{info} array at this point might look like (where the labels identify the corresponding attribute - e.g., ``Session ID'' corresponds to the \refattr{PMIX_SESSION_ID} attribute):

\begingroup
\begin{figure*}[ht!]
  \begin{center}
    \includegraphics[clip,width=0.3\textwidth]{figs/sessioninfo.pdf}
  \end{center}
  \caption{Session-level information elements}
  \label{fig:sessioninfo}
\end{figure*}
\endgroup


\item Job-level information includes all job-specific values such as \refattr{PMIX_JOB_SIZE}, \refattr{PMIX_JOB_NUM_APPS}, and \refattr{PMIX_JOBID}. Since each invocation of \refapi{PMIx_server_register_nspace} describes a single \refterm{job}, job-specific values can be specified independently - i.e., in their own \refstruct{pmix_info_t} elements of the \refarg{info} array. Alternatively, they can be provided as a \refstruct{pmix_data_array_t} array of \refstruct{pmix_info_t} identified by the \refattr{PMIX_JOB_INFO_ARRAY} attribute - this is required in cases where non-specific attributes (e.g., \refattr{PMIX_NODE_MAP}) are passed to describe aspects of the job. Note that since the invocation only involves a single namespace, there is no need to include the \refattr{PMIX_NSPACE} attribute in the array.

Upon conclusion of this step, the \refarg{info} array might look like:

\begingroup
\begin{figure*}[ht!]
  \begin{center}
    \includegraphics[clip,width=0.4\textwidth]{figs/jobinfo.pdf}
  \end{center}
  \caption{Job-level information elements}
  \label{fig:jobinfo}
\end{figure*}
\endgroup

Note that in this example, \refattr{PMIX_NUM_NODES} is not required as that information is contained in the \refattr{PMIX_NODE_MAP} attribute. Similarly, \refattr{PMIX_JOB_SIZE} is not technically required as that information is contained in the \refattr{PMIX_PROC_MAP} when combined with the corresponding node map - however, there is no issue with including the job size as a separate entry.

The example also illustrates the hierarchical use of the \refattr{PMIX_NODE_INFO_ARRAY} attribute. In this case, we have chosen to pass several job-related values for each node - since those values are non-unique across the job, they must be passed in a node-info container. Note that the choice of what information to pass into the \ac{PMIx} server library versus what information to derive from other values at time of request is left to the host environment. \ac{PMIx} implementors in turn may, if they choose, pre-parse registration data to create expanded views (thus enabling faster response to requests at the expense of memory footprint) or to compress views into tighter representations (thus trading minimized footprint for longer response times).

\item Application-level information includes all application-specific values such as \refattr{PMIX_APP_SIZE} and \refattr{PMIX_APPLDR}. If the \refterm{job} contains only a single \refterm{application}, then the application-specific values can be specified independently - i.e., in their own \refstruct{pmix_info_t} elements of the \refarg{info} array - or as a \refstruct{pmix_data_array_t} array of \refstruct{pmix_info_t} using the \refattr{PMIX_APP_INFO_ARRAY} attribute and identifed by including the \refattr{PMIX_APPNUM} attribute in the array. Use of the array format is  must in cases where non-specific attributes (e.g., \refattr{PMIX_NODE_MAP}) are passed to describe aspects of the application.

However, in the case of a job consisting of multiple applications, all application-specific values for each application must be provided using the \refattr{PMIX_APP_INFO_ARRAY} format, each identified by its \refattr{PMIX_APPNUM} value.

Upon conclusion of this step, the \refarg{info} array might look like that shown in \ref{fig:appinfo}, assuming there are two applications in the job being registered:

\begingroup
\begin{figure*}[ht!]
  \begin{center}
    \includegraphics[clip,width=0.5\textwidth]{figs/appinfo.pdf}
  \end{center}
  \caption{Application-level information elements}
  \label{fig:appinfo}
\end{figure*}
\endgroup

\item Process-level information includes an entry for each process in the job being registered, each entry marked with the \refattr{PMIX_PROC_INFO_ARRAY} attribute. The \refterm{rank} of the process must be the first entry in the array - this provides efficiency when storing the data. Upon conclusion of this step, the \refarg{info} array might look like the diagram in \ref{fig:procinfo}:

\begingroup
\begin{figure*}[ht!]
  \begin{center}
    \includegraphics[clip,width=0.8\textwidth]{figs/procinfo.pdf}
  \end{center}
  \caption{Process-level information elements}
  \label{fig:procinfo}
\end{figure*}
\endgroup

\item For purposes of this example, node-level information only includes values describing the local node - i.e., it does not include information about other nodes in the job or session. In many cases, the values included in this level are unique to it and can be specified independently - i.e., in their own \refstruct{pmix_info_t} elements of the \refarg{info} array. Alternatively, they can be provided as a \refstruct{pmix_data_array_t} array of \refstruct{pmix_info_t} using the \refattr{PMIX_NODE_INFO_ARRAY} attribute - this is required in cases where non-specific attributes are passed to describe aspects of the node, or where values for multiple nodes are being provided.

The node-level information requires two elements that must be constructed in a manner similar to that used for the node map. The \refattr{PMIX_LOCAL_PEERS} value is computed based on the processes on the local node, filtered to select those from the job being registered, as shown below using the tools provided by \ac{PMIx}:

\cspecificstart
\begin{codepar}
char **ndppn = NULL;
char rank[30];
char *localranks;
size_t m;
pmix_info_t info;

for (m=0; m < mynode->num_procs; m++) \{
    /* ignore processes that are not part of the target job */
    if (!PMIX_CHECK_NSPACE(targetjob,mynode->proc[m].nspace)) \{
        continue;
    \}
    snprintf(rank, 30, "%d", mynode->proc[m].rank);
    PMIX_ARGV_APPEND(&ndppn, rank);
\}
/* convert the array into a comma-delimited string of ranks */
localranks = PMIX_ARGV_JOIN(ndppn, ',');
/* release the local array */
PMIX_ARGV_FREE(ndppn);

/* pass the string as the value to the PMIX_LOCAL_PEERS key */
PMIx_Info_load(&info, PMIX_LOCAL_PEERS, localranks, PMIX_STRING);

/* release the list */
free(localranks);
\end{codepar}
\cspecificend

The \refattr{PMIX_LOCAL_CPUSETS} value is constructed in a similar manner. In the provided example, it is assumed that an \ac{HWLOC} cpuset representation (a comma-delimited string of processor IDs) of the processors assigned to each process has previously been generated and stored on the process description. Thus, the value can be constructed as shown below:

\cspecificstart
\begin{codepar}
char **ndcpus = NULL;
char *localcpus;
size_t m;
pmix_info_t info;

for (m=0; m < mynode->num_procs; m++) \{
    /* ignore processes that are not part of the target job */
    if (!PMIX_CHECK_NSPACE(targetjob,mynode->proc[m].nspace)) \{
        continue;
    \}
    PMIX_ARGV_APPEND(&ndcpus, mynode->proc[m].cpuset);
\}
/* convert the array into a colon-delimited string */
localcpus = PMIX_ARGV_JOIN(ndcpus, ':');
/* release the local array */
PMIX_ARGV_FREE(ndcpus);

/* pass the string as the value to the PMIX_LOCAL_CPUSETS key */
PMIx_Info_load(&info, PMIX_LOCAL_CPUSETS, localcpus, PMIX_STRING);

/* release the list */
free(localcpus);
\end{codepar}
\cspecificend

Note that for efficiency, these two values can be computed at the same time.

\end{itemize}

The final \refarg{info} array might therefore look like the diagram in \ref{fig:nodeinfo}:

\begingroup
\begin{figure*}[ht!]
  \begin{center}
    \includegraphics[clip,width=0.8\textwidth]{figs/nodeinfo.pdf}
  \end{center}
  \caption{Final information array}
  \label{fig:nodeinfo}
\end{figure*}
\endgroup


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_deregister_nspace}}
\declareapi{PMIx_server_deregister_nspace}

%%%%
\summary

Deregister a namespace.

%%%%
\format

\copySignature{PMIx_server_deregister_nspace}{1.0}{
void PMIx_server_deregister_nspace(const pmix_nspace_t nspace, \\
\hspace*{24\sigspace}pmix_op_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{nspace}{Namespace (string)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t}. A \code{NULL} function reference indicates that the function is to be executed as a blocking operation. (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

%%%%
\descr

Deregister the specified \refarg{nspace} and purge all objects relating to it, including any client information from that namespace.
This is intended to support persistent \ac{PMIx} servers by providing an opportunity for the host \ac{RM} to tell the \ac{PMIx} server library to release all memory for a completed job. Note that the library must not invoke the callback function prior to returning from the \ac{API}, and that a \code{NULL} \refarg{cbfunc} reference indicates that the function is to be executed as a blocking operation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_register_resources}}
\declareapi{PMIx_server_register_resources}

%%%%
\summary

Register non-namespace related information with the local \ac{PMIx} server library.

%%%%
\format

\copySignature{PMIx_server_register_resources}{4.0}{
pmix_status_t \\
PMIx_server_register_resources(pmix_info_t info[], size_t ninfo, \\
\hspace*{31\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{31\sigspace}void *cbdata);
}

\begin{arglist}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t}. A \code{NULL} function reference indicates that the function is to be executed as a blocking operation (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

%%%%
\descr

Pass information about resources not associated with a given namespace to the \ac{PMIx} server library for distribution to local client processes. This includes information on fabric devices, \acp{GPU}, and other resources. All information provided through this \ac{API} shall be made available to each job as part of its job-level information. Duplicate information provided with the \refapi{PMIx_server_register_nspace} \ac{API} shall override any information provided by this function for that namespace, but only for that specific namespace.

\returnsimple

\advicermstart
Note that information passed in this manner could also have been included in a call to \refapi{PMIx_server_register_nspace} - e.g., as part of a \refattr{PMIX_NODE_INFO_ARRAY} array. This \ac{API} is provided as a logical alternative for code clarity, especially where multiple jobs may be supported by a single \ac{PMIx} server library instance, to avoid multiple registration of static resource information.

A \code{NULL} \refarg{cbfunc} reference indicates that the function is to be executed as a blocking operation.
\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_deregister_resources}}
\declareapi{PMIx_server_deregister_resources}

%%%%
\summary

Remove specified non-namespace related information from the local \ac{PMIx} server library.

%%%%
\format

\copySignature{PMIx_server_deregister_resources}{4.0}{
pmix_status_t \\
PMIx_server_deregister_resources(pmix_info_t info[], size_t ninfo, \\
\hspace*{33\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{33\sigspace}void *cbdata);
}

\begin{arglist}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t}. A \code{NULL} function reference indicates that the function is to be executed as a blocking operation (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

%%%%
\descr

Remove information about resources not associated with a given namespace from the \ac{PMIx} server library. Only the \refarg{key} fields of the provided \refarg{info} array shall be used for the operation - the associated values shall be ignored except where they serve as qualifiers to the request. For example, to remove a specific fabric device from a given node, the \refarg{info} array might include a \refattr{PMIX_NODE_INFO_ARRAY} containing the \refattr{PMIX_NODEID} or \refattr{PMIX_HOSTNAME} identifying the node hosting the device, and the \refattr{PMIX_FABRIC_DEVICE_NAME} specifying the device to be removed. Alternatively, the device could be removed using only the \refattr{PMIX_DEVICE_ID} as this is unique across the overall system.

\returnsimple

\advicermstart
As information not related to namespaces is considered \emph{static}, there is no requirement that the host environment deregister resources prior to finalizing the \ac{PMIx} server library. The server library shall properly cleanup as part of its normal finalize operations. Deregistration of resources is only required, therefore, when the host environment determines that client processes should no longer have access to that information.

A \code{NULL} \refarg{cbfunc} reference indicates that the function is to be executed as a blocking operation.
% TODO: If this is one of those functions that can be blocking or non-blocking, then
% it needs a full return code explanation, not the returnsimple 

\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_register_client}}
\declareapi{PMIx_server_register_client}

%%%%
\summary

Register a client process with the PMIx server library.

%%%%
\format

\copySignature{PMIx_server_register_client}{1.0}{
pmix_status_t \\
PMIx_server_register_client(const pmix_proc_t *proc, \\
\hspace*{24\sigspace}uid_t uid, gid_t gid, \\
\hspace*{24\sigspace}void *server_object, \\
\hspace*{24\sigspace}pmix_op_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure (handle)}
\argin{uid}{user id (integer)}
\argin{gid}{group id (integer)}
\argin{server_object}{(memory reference)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t}. A \code{NULL} function reference indicates that the function is to be executed as a blocking operation (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\returnsimplenb

\returnstart
\begin{itemize}
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
\end{itemize}
\returnend

%%%%
\descr

Register a client process with the PMIx server library.

The host server can also, if it desires, provide an object it wishes to be returned when a server function is called that relates to a specific process.
For example, the host server may have an object that tracks the specific client.
Passing the object to the library allows the library to provide that object to the host server during subsequent calls related to that client, such as a \refapi{pmix_server_client_connected2_fn_t} function.  This allows the host server to access the object without performing a lookup based on the client's namespace and rank.

\advicermstart
Host environments are required to execute this operation prior to starting the client process.
The expected user ID and group ID of the child process allows the server library to properly authenticate clients as they connect by requiring the two values to match. Accordingly, the detected user and group ID's of the connecting process are not included in the \refapi{pmix_server_client_connected2_fn_t} server module function.
\advicermend

\adviceimplstart
For security purposes, the \ac{PMIx} server library should check the user and group ID's of a connecting process against those provided for the declared client process identifier via the \refapi{PMIx_server_register_client} prior to completing the connection.
\adviceimplend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_deregister_client}}
\declareapi{PMIx_server_deregister_client}

%%%%
\summary

Deregister a client and purge all data relating to it.

%%%%
\format

\copySignature{PMIx_server_deregister_client}{1.0}{
void \\
PMIx_server_deregister_client(const pmix_proc_t *proc, \\
\hspace*{24\sigspace}pmix_op_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure (handle)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t}. A \code{NULL} function reference indicates that the function is to be executed as a blocking operation (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}


%%%%
\descr

The \refapi{PMIx_server_deregister_nspace} \ac{API} will delete all client information for that namespace. The \ac{PMIx} server library will automatically perform that operation upon disconnect of all local clients.
This \ac{API} is therefore intended primarily for use in exception cases, but can be called in non-exception cases if desired. Note that the library must not invoke the callback function prior to returning from the \ac{API}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_setup_fork}}
\declareapi{PMIx_server_setup_fork}

%%%%
\summary

Setup the environment of a child process to be forked by the host.

%%%%
\format

\copySignature{PMIx_server_setup_fork}{1.0}{
pmix_status_t \\
PMIx_server_setup_fork(const pmix_proc_t *proc, \\
\hspace*{24\sigspace}char ***env);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure (handle)}
\argin{env}{Environment array (array of strings)}
\end{arglist}

\returnsimple

%%%%
\descr

Setup the environment of a child process to be forked by the host so it can correctly interact with the PMIx server.

The \ac{PMIx} client needs some setup information so it can properly connect back to the server.
This function will set appropriate environmental variables for this purpose, and will also provide any environmental variables that were specified in the launch command (e.g., via \refapi{PMIx_Spawn}) plus other values (e.g., variables required to properly initialize the client's fabric library).

\advicermstart
Host environments are required to execute this operation prior to starting the client process.
\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_dmodex_request}}
\declareapi{PMIx_server_dmodex_request}

%%%%
\summary

Define a function by which the host server can request modex data from the local PMIx server.

%%%%
\format

\copySignature{PMIx_server_dmodex_request}{1.0}{
pmix_status_t \\
PMIx_server_dmodex_request(const pmix_proc_t *proc, \\
\hspace*{27\sigspace}pmix_dmodex_response_fn_t cbfunc, \\
\hspace*{27\sigspace}void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure (handle)}
\argin{cbfunc}{Callback function \refapi{pmix_dmodex_response_fn_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\returnsimplenb

%%%%
\descr

Define a function by which the host server can request modex data from the local \ac{PMIx} server. Traditional wireup procedures revolve around the per-process posting of data (e.g., location and endpoint information) via the \refapi{PMIx_Put} and \refapi{PMIx_Commit} functions followed by a \refapi{PMIx_Fence} barrier that globally exchanges the posted information. However, the barrier operation represents a signficant time impact at large scale.

\ac{PMIx} supports an alternative wireup method known as \declareterm{Direct Modex} that replaces the barrier-based exchange of all process-posted information with on-demand fetch of a peer's data. In place of the barrier operation, data posted by each process is cached on the local \ac{PMIx} server. When a process requests the information posted by a particular peer, it first checks the local cache to see if the data is already available. If not, then the request is passed to the local \ac{PMIx} server, which subsequently requests that its \ac{RM} host request the data from the \ac{RM} daemon on the node where the specified peer process is located. Upon receiving the request, the \ac{RM} daemon passes the request into its \ac{PMIx} server library using the \refapi{PMIx_server_dmodex_request} function, receiving the response in the provided \refarg{cbfunc} once the indicated process has posted its information. The \ac{RM} daemon then returns the data to the requesting daemon, who subsequently passes the data to its \ac{PMIx} server library for transfer to the requesting client.

\adviceuserstart
While direct modex allows for faster launch times by eliminating the barrier operation, per-peer retrieval of posted information is less efficient. Optimizations can be implemented - e.g., by returning posted information from all processes on a node upon first request - but in general direct modex remains best suited for sparsely connected applications.
\adviceuserend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Server Direct Modex Response Callback Function}
\declareapi{pmix_dmodex_response_fn_t}

The \refapi{PMIx_server_dmodex_request} callback function.

%%%%
\summary

Provide a function by which the local \ac{PMIx} server library can return connection and other data posted by local application processes to the host resource manager.

%%%%
\format

\copySignature{pmix_dmodex_response_fn_t}{1.0}{
typedef void (*pmix_dmodex_response_fn_t)( \\
\hspace*{20\sigspace}pmix_status_t status, \\
\hspace*{20\sigspace}char *data, size_t sz, \\
\hspace*{20\sigspace}void *cbdata);
}

\begin{arglist}
\argin{status}{Returned status of the request (\refstruct{pmix_status_t})}
\argin{data}{Pointer to a data "blob" containing the requested information (handle)}
\argin{sz}{Number of bytes in the \refarg{data} blob (integer)}
\argin{cbdata}{Data passed into the initial call to \refapi{PMIx_server_dmodex_request} (memory reference)}
\end{arglist}

\descr
Define a function to be called by the PMIx server library for return of information posted by a local application process (via \refapi{PMIx_Put} with subsequent \refapi{PMIx_Commit}) in response to a request from the host RM. The returned \refarg{data} blob is owned by the PMIx server library and will be free’d upon return from the function.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_setup_application}}
\declareapi{PMIx_server_setup_application}

%%%%
\summary

Provide a function by which a launcher can request application-specific setup data prior to launch of a \refterm{job}.

%%%%
\format

\copySignature{PMIx_server_setup_application}{2.0}{
pmix_status_t \\
PMIx_server_setup_application(const pmix_nspace_t nspace, \\
\hspace*{24\sigspace}pmix_info_t info[], size_t ninfo, \\
\hspace*{24\sigspace}pmix_setup_application_cbfunc_t cbfunc, \\
\hspace*{24\sigspace}void *cbdata);
}

\begin{arglist}
\argin{nspace}{namespace (string)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_setup_application_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the \refarg{cbfunc} callback function (memory reference)}
\end{arglist}

\returnsimplenb

\reqattrstart
\ac{PMIx} libraries that support this operation are required to support the following:

\pasteAttributeItem{PMIX_SETUP_APP_ENVARS}
\pasteAttributeItem{PMIX_SETUP_APP_NONENVARS}
\pasteAttributeItem{PMIX_SETUP_APP_ALL}
\pasteAttributeItem{PMIX_ALLOC_FABRIC}
\pasteAttributeItem{PMIX_ALLOC_FABRIC_ID}
\pasteAttributeItem{PMIX_ALLOC_FABRIC_SEC_KEY}
\pasteAttributeItem{PMIX_ALLOC_FABRIC_TYPE}
\pasteAttributeItem{PMIX_ALLOC_FABRIC_PLANE}
\pasteAttributeItem{PMIX_ALLOC_FABRIC_ENDPTS}
\pasteAttributeItem{PMIX_ALLOC_FABRIC_ENDPTS_NODE}
\pasteAttributeItem{PMIX_PROC_MAP}
\pasteAttributeItem{PMIX_NODE_MAP}

\reqattrend

\optattrstart
\ac{PMIx} libraries that support this operation may support the following:

\pasteAttributeItem{PMIX_ALLOC_BANDWIDTH}
\pasteAttributeItem{PMIX_ALLOC_FABRIC_QOS}
\pasteAttributeItemBegin{PMIX_SESSION_INFO}In this context, indicates that the information provided in the \refattr{PMIX_NODE_MAP} is for the entire session and not just the indicated namespace. Thus, subsequent calls to this \ac{API} may omit node-level information - e.g., the library may not need to include information on the devices on each node in a subsequent call.
\pasteAttributeItemEnd

The following optional attributes may be provided by the host environment to identify the programming model (as specified by the user) being executed within the application. The \ac{PMIx} server library may utilize this information to harvest/forward model-specific environmental variables, record the programming model associated with the application, etc.

\begin{itemize}
    \item \pasteAttributeItem{PMIX_PROGRAMMING_MODEL}
    \item \pasteAttributeItem{PMIX_MODEL_LIBRARY_NAME}
    \item \pasteAttributeItem{PMIX_MODEL_LIBRARY_VERSION}
\end{itemize}

\optattrend

%%%%
\descr

Provide a function by which the \ac{RM} can request application-specific setup data (e.g., environmental variables, fabric configuration and security credentials) from supporting \ac{PMIx} server library subsystems prior to initiating launch of a job.

This is defined as a non-blocking operation in case contributing subsystems need to perform some potentially time consuming action (e.g., query a remote service) before responding. The returned data must be distributed by the host environment and subsequently delivered to the local \ac{PMIx} server on each node where application processes will execute, prior to initiating execution of those processes.

\advicermstart
Host environments are required to execute this operation prior to launching a job. In addition to supported directives, the \refarg{info} array must include a description of the \refterm{job} using the \refattr{PMIX_NODE_MAP} and \refattr{PMIX_PROC_MAP} attributes.

Note that the function can be called on a per-application basis if the \refattr{PMIX_PROC_MAP} and \refattr{PMIX_NODE_MAP} are provided only for the corresponding application (as opposed to the entire job) each time.
\advicermend

\adviceimplstart
Support for harvesting of environmental variables and providing of local configuration information by the \ac{PMIx} implementation is optional.
\adviceimplend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Server Setup Application Callback Function}
\declareapi{pmix_setup_application_cbfunc_t}

The \refapi{PMIx_server_setup_application} callback function.

%%%%
\summary

Provide a function by which the resource manager can receive application-specific environmental variables and other setup data prior to launch of an application.

%%%%
\format

\copySignature{pmix_setup_application_cbfunc_t}{2.0}{
typedef void (*pmix_setup_application_cbfunc_t)( \\
\hspace*{24\sigspace}pmix_status_t status, \\
\hspace*{24\sigspace}pmix_info_t info[], size_t ninfo, \\
\hspace*{24\sigspace}void *provided_cbdata, \\
\hspace*{24\sigspace}pmix_op_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{status}{returned status of the request (\refstruct{pmix_status_t})}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{provided_cbdata}{Data originally passed to call to \refapi{PMIx_server_setup_application} (memory reference)}
\argin{cbfunc}{\refapi{pmix_op_cbfunc_t} function to be called when processing completed (function reference)}
\argin{cbdata}{Data to be passed to the \refarg{cbfunc} callback function (memory reference)}
\end{arglist}

\descr

Define a function to be called by the \ac{PMIx} server library for return of application-specific setup data in response to a request from the host \ac{RM}. The returned \refarg{info} array is owned by the \ac{PMIx} server library and will be free'd when the provided \refarg{cbfunc} is called.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Server Setup Application Attributes}
\label{api:struct:attributes:security}

\versionMarker{3.0}
Attributes specifically defined for controlling contents of application setup data.

%
\declareAttribute{PMIX_SETUP_APP_ENVARS}{"pmix.setup.env"}{bool}{
Harvest and include relevant environmental variables.
}
%
\declareAttribute{PMIX_SETUP_APP_NONENVARS}{""pmix.setup.nenv"}{bool}{
Include all relevant data other than environmental variables.
}
%
\declareAttribute{PMIX_SETUP_APP_ALL}{"pmix.setup.all"}{bool}{
Include all relevant data.
}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_Register_attributes}}
\declareapi{PMIx_Register_attributes}

%%%%
\summary

Register host environment attribute support for a function.

%%%%
\format

\copySignature{PMIx_Register_attributes}{4.0}{
pmix_status_t \\
PMIx_Register_attributes(const char *function, \\
\hspace*{25\sigspace}pmix_regattr_t attrs[], \\
\hspace*{25\sigspace}size_t nattrs);
}

\begin{arglist}
\argin{function}{String name of function (string)}
\argin{attrs}{Array of \refstruct{pmix_regattr_t} describing the supported attributes (handle)}
\argin{nattrs}{Number of elements in \refarg{attrs} (\code{size_t})}
\end{arglist}

\returnsimple

%%%%
\descr

The \code{PMIx_Register_attributes} function is used by the host environment to register with its \ac{PMIx} server library the attributes it supports for each \refapi{pmix_server_module_t} function. The \refarg{function} is the string name of the server module function (e.g., "register_events", "validate_credential", or "allocate") whose attributes are being registered. See the \refstruct{pmix_regattr_t} entry for a description of the \refarg{attrs} array elements.

Note that the host environment can also query the library (using the \refapi{PMIx_Query_info_nb} \ac{API}) for its attribute support both at the server, client, and tool levels once the host has executed \refapi{PMIx_server_init} since the server will internally register those values.

\advicermstart
Host environments are strongly encouraged to register all supported attributes immediately after initializing the library to ensure that user requests are correctly serviced.
\advicermend

\adviceimplstart
\ac{PMIx} implementations are \emph{required} to register all internally supported attributes for each \ac{API} during initialization of the library (i.e., when the process calls their respective \ac{PMIx} init function). Specifically, the implementation \emph{must not} register supported attributes upon first call to a given \ac{API} as this would prevent users from discovering supported attributes prior to first use of an \ac{API}.

It is the implementation's responsibility to associate registered attributes for a given \refapi{pmix_server_module_t} function with their corresponding user-facing \ac{API}. Supported attributes \emph{must} be reported to users in terms of their support for user-facing \acp{API}, broken down by the level (see Section \ref{api:struct:attributes:query}) at which the attribute is supported.

Note that attributes can/will be registered on an \ac{API} for each level. It is \emph{required} that the implementation support user queries for supported attributes on a per-level basis. Duplicate registrations at the \emph{same} level for a function \emph{shall} return an error - however, duplicate registrations at \emph{different} levels \emph{shall} be independently tracked.
\adviceimplend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Attribute registration constants}

Constants supporting attribute registration.

\begin{constantdesc}
%
\declareconstitemvalue{PMIX_ERR_REPEAT_ATTR_REGISTRATION}{-171}
The attributes for an identical function have already been registered at the specified level (host, server, or client).
%
\end{constantdesc}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Attribute registration structure}
\declarestruct{pmix_regattr_t}

The \refstruct{pmix_regattr_t} structure is used to register attribute support for a \ac{PMIx} function.

\copySignature{pmix_regattr_t}{4.0}{
typedef struct pmix_regattr \{ \\
\hspace*{4\sigspace}char *name; \\
\hspace*{4\sigspace}pmix_key_t *string; \\
\hspace*{4\sigspace}pmix_data_type_t type; \\
\hspace*{4\sigspace}pmix_info_t *info; \\
\hspace*{4\sigspace}size_t ninfo; \\
\hspace*{4\sigspace}char **description; \\
\} pmix_regattr_t;;
}

Note that in this structure:

\begin{itemize}
    \item the \refarg{name} is the actual name of the attribute - e.g., "PMIX_MAX_PROCS"
    \item the \refarg{string} is the literal string value of the attribute - e.g., "pmix.max.size" for the \refattr{PMIX_MAX_PROCS} attribute
    \item \refarg{type} must be a \ac{PMIx} data type identifying the type of data associated with this attribute.
    \item the \refarg{info} array contains machine-usable information regarding the range of accepted values. This may include entries for \refattr{PMIX_MIN_VALUE}, \refattr{PMIX_MAX_VALUE}, \refattr{PMIX_ENUM_VALUE}, or a combination of them. For example, an attribute that supports all positive integers might delineate it by including a \refstruct{pmix_info_t} with a key of \refattr{PMIX_MIN_VALUE}, type of \refconst{PMIX_INT}, and value of zero. The lack of an entry for \refattr{PMIX_MAX_VALUE} indicates that there is no ceiling to the range of accepted values.
    \item \refarg{ninfo} indicates the number of elements in the \refarg{info} array
    \item The \refarg{description} field consists of a \code{NULL}-terminated array of strings describing the attribute, optionally including a human-readable description of the range of accepted values - e.g., "ALL POSITIVE INTEGERS", or a comma-delimited list of enum value names. No correlation between the number of entries in the \refarg{description} and the number of elements in the \refarg{info} array is implied or required.
\end{itemize}

The attribute \refarg{name} and \refarg{string} fields must be \code{NULL}-terminated strings composed of standard alphanumeric values supported by common utilities such as \textit{strcmp}.

Although not strictly required, both \ac{PMIx} library implementers and host environments are strongly encouraged to provide both human-readable and machine-parsable descriptions of supported attributes when registering them.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Attribute registration structure descriptive attributes}
\label{api:struct:attributes:descr}

The following attributes relate to the nature of the values being reported
in the \refstruct{pmix_regattr_t} structures.

\declareAttribute{PMIX_MAX_VALUE}{"pmix.descr.maxval"}{varies}{
Used in \refstruct{pmix_regattr_t} to describe the maximum valid value for the associated attribute.
}
%
\declareAttribute{PMIX_MIN_VALUE}{"pmix.descr.minval"}{varies}{
Used in \refstruct{pmix_regattr_t} to describe the minimum valid value for the associated attribute.
}
%
\declareAttribute{PMIX_ENUM_VALUE}{"pmix.descr.enum"}{char*}{
Used in \refstruct{pmix_regattr_t} to describe accepted values for the associated attribute. Numerical values shall be presented in a form convertible to the attribute's declared data type. Named values (i.e., values defined by constant names via a typical C-language enum declaration) must be provided as their numerical equivalent.
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Attribute registration structure support macros}

The following macros are provided to support the \refstruct{pmix_regattr_t} structure.

%%%%
\littleheader{Static initializer for the regattr structure}
\declaremacroProvisional{PMIX_REGATTR_STATIC_INIT}

Provide a static initializer for the \refstruct{pmix_regattr_t} fields.

\versionMarker{5.0}
\cspecificstart
\begin{codepar}
PMIX_REGATTR_STATIC_INIT
\end{codepar}
\cspecificend


%%%%%%%%%%%
\littleheader{Initialize the regattr structure}
\declaremacro{PMIX_REGATTR_CONSTRUCT}

Initialize the \refstruct{pmix_regattr_t} fields

\copySignature{PMIX_REGATTR_CONSTRUCT}{4.0}{
PMIX_REGATTR_CONSTRUCT(m)
}

\begin{arglist}
\argin{m}{Pointer to the structure to be initialized (pointer to \refstruct{pmix_regattr_t})}
\end{arglist}

%%%%%%%%%%%
\littleheader{Destruct the regattr structure}
\declaremacro{PMIX_REGATTR_DESTRUCT}

Destruct the \refstruct{pmix_regattr_t} fields, releasing all strings.

\copySignature{PMIX_REGATTR_DESTRUCT}{4.0}{
PMIX_REGATTR_DESTRUCT(m)
}

\begin{arglist}
\argin{m}{Pointer to the structure to be destructed (pointer to \refstruct{pmix_regattr_t})}
\end{arglist}

%%%%%%%%%%%
\littleheader{Create a regattr array}
\declaremacro{PMIX_REGATTR_CREATE}

Allocate and initialize an array of \refstruct{pmix_regattr_t} structures.

\copySignature{PMIX_REGATTR_CREATE}{4.0}{
PMIX_REGATTR_CREATE(m, n)
}

\begin{arglist}
\arginout{m}{Address where the pointer to the array of \refstruct{pmix_regattr_t} structures shall be stored (handle)}
\argin{n}{Number of structures to be allocated (\code{size_t})}
\end{arglist}

%%%%%%%%%%%
\littleheader{Free a regattr array}
\declaremacro{PMIX_REGATTR_FREE}

Release an array of \refstruct{pmix_regattr_t} structures.

\copySignature{PMIX_REGATTR_FREE}{4.0}{
PMIX_REGATTR_FREE(m, n)
}

\begin{arglist}
\arginout{m}{Pointer to the array of \refstruct{pmix_regattr_t} structures (handle)}
\argin{n}{Number of structures in the array (\code{size_t})}
\end{arglist}

%%%%%%%%%%%
\littleheader{Load a regattr structure}
\declaremacro{PMIX_REGATTR_LOAD}

Load values into a \refstruct{pmix_regattr_t} structure. The macro can be called multiple times to add as many strings as desired to the same structure by passing the same address and a \code{NULL} key to the macro. Note that the \refarg{t} type value must be given each time.

\copySignature{PMIX_REGATTR_LOAD}{4.0}{
PMIX_REGATTR_LOAD(a, n, k, t, ni, v)
}

\begin{arglist}
\argin{a}{Pointer to the structure to be loaded (pointer to \refstruct{pmix_proc_t})}
\argin{n}{String name of the attribute (string)}
\argin{k}{Key value to be loaded (\refstruct{pmix_key_t})}
\argin{t}{Type of data associated with the provided key (\refstruct{pmix_data_type_t})}
\argin{ni}{Number of \refstruct{pmix_info_t} elements to be allocated in \refarg{info} (\code{size_t})}
\argin{v}{One-line description to be loaded (more can be added separately) (string)}
\end{arglist}

%%%%%%%%%%%
\littleheader{Transfer a regattr to another regattr}
\declaremacro{PMIX_REGATTR_XFER}

Non-destructively transfer the contents of a \refstruct{pmix_regattr_t} structure to another one.

\copySignature{PMIX_REGATTR_XFER}{4.0}{
PMIX_REGATTR_XFER(m, n)
}

\begin{arglist}
\arginout{m}{Pointer to the destination \refstruct{pmix_regattr_t} structure (handle)}
\argin{m}{Pointer to the source \refstruct{pmix_regattr_t} structure (handle)}
\end{arglist}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_setup_local_support}}
\declareapi{PMIx_server_setup_local_support}

%%%%
\summary

Provide a function by which the local \ac{PMIx} server can perform any application-specific operations prior to spawning local clients of a given application.

%%%%
\format

\copySignature{PMIx_server_setup_local_support}{2.0}{
pmix_status_t \\
PMIx_server_setup_local_support(const pmix_nspace_t nspace, \\
\hspace*{32\sigspace}pmix_info_t info[], size_t ninfo, \\
\hspace*{32\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{32\sigspace}void *cbdata);
}

\begin{arglist}
\argin{nspace}{Namespace (string)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (\code{size_t})}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t}. A \code{NULL} function reference indicates that the function is to be executed as a blocking operation (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\returnsimplenb

\returnstart
\begin{itemize}
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
\end{itemize}
\returnend

%%%%
\descr

Provide a function by which the local \ac{PMIx} server can perform any application-specific operations prior to spawning local clients of a given application. For example, a fabric library might need to setup the local driver for ``instant on'' addressing. The data provided in the \refarg{info} array is the data returned to the host \ac{RM} by the callback function executed as a result of a call to \refapi{PMIx_server_setup_application}.

\advicermstart
Host environments are required to execute this operation prior to starting any local application processes from the specified namespace if information was obtained from a call to \refapi{PMIx_server_setup_application}.

Host environments must register the \refarg{nspace} using \refapi{PMIx_server_register_nspace} prior to calling this \ac{API} to ensure that all namespace-related information required to support this function is available to the library. This eliminates the need to include any of the registration information in the \refarg{info} array passed to this \ac{API}.
\advicermend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_IOF_deliver}}
\declareapi{PMIx_server_IOF_deliver}

%%%%
\summary

Provide a function by which the host environment can pass forwarded \ac{IO} to the \ac{PMIx} server library for distribution to its clients.

%%%%
\format

\copySignature{PMIx_server_IOF_deliver}{3.0}{
pmix_status_t \\
PMIx_server_IOF_deliver(const pmix_proc_t *source, \\
\hspace*{24\sigspace}pmix_iof_channel_t channel, \\
\hspace*{24\sigspace}const pmix_byte_object_t *bo, \\
\hspace*{24\sigspace}const pmix_info_t info[], size_t ninfo, \\
\hspace*{24\sigspace}pmix_op_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{source}{Pointer to \refstruct{pmix_proc_t} identifying source of the \ac{IO} (handle)}
\argin{channel}{\ac{IO} channel of the data (\refstruct{pmix_iof_channel_t})}
\argin{bo}{Pointer to \refstruct{pmix_byte_object_t} containing the payload to be delivered (handle)}
\argin{info}{Array of \refstruct{pmix_info_t} metadata describing the data (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (\code{size_t})}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t}. A \code{NULL} function reference indicates that the function is to be executed as a blocking operation (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\returnsimplenb

\returnstart
\begin{itemize}
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
\end{itemize}
\returnend

%%%%
\descr

Provide a function by which the host environment can pass forwarded \ac{IO} to the \ac{PMIx} server library for distribution to its clients. The \ac{PMIx} server library is responsible for determining which of its clients have actually registered for the provided data and delivering it. The \refarg{cbfunc} callback function will be called once the \ac{PMIx} server library no longer requires access to the provided data.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_collect_inventory}}
\declareapi{PMIx_server_collect_inventory}

%%%%
\summary

Collect inventory of resources on a node.

%%%%
\format

\copySignature{PMIx_server_collect_inventory}{3.0}{
pmix_status_t \\
PMIx_server_collect_inventory(const pmix_info_t directives[], \\
\hspace*{30\sigspace}size_t ndirs, \\
\hspace*{30\sigspace}pmix_info_cbfunc_t cbfunc, \\
\hspace*{30\sigspace}void *cbdata);
}

\begin{arglist}
\argin{directives}{Array of \refstruct{pmix_info_t} directing the request (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\argin{cbfunc}{Callback function to return collected data (\refapi{pmix_info_cbfunc_t} function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\returnsimplenb

%%%%
\descr

Provide a function by which the host environment can request its \ac{PMIx} server library collect an inventory of local resources. Supported resources depends upon the \ac{PMIx} implementation, but may include the local node topology and fabric interfaces.

\advicermstart
This is a non-blocking \ac{API} as it may involve somewhat lengthy operations to obtain the requested information. Inventory collection is expected to be a rare event – at system startup and upon command from a system administrator. Inventory updates are expected to initiate a smaller operation involving only the changed information. For example, replacement of a node would generate an event to notify the scheduler with an inventory update without invoking a global inventory operation.
\advicermend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_deliver_inventory}}
\declareapi{PMIx_server_deliver_inventory}

%%%%
\summary

Pass collected inventory to the \ac{PMIx} server library for storage.

%%%%
\format

\copySignature{PMIx_server_deliver_inventory}{3.0}{
pmix_status_t \\
PMIx_server_deliver_inventory(const pmix_info_t info[], \\
\hspace*{30\sigspace}size_t ninfo, \\
\hspace*{30\sigspace}const pmix_info_t directives[], \\
\hspace*{30\sigspace}size_t ndirs, \\
\hspace*{30\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{30\sigspace}void *cbdata);
}

\begin{arglist}
\argin{info}{Array of \refstruct{pmix_info_t} containing the inventory (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (\code{size_t})}
\argin{directives}{Array of \refstruct{pmix_info_t} directing the request (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (\code{size_t})}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t}. A \code{NULL} function reference indicates that the function is to be executed as a blocking operation (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\returnsimplenb

\returnstart
\begin{itemize}
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
\end{itemize}
\returnend


%%%%
\descr

Provide a function by which the host environment can pass inventory information obtained from a node (as a result of a call to \refapi{PMIx_server_collect_inventory}) to the \ac{PMIx} server library for storage. Inventory data is subsequently used by the \ac{PMIx} server library for allocations in response to \refapi{PMIx_server_setup_application}, and may be available to the library's host via the \refapi{PMIx_Get} \ac{API} (depending upon \ac{PMIx} implementation). The \refarg{cbfunc} callback function will be called once the \ac{PMIx} server library no longer requires access to the provided data.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_generate_locality_string}}
\declareapi{PMIx_server_generate_locality_string}

%%%%
\summary

Generate a \ac{PMIx} locality string from a given cpuset.

%%%%
\format

\copySignature{PMIx_server_generate_locality_string}{4.0}{
pmix_status_t \\
PMIx_server_generate_locality_string(const pmix_cpuset_t *cpuset, \\
\hspace*{37\sigspace}char **locality);
}

\begin{arglist}
\argin{cpuset}{Pointer to a \refstruct{pmix_cpuset_t} containing the bitmap of assigned \acp{PU} (handle)}
\argout{locality}{String representation of the \ac{PMIx} locality corresponding to the input bitmap (\code{char*})}
\end{arglist}

A successful return indicates that the returned string contains the generated locality string.

\returnsimple

%%%%
\descr

Provide a function by which the host environment can generate a \ac{PMIx} locality string for inclusion in the call to \refapi{PMIx_server_register_nspace}. This function shall only be called for local client processes, with the returned locality included in the job-level information (via the \refattr{PMIX_LOCALITY_STRING} attribute) provided to local clients. Local clients can use these strings as input to determine the relative locality of their local peers via the \refapi{PMIx_Get_relative_locality} \ac{API}.

The function is required to return a string prefixed by the \refarg{source} field of the provided \refarg{cpuset} followed by a colon. The remainder of the string shall represent the corresponding locality as expressed by the underlying implementation.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_generate_cpuset_string}}
\declareapi{PMIx_server_generate_cpuset_string}

%%%%
\summary

Generate a \ac{PMIx} string representation of the provided cpuset.

%%%%
\format

\copySignature{PMIx_server_generate_cpuset_string}{4.0}{
pmix_status_t \\
PMIx_server_generate_cpuset_string(const pmix_cpuset_t *cpuset, \\
\hspace*{35\sigspace}char **cpuset_string);
}

\begin{arglist}
\argin{cpuset}{Pointer to a \refstruct{pmix_cpuset_t} containing the bitmap of assigned \acp{PU} (handle)}
\argout{cpuset_string}{String representation of the input bitmap (\code{char*})}
\end{arglist}

A successful return indicates that the returned string contains the generated cpuset representation string.

\returnsimple

%%%%
\descr

Provide a function by which the host environment can generate a string representation of the cpuset bitmap for inclusion in the call to \refapi{PMIx_server_register_nspace}. This function shall only be called for local client processes, with the returned string included in the job-level information (via the \refattr{PMIX_CPUSET} attribute) provided to local clients. Local clients can use these strings as input to obtain their \ac{PU} bindings via the \refapi{PMIx_Parse_cpuset_string} \ac{API}.

The function is required to return a string prefixed by the \refarg{source} field of the provided \refarg{cpuset} followed by a colon. The remainder of the string shall represent the \acp{PU} to which the process is bound as expressed by the underlying implementation.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Cpuset Structure}
\declarestruct{pmix_cpuset_t}

The \refstruct{pmix_cpuset_t} structure contains a character string identifying the source of the bitmap (e.g., "hwloc") and a pointer to the corresponding implementation-specific structure (e.g., \code{hwloc_cpuset_t}).

\copySignature{pmix_cpuset_t}{4.0}{
typedef struct pmix_cpuset \{ \\
\hspace*{4\sigspace}char *source; \\
\hspace*{4\sigspace}void *bitmap; \\
\} pmix_cpuset_t;
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Cpuset support macros}
The following macros support the \refstruct{pmix_cpuset_t} structure.

%%%%
\littleheader{Static initializer for the cpuset structure}
\declaremacroProvisional{PMIX_CPUSET_STATIC_INIT}

Provide a static initializer for the \refstruct{pmix_cpuset_t} fields.

\versionMarker{5.0}
\cspecificstart
\begin{codepar}
PMIX_CPUSET_STATIC_INIT
\end{codepar}
\cspecificend


\littleheader{Initialize the cpuset structure}
\declaremacro{PMIX_CPUSET_CONSTRUCT}

Initialize the \refstruct{pmix_cpuset_t} fields.

\copySignature{PMIX_CPUSET_CONSTRUCT}{4.0}{
PMIX_CPUSET_CONSTRUCT(m)
}

\begin{arglist}
\argin{m}{Pointer to the structure to be initialized (pointer to \refstruct{pmix_cpuset_t})}
\end{arglist}

\littleheader{Destruct the cpuset structure}
\declaremacro{PMIX_CPUSET_DESTRUCT}

Destruct the \refstruct{pmix_cpuset_t} fields.

\copySignature{PMIX_CPUSET_DESTRUCT}{4.0}{
PMIX_CPUSET_DESTRUCT(m)
}

\begin{arglist}
\argin{m}{Pointer to the structure to be destructed (pointer to \refstruct{pmix_cpuset_t})}
\end{arglist}



\littleheader{Create a cpuset array}
\declaremacro{PMIX_CPUSET_CREATE}

Allocate and initialize a \refstruct{pmix_cpuset_t} array.

\copySignature{PMIX_CPUSET_CREATE}{4.0}{
PMIX_CPUSET_CREATE(m, n)
}

\begin{arglist}
\arginout{m}{Address where the pointer to the array of \refstruct{pmix_cpuset_t} structures shall be stored (handle)}
\argin{n}{Number of structures to be allocated (size_t)}
\end{arglist}

\littleheader{Release a cpuset array}
\declaremacro{PMIX_CPUSET_FREE}

Deconstruct and free a \refstruct{pmix_cpuset_t} array.

\copySignature{PMIX_CPUSET_FREE}{4.0}{
PMIX_CPUSET_FREE(m, n)
}

\begin{arglist}
\arginout{m}{Address the array of \refstruct{pmix_cpuset_t} structures to be released (handle)}
\argin{n}{Number of structures in the array (size_t)}
\end{arglist}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_define_process_set}}
\declareapi{PMIx_server_define_process_set}

%%%%
\summary

Define a \ac{PMIx} process set.

%%%%
\format

\copySignature{PMIx_server_define_process_set}{4.0}{
pmix_status_t \\
PMIx_server_define_process_set(const pmix_proc_t members[], \\
\hspace*{31\sigspace}size_t nmembers, \\
\hspace*{31\sigspace}const char *pset_name);
}

\begin{arglist}
\argin{members}{Pointer to an array of \refstruct{pmix_proc_t} containing the
identifiers of the processes in the process set (handle)}
\argin{nmembers}{Number of elements in \refarg{members} (integer)}
\argin{pset_name}{String name of the process set being defined (\code{const char*})}
\end{arglist}

\returnsimple


%%%%
\descr

Provide a function by which the host environment can create a process set. The
\ac{PMIx} server shall alert all local clients of the new process set
(including process set name and membership) via the
\refconst{PMIX_PROCESS_SET_DEFINE} event.

\advicermstart
The host environment is responsible for ensuring:

\begin{itemize}
    \item consistent knowledge of process set membership across all involved
    \ac{PMIx} servers; and
    \item that process set names do not conflict with system-assigned
    namespaces within the scope of the set
\end{itemize}

\advicermend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{PMIx_server_delete_process_set}}
\declareapi{PMIx_server_delete_process_set}

%%%%
\summary

Delete a \ac{PMIx} process set name

%%%%
\format

\copySignature{PMIx_server_delete_process_set}{4.0}{
pmix_status_t \\
PMIx_server_delete_process_set(const char *pset_name);
}

\begin{arglist}
\argin{pset_name}{String name of the process set being deleted (\code{const char*})}
\end{arglist}

\returnsimple

%%%%
\descr

Provide a function by which the host environment can delete a process set name.
The \ac{PMIx} server shall alert all local clients of the process set name
being deleted via the \refconst{PMIX_PROCESS_SET_DELETE} event. Deletion of the name has no impact on the member processes.

\advicermstart
The host environment is responsible for ensuring consistent knowledge of
process set membership across all involved \ac{PMIx} servers.
\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Server Function Pointers}

\label{chap:server:func_pointers}

\ac{PMIx} utilizes a "function-shipping" approach to support for implementing the server-side of the protocol. This method allows \acp{RM} to implement the server without being burdened with \ac{PMIx} internal details. When a request is received from the client, the corresponding server function will be called with the information.

Any functions not supported by the \ac{RM} can be indicated by a \code{NULL} for the function pointer. \ac{PMIx} implementations are required to return a \refconst{PMIX_ERR_NOT_SUPPORTED} status to all calls to functions that require host environment support and are not backed by a corresponding server module entry. Host environments may, if they choose, include a function pointer for operations they have not yet implemented and simply return \refconst{PMIX_ERR_NOT_SUPPORTED}.

Functions that accept directives (i.e., arrays of \refstruct{pmix_info_t} structures) must check any provided directives for those marked as \emph{required} via the \refconst{PMIX_INFO_REQD} flag. \ac{PMIx} client and server libraries are required to mark any such directives with the \refconst{PMIX_INFO_REQD_PROCESSED} flag should they have handled the request. Any required directive that has not been marked therefore becomes the responsibility of the host environment. If a required directive that hasn't been processed by a lower level cannot be supported by the host, then the \refconst{PMIX_ERR_NOT_SUPPORTED} error constant must be returned. If the directive can be processed by the host, then the host shall do so and mark the attribute with the \refconst{PMIX_INFO_REQD_PROCESSED} flag.

The host \ac{RM} will provide the function pointers in a \refapi{pmix_server_module_t} structure passed to \refapi{PMIx_server_init}.
The module structure and associated function references are defined in this section.

\advicermstart
For performance purposes, the host server is required to return as quickly as possible from all functions. Execution of
the function is thus to be done asynchronously so as to allow the \ac{PMIx} server support library to handle multiple client requests
as quickly and scalably as possible.

All data passed to the host server functions is ``owned'' by the
PMIX server support library and must not be free'd. Data returned
by the host server via callback function is owned by the host
server, which is free to release it upon return from the callback
\advicermend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_module_t} Module}
\declareapi{pmix_server_module_t}
\label{server:module_fns}
%%%%
\summary

List of function pointers that a PMIx server passes to \refapi{PMIx_server_init} during startup.

%%%%
\format

\cspecificstart
\begin{codepar}
typedef struct pmix_server_module_4_0_0_t \{
    /* v1x interfaces */
    pmix_server_client_connected_fn_t   client_connected;  // DEPRECATED
    pmix_server_client_finalized_fn_t   client_finalized;
    pmix_server_abort_fn_t              abort;
    pmix_server_fencenb_fn_t            fence_nb;
    pmix_server_dmodex_req_fn_t         direct_modex;
    pmix_server_publish_fn_t            publish;
    pmix_server_lookup_fn_t             lookup;
    pmix_server_unpublish_fn_t          unpublish;
    pmix_server_spawn_fn_t              spawn;
    pmix_server_connect_fn_t            connect;
    pmix_server_disconnect_fn_t         disconnect;
    pmix_server_register_events_fn_t    register_events;
    pmix_server_deregister_events_fn_t  deregister_events;
    pmix_server_listener_fn_t           listener;
    /* v2x interfaces */
    pmix_server_notify_event_fn_t       notify_event;
    pmix_server_query_fn_t              query;
    pmix_server_tool_connection_fn_t    tool_connected;
    pmix_server_log_fn_t                log;
    pmix_server_alloc_fn_t              allocate;
    pmix_server_job_control_fn_t        job_control;
    pmix_server_monitor_fn_t            monitor;
    /* v3x interfaces */
    pmix_server_get_cred_fn_t           get_credential;
    pmix_server_validate_cred_fn_t      validate_credential;
    pmix_server_iof_fn_t                iof_pull;
    pmix_server_stdin_fn_t              push_stdin;
    /* v4x interfaces */
    pmix_server_grp_fn_t                group;
    pmix_server_fabric_fn_t             fabric;
    pmix_server_client_connected2_fn_t  client_connected2;
\} pmix_server_module_t;
\end{codepar}
\cspecificend

\advicermstart
Note that some \ac{PMIx} implementations \emph{require} the use of C99-style
designated initializers to clearly correlate each provided function pointer with the correct member of the \refapi{pmix_server_module_t} structure as the location/ordering of struct members may change over time.
\advicermend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_client_connected_fn_t}}
\declareapiDEPNODISP{pmix_server_client_connected_fn_t}

%%%%
\summary

Notify the host server that a client connected to this server. This function module entry has been \textbf{DEPRECATED} in favor of \refapi{pmix_server_client_connected2_fn_t}.

%%%%
\format

\copySignature{pmix_server_client_connected_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_client_connected_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *proc, \\
\hspace*{29\sigspace}void* server_object, \\
\hspace*{29\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure (handle)}
\argin{server_object}{object reference (memory reference)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

%%%%
\descr

This function module entry has been DEPRECATED in favor of \refapi{pmix_server_client_connected2_fn_t}. If both functions are provided, the \ac{PMIx} library will ignore this function module entry in favor of its replacement.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_client_connected2_fn_t}}
\declareapi{pmix_server_client_connected2_fn_t}

%%%%
\summary

Notify the host server that a client connected to this server - this version of the original function definition has been extended to include an array of \refstruct{pmix_info_t}, thereby allowing the \ac{PMIx} server library to pass additional information identifying the client to the host environment.

%%%%
\format

\copySignature{pmix_server_client_connected2_fn_t}{4.0}{
typedef pmix_status_t (*pmix_server_client_connected2_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *proc, \\
\hspace*{29\sigspace}void* server_object, \\
\hspace*{29\sigspace}pmix_info_t info[], size_t ninfo, \\
\hspace*{29\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata)
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure (handle)}
\argin{server_object}{object reference (memory reference)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called. The \ac{PMIx} server library is to immediately terminate the connection.
\end{itemize}

%%%%
\descr

Notify the host environment that a client has called \refapi{PMIx_Init}.
Note that the client will be in a blocked state until the host server executes the callback function, thus allowing the \ac{PMIx} server support library to release
the client.
The server_object parameter will be the value of the server_object parameter passed to
\refapi{PMIx_server_register_client} by the host server when registering the connecting client. A host server can choose to not be notified when clients connect by setting \refapi{pmix_server_client_connected2_fn_t} to \code{NULL}.

It is possible that only a subset of the clients in a namespace call
\refapi{PMIx_Init}.   The server's \refapi{pmix_server_client_connected2_fn_t}
implementation should therefore not depend on being called once per rank in a
namespace or delay calling the callback function until all ranks have
connected. However, the host may rely on the
\refapi{pmix_server_client_connected2_fn_t}
function module entry being called for a given rank prior to any other function
module entries being executed on behalf of that rank.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_client_finalized_fn_t}}
\declareapi{pmix_server_client_finalized_fn_t}

%%%%
\summary

Notify the host environment that a client called \refapi{PMIx_Finalize}.

%%%%
\format

\copySignature{pmix_server_client_finalized_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_client_finalized_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *proc, \\
\hspace*{29\sigspace}void* server_object, \\
\hspace*{29\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure (handle)}
\argin{server_object}{object reference (memory reference)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

%%%%
\descr

Notify the host environment that a client called \refapi{PMIx_Finalize}.
Note that the client will be in a blocked state until the host server executes the callback function, thus allowing the PMIx server support library to release the client.
The server_object parameter will be the value of the server_object parameter passed to
\refapi{PMIx_server_register_client} by the host server when registering the connecting client.  If provided, an implementation of \refapi{pmix_server_client_finalized_fn_t}
is only required to
call the callback function designated.  A host server can choose to not be notified when clients finalize by setting \refapi{pmix_server_client_finalized_fn_t} to \code{NULL}.

Note that the host server is only being informed that the client has called \refapi{PMIx_Finalize}.  The client might not have exited.  If a client
exits without calling \refapi{PMIx_Finalize}, the server support library will not call the \refapi{pmix_server_client_finalized_fn_t} implementation.

\advicermstart
This operation is an opportunity for a host server
to update the status of the tasks it manages.  It is also a convenient and well defined time to release resources used to support that client.
\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_abort_fn_t}}
\declareapi{pmix_server_abort_fn_t}

%%%%
\summary

Notify the host environment that a local client called \refapi{PMIx_Abort}.

%%%%
\format

\copySignature{pmix_server_abort_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_abort_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *proc, \\
\hspace*{29\sigspace}void *server_object, \\
\hspace*{29\sigspace}int status, \\
\hspace*{29\sigspace}const char msg[], \\
\hspace*{29\sigspace}pmix_proc_t procs[], \\
\hspace*{29\sigspace}size_t nprocs, \\
\hspace*{29\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}


\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure identifying the process requesting the abort (handle)}
\argin{server_object}{object reference (memory reference)}
\argin{status}{exit status (integer)}
\argin{msg}{exit status message (string)}
\argin{procs}{Array of \refstruct{pmix_proc_t} structures identifying the processes to be terminated (array of handles)}
\argin{nprocs}{Number of elements in the \refarg{procs} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_PARAM_VALUE_NOT_SUPPORTED} indicating that the host environment supports this \ac{API}, but the request includes processes that the host environment cannot abort - e.g., if the request is to abort subsets of processes from a namespace, or processes outside of the caller's own namespace, and the host environment does not permit such operations. In this case, none of the specified processes will be terminated - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

%%%%
\descr

A local client called \refapi{PMIx_Abort}.
Note that the client will be in a blocked state until the host server executes the callback function, thus allowing the \ac{PMIx} server library to release the client.
The array of \refarg{procs} indicates which processes are to be terminated.
A \code{NULL} for the \refarg{procs} array indicates that all processes in the caller's namespace are to be aborted, including itself - this is the equivalent of passing a \refstruct{pmix_proc_t} array element containing the caller's namespace and a rank value of \refconst{PMIX_RANK_WILDCARD}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_fencenb_fn_t}}
\declareapi{pmix_server_fencenb_fn_t}

%%%%
\summary

At least one client called either \refapi{PMIx_Fence} or \refapi{PMIx_Fence_nb}.

%%%%
\format

\copySignature{pmix_server_fencenb_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_fencenb_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t procs[], \\
\hspace*{29\sigspace}size_t nprocs, \\
\hspace*{29\sigspace}const pmix_info_t info[], \\
\hspace*{29\sigspace}size_t ninfo, \\
\hspace*{29\sigspace}char *data, size_t ndata, \\
\hspace*{29\sigspace}pmix_modex_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{procs}{Array of \refstruct{pmix_proc_t} structures identifying operation participants(array of handles)}
\argin{nprocs}{Number of elements in the \refarg{procs} array (integer)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{data}{(string)}
\argin{ndata}{(integer)}
\argin{cbfunc}{Callback function \refapi{pmix_modex_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing.

The following attributes are required to be supported by all host environments:

\pasteAttributeItem{PMIX_COLLECT_DATA}
\pasteAttributeItem{PMIX_LOCAL_COLLECTIVE_STATUS}

\reqattrend

\optattrstart
The following attributes are optional for host environments:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

\advicermstart
Host environment are required to return \refconst{PMIX_ERR_NOT_SUPPORTED} if passed an attributed marked as \refconst{PMIX_INFO_REQD} that they do not support, even if support for that attribute is optional.
\advicermend

%%%%
\descr

All local clients in the provided array of \refarg{procs} called either \refapi{PMIx_Fence} or \refapi{PMIx_Fence_nb}.
In either case, the host server will be called via a non-blocking function to execute the specified operation once all participating local processes have contributed.
All processes in the specified \refarg{procs} array are required to participate in the \refapi{PMIx_Fence}/\refapi{PMIx_Fence_nb} operation.
The callback is to be executed once every daemon hosting at least one participant has called the host server's \refapi{pmix_server_fencenb_fn_t} function.

The provided data is to be collectively shared with all \ac{PMIx} servers involved in the fence operation, and returned in the modex \refarg{cbfunc}.
A \code{NULL} data value indicates that the local processes had no data to contribute.

The array of \refarg{info} structs is used to pass user-requested options to the server.
This can include directives as to the algorithm to be used to execute the fence operation.
The directives are optional unless the \refconst{PMIX_INFO_REQD} flag has been set - in such cases, the host \ac{RM} is required to return an error if the directive cannot be met.

\adviceimplstart
The \ac{PMIx} server library is required to aggregate participation by local clients, passing the request to the host environment once all local participants have executed the \ac{API}.
\adviceimplend

\advicermstart
The host will receive a single call for each collective operation. It is the responsibility of the host to identify the nodes containing participating processes, execute the collective across all participating nodes, and notify the local \ac{PMIx} server library upon completion of the global collective. Data received from each node must be simply concatenated to form an aggregated unit, as shown in the following example:

\cspecificstart
\begin{codepar}
uint8_t *blob1, *blob2, *total;
size_t sz_blob1, sz_blob2, sz_total;

sz_total = sz_blob1 + sz_blob2;
total = (uint8_t*)malloc(sz_total);
memcpy(total, blob1, sz_blob1);
memcpy(\&total[sz_blob1], blob2, sz_blob2);
\end{codepar}
\cspecificend

Note that the ordering of the data blobs does not matter. The host is responsible for free'ing the \refarg{data} object passed to it by the \ac{PMIx} server library.
\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Modex Callback Function}
\declareapi{pmix_modex_cbfunc_t}

%%%%
\summary

The \refapi{pmix_modex_cbfunc_t} is used by the \refapi{pmix_server_fencenb_fn_t} and \refapi{pmix_server_dmodex_req_fn_t} PMIx server operations to return modex \ac{BCX} data.

\copySignature{pmix_modex_cbfunc_t}{1.0}{
typedef void (*pmix_modex_cbfunc_t) \\
\hspace*{4\sigspace}(pmix_status_t status, \\
\hspace*{5\sigspace}const char *data, size_t ndata, \\
\hspace*{5\sigspace}void *cbdata, \\
\hspace*{5\sigspace}pmix_release_cbfunc_t release_fn, \\
\hspace*{5\sigspace}void *release_cbdata);
}

\begin{arglist}
\argin{status}{Status associated with the operation (handle)}
\argin{data}{Data to be passed (pointer)}
\argin{ndata}{size of the data (\code{size_t})}
\argin{cbdata}{Callback data passed to original API call (memory reference)}
\argin{release_fn}{Callback for releasing \argref{data} (function pointer)}
\argin{release_cbdata}{Pointer to be passed to \argref{release_fn} (memory reference)}
\end{arglist}

%%%%
\descr

A callback function that is solely used by PMIx servers, and not clients, to return modex \ac{BCX} data in response to ``fence'' and ``get'' operations.
The returned blob contains the data collected from each server participating in the operation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_dmodex_req_fn_t}}
\declareapi{pmix_server_dmodex_req_fn_t}

%%%%
\summary

Used by the PMIx server to request its local host contact the \ac{PMIx} server on the remote node that hosts the specified process to obtain and return a direct modex blob for that process.

%%%%
\format

\copySignature{pmix_server_dmodex_req_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_dmodex_req_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *proc, \\
\hspace*{29\sigspace}const pmix_info_t info[], \\
\hspace*{29\sigspace}size_t ninfo, \\
\hspace*{29\sigspace}pmix_modex_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure identifying the process whose data is being requested (handle)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_modex_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing.

All host environments are required to support the following attributes:

\pasteAttributeItem{PMIX_REQUIRED_KEY}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Used by the \ac{PMIx} server to request its local host contact the \ac{PMIx} server on the remote node that hosts the specified proc to obtain (using \refapi{PMIx_server_dmodex_request}) and return any information that the process, \refarg{proc}, posted via calls to \refapi{PMIx_Put} and \refapi{PMIx_Commit}.

The array of \refarg{info} structs is used to pass user-requested options to the server.
This can include a timeout to preclude an indefinite wait for data that may never become available.
The directives are optional unless the \emph{mandatory} flag has been set - in such cases, the host \ac{RM} is required to return an error if the directive cannot be met.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Dmodex attributes}

%
\declareAttribute{PMIX_REQUIRED_KEY}{"pmix.req.key"}{char*}{
Identifies a key that must be included in the requested information. If the specified key is not already available, then the \ac{PMIx} servers are required to delay response to the dmodex request until either the key becomes available or the request times out.}

\vspace{\baselineskip}
\advicermstart
The server can test the availability of a required key by calling \refapi{PMIx_Get} on the key.  Since there is no mechanism to retract a key-value that has been made available by a client through PMIx_Put, a successful \refapi{PMIx_Get} call on the key by the server is sufficient to ensure that a subsequent \refapi{PMIx_server_dmodex_request} call targeting the process will include the key.
\advicermend

%%%%%%%%%%%
\subsection{\code{pmix_server_publish_fn_t}}
\declareapi{pmix_server_publish_fn_t}

%%%%
\summary

Publish data per the PMIx API specification.

%%%%
\format

\copySignature{pmix_server_publish_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_publish_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *proc, \\
\hspace*{29\sigspace}const pmix_info_t info[], \\
\hspace*{29\sigspace}size_t ninfo, \\
\hspace*{29\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure of the process publishing the data (handle)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing. In addition, the following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\divider

Host environments that implement this entry point are required to support the following attributes:

\pasteAttributeItem{PMIX_RANGE}
\pasteAttributeItem{PMIX_PERSISTENCE}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Publish data per the \refapi{PMIx_Publish} specification.
The callback is to be executed upon completion of the operation.
The default data range is left to the host environment, but expected to be \refconst{PMIX_RANGE_SESSION}, and the default persistence \refconst{PMIX_PERSIST_SESSION} or their equivalent.
These values can be specified by including the respective attributed in the \refarg{info} array.

The persistence indicates how long the server should retain the data.

\advicermstart
The host environment is not required to guarantee support for any specific range - i.e., the environment does not need to return an error if the data store doesn't support a specified range so long as it is covered by some internally defined range.
However, the server must return an error (a) if the key is duplicative within the storage range, and (b) if the server does not allow overwriting of published info by the original publisher - it is left to the discretion of the host environment to allow info-key-based flags to modify this behavior.

The \refattr{PMIX_USERID} and \refattr{PMIX_GRPID} of the publishing process will be provided to support authorization-based access to published information and must be returned on any subsequent lookup request.
\advicermend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_lookup_fn_t}}
\declareapi{pmix_server_lookup_fn_t}

%%%%
\summary

Lookup published data.

%%%%
\format

\copySignature{pmix_server_lookup_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_lookup_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *proc, \\
\hspace*{29\sigspace}char **keys, \\
\hspace*{29\sigspace}const pmix_info_t info[], \\
\hspace*{29\sigspace}size_t ninfo, \\
\hspace*{29\sigspace}pmix_lookup_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure of the process seeking the data (handle)}
\argin{keys}{(array of strings)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_lookup_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing. In addition, the following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\divider

Host environments that implement this entry point are required to support the following attributes:

\pasteAttributeItem{PMIX_RANGE}
\pasteAttributeItem{PMIX_WAIT}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend


%%%%
\descr

Lookup published data.
The host server will be passed a \code{NULL}-terminated array of string keys identifying the data being requested.

The array of \refarg{info} structs is used to pass user-requested options to the server. The default data range is left to the host environment, but expected to be \refconst{PMIX_RANGE_SESSION}.
This can include a wait flag to indicate that the server should wait for all data to become available before executing the callback function, or should immediately callback with whatever data is available.
In addition, a timeout can be specified on the wait to preclude an indefinite wait for data that may never be published.

\advicermstart
The \refattr{PMIX_USERID} and \refattr{PMIX_GRPID} of the requesting process will be provided to support authorization-based access to published information. The host environment is not required to guarantee support for any specific range - i.e., the environment does not need to return an error if the data store doesn't support a specified range so long as it is covered by some internally defined range.
\advicermend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_unpublish_fn_t}}
\declareapi{pmix_server_unpublish_fn_t}

%%%%
\summary

Delete data from the data store.

%%%%
\format

\copySignature{pmix_server_unpublish_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_unpublish_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *proc, \\
\hspace*{29\sigspace}char **keys, \\
\hspace*{29\sigspace}const pmix_info_t info[], \\
\hspace*{29\sigspace}size_t ninfo, \\
\hspace*{29\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure identifying the process making the request (handle)}
\argin{keys}{(array of strings)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing. In addition, the following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\divider

Host environments that implement this entry point are required to support the following attributes:

\pasteAttributeItem{PMIX_RANGE}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Delete data from the data store.
The host server will be passed a \code{NULL}-terminated array of string keys, plus potential directives such as the data range within which the keys should be deleted. The default data range is left to the host environment, but expected to be \refconst{PMIX_RANGE_SESSION}.
The callback is to be executed upon completion of the delete procedure.

\advicermstart
The \refattr{PMIX_USERID} and \refattr{PMIX_GRPID} of the requesting process will be provided to support authorization-based access to published information. The host environment is not required to guarantee support for any specific range - i.e., the environment does not need to return an error if the data store doesn't support a specified range so long as it is covered by some internally defined range.
\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_spawn_fn_t}}
\declareapi{pmix_server_spawn_fn_t}

%%%%
\summary

Spawn a set of applications/processes as per the \refapi{PMIx_Spawn} API.

%%%%
\format

\copySignature{pmix_server_spawn_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_spawn_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *proc, \\
\hspace*{29\sigspace}const pmix_info_t job_info[], \\
\hspace*{29\sigspace}size_t ninfo, \\
\hspace*{29\sigspace}const pmix_app_t apps[], \\
\hspace*{29\sigspace}size_t napps, \\
\hspace*{29\sigspace}pmix_spawn_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure of the process making the request (handle)}
\argin{job_info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{jobinfo} array (integer)}
\argin{apps}{Array of \refstruct{pmix_app_t} structures (array of handles)}
\argin{napps}{Number of elements in the \refarg{apps} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_spawn_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\ac{PMIx} server libraries are required to pass any provided attributes to the host environment for processing. In addition, the following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}
\pasteAttributeItem{PMIX_SPAWNED}
\pasteAttributeItem{PMIX_PARENT_ID}
\pasteAttributeItem{PMIX_REQUESTOR_IS_TOOL}
\pasteAttributeItem{PMIX_REQUESTOR_IS_CLIENT}

\divider

Host environments that provide this module entry point are required to pass the \refattr{PMIX_SPAWNED} and \refattr{PMIX_PARENT_ID} attributes to all \ac{PMIx} servers launching new child processes so those values can be returned to clients upon connection to the \ac{PMIx} server. In addition, they are required to support the following attributes when present in either the \refarg{job_info} or the \textit{info} array of an element of the \refarg{apps} array:

\pasteAttributeItem{PMIX_WDIR}
\pasteAttributeItem{PMIX_SET_SESSION_CWD}
\pasteAttributeItem{PMIX_PREFIX}
\pasteAttributeItem{PMIX_HOST}
\pasteAttributeItem{PMIX_HOSTFILE}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_ADD_HOSTFILE}
\pasteAttributeItem{PMIX_ADD_HOST}
\pasteAttributeItem{PMIX_PRELOAD_BIN}
\pasteAttributeItem{PMIX_PRELOAD_FILES}
\pasteAttributeItem{PMIX_PERSONALITY}
\pasteAttributeItem{PMIX_DISPLAY_MAP}
\pasteAttributeItem{PMIX_PPR}
\pasteAttributeItem{PMIX_MAPBY}
\pasteAttributeItem{PMIX_RANKBY}
\pasteAttributeItem{PMIX_BINDTO}
\pasteAttributeItem{PMIX_STDIN_TGT}
\pasteAttributeItem{PMIX_FWD_STDIN}
\pasteAttributeItem{PMIX_FWD_STDOUT}
\pasteAttributeItem{PMIX_FWD_STDERR}
\pasteAttributeItem{PMIX_DEBUGGER_DAEMONS}
\pasteAttributeItem{PMIX_TAG_OUTPUT}
\pasteAttributeItem{PMIX_TIMESTAMP_OUTPUT}
\pasteAttributeItem{PMIX_MERGE_STDERR_STDOUT}
\pasteAttributeItem{PMIX_OUTPUT_TO_FILE}
\pasteAttributeItem{PMIX_INDEX_ARGV}
\pasteAttributeItem{PMIX_CPUS_PER_PROC}
\pasteAttributeItem{PMIX_NO_PROCS_ON_HEAD}
\pasteAttributeItem{PMIX_NO_OVERSUBSCRIBE}
\pasteAttributeItem{PMIX_REPORT_BINDINGS}
\pasteAttributeItem{PMIX_CPU_LIST}
\pasteAttributeItem{PMIX_JOB_RECOVERABLE}
\pasteAttributeItem{PMIX_JOB_CONTINUOUS}
\pasteAttributeItem{PMIX_MAX_RESTARTS}
\pasteAttributeItem{PMIX_TIMEOUT}
\pasteAttributeItem{PMIX_JOB_TIMEOUT}
\pasteAttributeItem{PMIX_SPAWN_TIMEOUT}

\optattrend

%%%%
\descr

Spawn a set of applications/processes as per the \refapi{PMIx_Spawn} API.
Note that applications are not required to be \ac{MPI} or any other programming model.
Thus, the host server cannot make any assumptions as to their required support.
The callback function is to be executed once all processes have been started.
An error in starting any application or process in this request shall cause all applications and processes in the request to be terminated, and an error returned to the originating caller.

Note that a timeout can be specified in the job_info array to indicate that failure to start the requested job within the given time should result in termination to avoid hangs.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Server spawn attributes}

%
\declareAttribute{PMIX_REQUESTOR_IS_TOOL}{"pmix.req.tool"}{bool}{
The requesting process is a \ac{PMIx} tool.
}
%
\declareAttribute{PMIX_REQUESTOR_IS_CLIENT}{"pmix.req.client"}{bool}{
The requesting process is a \ac{PMIx} client.
}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_connect_fn_t}}
\declareapi{pmix_server_connect_fn_t}

%%%%
\summary

Record the specified processes as \textit{connected}.

%%%%
\format

\copySignature{pmix_server_connect_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_connect_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t procs[], \\
\hspace*{29\sigspace}size_t nprocs, \\
\hspace*{29\sigspace}const pmix_info_t info[], \\
\hspace*{29\sigspace}size_t ninfo, \\
\hspace*{29\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{procs}{Array of \refstruct{pmix_proc_t} structures identifying participants (array of handles)}
\argin{nprocs}{Number of elements in the \refarg{procs} array (integer)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\pasteAttributeItem{PMIX_LOCAL_COLLECTIVE_STATUS}

\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing.
\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Record the processes specified by the \refarg{procs} array as \textit{connected} as per the \ac{PMIx} definition\refsection{chap:api_proc_mgmt:connect}. The callback is to be executed once every daemon hosting at least one participant has called the host server's \refapi{pmix_server_connect_fn_t} function, and the host environment has completed any supporting operations required to meet the terms of the \ac{PMIx} definition of \textit{connected} processes.

\adviceimplstart
The \ac{PMIx} server library is required to aggregate participation by local clients, passing the request to the host environment once all local participants have executed the \ac{API}.
\adviceimplend

\advicermstart
The host will receive a single call for each collective operation. It is the responsibility of the host to identify the nodes containing participating processes, execute the collective across all participating nodes, and notify the local \ac{PMIx} server library upon completion of the global collective.
\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_disconnect_fn_t}}
\declareapi{pmix_server_disconnect_fn_t}

%%%%
\summary

Disconnect a previously connected set of processes.

%%%%
\format

\copySignature{pmix_server_disconnect_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_disconnect_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t procs[], \\
\hspace*{29\sigspace}size_t nprocs, \\
\hspace*{29\sigspace}const pmix_info_t info[], \\
\hspace*{29\sigspace}size_t ninfo, \\
\hspace*{29\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{procs}{Array of \refstruct{pmix_proc_t} structures identifying participants (array of handles)}
\argin{nprocs}{Number of elements in the \refarg{procs} array (integer)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\pasteAttributeItem{PMIX_LOCAL_COLLECTIVE_STATUS}

\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing.
\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Disconnect a previously connected set of processes. The callback is to be executed once every daemon hosting at least one participant has called the host server's has called the \refapi{pmix_server_disconnect_fn_t} function, and the host environment has completed any required supporting operations.

\adviceimplstart
The \ac{PMIx} server library is required to aggregate participation by local clients, passing the request to the host environment once all local participants have executed the \ac{API}.
\adviceimplend

\advicermstart
The host will receive a single call for each collective operation. It is the responsibility of the host to identify the nodes containing participating processes, execute the collective across all participating nodes, and notify the local \ac{PMIx} server library upon completion of the global collective.

A \refconst{PMIX_ERR_INVALID_OPERATION} error must be returned if the specified set of \refarg{procs} was not previously \textit{connected} via a call to the \refapi{pmix_server_connect_fn_t} function.
\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_register_events_fn_t}}
\declareapi{pmix_server_register_events_fn_t}

%%%%
\summary

Register to receive notifications for the specified events.

%%%%
\format

\copySignature{pmix_server_register_events_fn_t}{1.0}{
\hspace*{1\sigspace}typedef pmix_status_t (*pmix_server_register_events_fn_t)( \\
\hspace*{30\sigspace}pmix_status_t *codes, \\
\hspace*{30\sigspace}size_t ncodes, \\
\hspace*{30\sigspace}const pmix_info_t info[], \\
\hspace*{30\sigspace}size_t ninfo, \\
\hspace*{30\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{30\sigspace}void *cbdata);
}

\begin{arglist}
\argin{codes}{Array of \refstruct{pmix_status_t} values (array of handles)}
\argin{ncodes}{Number of elements in the \refarg{codes} array (integer)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing. In addition, the following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\reqattrend

%%%%
\descr

Register to receive notifications for the specified status codes. The \refarg{info} array included in this API is reserved for possible future directives to further steer notification.

\adviceimplstart
The \ac{PMIx} server library must track all client registrations for subsequent notification. This module function shall only be called when:

\begin{itemize}
    \item the client has requested notification of an environmental code (i.e., a \ac{PMIx} codes in the range between \refconst{PMIX_EVENT_SYS_BASE} and \refconst{PMIX_EVENT_SYS_OTHER}, inclusive) or codes that lies outside the defined \ac{PMIx} range of constants; and
    \item the \ac{PMIx} server library has not previously requested notification of that code - i.e., the host environment is to be contacted only once a given unique code value
\end{itemize}

\adviceimplend

\advicermstart
The host environment is required to pass to its \ac{PMIx} server library all non-environmental events that directly relate to a registered namespace without the \ac{PMIx} server library explicitly requesting them. Environmental events are to be translated to their nearest \ac{PMIx} equivalent code as defined in the range between \refconst{PMIX_EVENT_SYS_BASE} and \refconst{PMIX_EVENT_SYS_OTHER} (inclusive).
\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_deregister_events_fn_t}}
\declareapi{pmix_server_deregister_events_fn_t}

%%%%
\summary

Deregister to receive notifications for the specified events.

%%%%
\format

\copySignature{pmix_server_deregister_events_fn_t}{1.0}{
\hspace*{1\sigspace}typedef pmix_status_t (*pmix_server_deregister_events_fn_t)( \\
\hspace*{30\sigspace}pmix_status_t *codes, \\
\hspace*{30\sigspace}size_t ncodes, \\
\hspace*{30\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{30\sigspace}void *cbdata);
}

\begin{arglist}
\argin{codes}{Array of \refstruct{pmix_status_t} values (array of handles)}
\argin{ncodes}{Number of elements in the \refarg{codes} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

%%%%
\descr

Deregister to receive notifications for the specified events to which the \ac{PMIx} server has previously registered.

\adviceimplstart
The \ac{PMIx} server library must track all client registrations. This module function shall only be called when:

\begin{itemize}
    \item the library is deregistering environmental codes (i.e., a \ac{PMIx} codes in the range between \refconst{PMIX_EVENT_SYS_BASE} and \refconst{PMIX_EVENT_SYS_OTHER}, inclusive) or codes that lies outside the defined \ac{PMIx} range of constants; and
    \item no client (including the server library itself) remains registered for notifications on any included code - i.e., a code should be included in this call only when no registered notifications against it remain.
\end{itemize}

\adviceimplend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_notify_event_fn_t}}
\declareapi{pmix_server_notify_event_fn_t}

%%%%
\summary

Notify the specified processes of an event.

%%%%
\format

\copySignature{pmix_server_notify_event_fn_t}{2.0}{
typedef pmix_status_t (*pmix_server_notify_event_fn_t)( \\
\hspace*{29\sigspace}pmix_status_t code, \\
\hspace*{29\sigspace}const pmix_proc_t *source, \\
\hspace*{29\sigspace}pmix_data_range_t range, \\
\hspace*{29\sigspace}pmix_info_t info[], \\
\hspace*{29\sigspace}size_t ninfo, \\
\hspace*{29\sigspace}pmix_op_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{code}{The \refstruct{pmix_status_t} event code being referenced structure (handle)}
\argin{source}{\refstruct{pmix_proc_t} of process that generated the event (handle)}
\argin{range}{\refstruct{pmix_data_range_t} range over which the event is to be distributed (handle)}
\argin{info}{Optional array of \refstruct{pmix_info_t} structures containing additional information on the event (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing.

Host environments that provide this module entry point are required to support the following attributes:

\pasteAttributeItem{PMIX_RANGE}

\reqattrend

%%%%
\descr

Notify the specified processes (described through a combination of \refarg{range} and attributes provided in the \refarg{info} array) of an event generated either by the \ac{PMIx} server itself or by one of its local clients.
The process generating the event is provided in the \refarg{source} parameter, and any further descriptive information is
included in the \refarg{info} array.

Note that the \ac{PMIx} server library is not allowed to echo any event given to it by its host via the \refapi{PMIx_Notify_event} \ac{API} back to the host through the \refapi{pmix_server_notify_event_fn_t} server module function.

\advicermstart
The callback function is to be executed once the host environment no longer requires that the \ac{PMIx} server library maintain the provided data structures. It does not necessarily indicate that the event has been delivered to any process, nor that the event has been distributed for delivery
\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_listener_fn_t}}
\declareapi{pmix_server_listener_fn_t}

%%%%
\summary

Register a socket the host server can monitor for connection requests.

%%%%
\format

\copySignature{pmix_server_listener_fn_t}{1.0}{
typedef pmix_status_t (*pmix_server_listener_fn_t)( \\
\hspace*{29\sigspace}int listening_sd, \\
\hspace*{29\sigspace}pmix_connection_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{incoming_sd}{(integer)}
\argin{cbfunc}{Callback function \refapi{pmix_connection_cbfunc_t} (function reference)}
\argin{cbdata}{ (memory reference)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} indicating that the request is accepted, or a negative value corresponding to a PMIx error constant indicating that the request has been rejected.

%%%%
\descr

Register a socket the host environment can monitor for connection requests, harvest them, and then call the \ac{PMIx} server library's internal callback function for further processing.
A listener thread is essential to efficiently harvesting connection requests from large numbers of local clients such as occur when running on large SMPs.
The host server listener is required to call accept on the incoming connection request, and then pass the resulting socket to the provided cbfunc.
A \code{NULL} for this function will cause the internal \ac{PMIx} server to spawn its own listener thread.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{PMIx Client Connection Callback Function}
\declareapi{pmix_connection_cbfunc_t}

%%%%
\summary

Callback function for incoming connection request from a local client.

%%%%
\format

\copySignature{pmix_connection_cbfunc_t}{1.0}{
typedef void (*pmix_connection_cbfunc_t)( \\
\hspace*{20\sigspace}int incoming_sd, void *cbdata);
}

\begin{arglist}
\argin{incoming_sd}{(integer)}
\argin{cbdata}{ (memory reference)}
\end{arglist}

%%%%
\descr

Callback function for incoming connection requests from local clients - only used by host environments that wish to directly handle socket connection requests.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_query_fn_t}}
\declareapi{pmix_server_query_fn_t}

%%%%
\summary

Query information from the resource manager.

%%%%
\format

\copySignature{pmix_server_query_fn_t}{2.0}{
typedef pmix_status_t (*pmix_server_query_fn_t)( \\
\hspace*{29\sigspace}pmix_proc_t *proct, \\
\hspace*{29\sigspace}pmix_query_t *queries, \\
\hspace*{29\sigspace}size_t nqueries, \\
\hspace*{29\sigspace}pmix_info_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{proct}{\refstruct{pmix_proc_t} structure of the requesting process (handle)}
\argin{queries}{Array of \refstruct{pmix_query_t} structures (array of handles)}
\argin{nqueries}{Number of elements in the \refarg{queries} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_info_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing. In addition, the following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\reqattrend


\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_QUERY_NAMESPACES}
\pasteAttributeItem{PMIX_QUERY_JOB_STATUS}
\pasteAttributeItem{PMIX_QUERY_QUEUE_LIST}
\pasteAttributeItem{PMIX_QUERY_QUEUE_STATUS}
\pasteAttributeItem{PMIX_QUERY_PROC_TABLE}
\pasteAttributeItem{PMIX_QUERY_LOCAL_PROC_TABLE}
\pasteAttributeItem{PMIX_QUERY_SPAWN_SUPPORT}
\pasteAttributeItem{PMIX_QUERY_DEBUG_SUPPORT}
\pasteAttributeItem{PMIX_QUERY_MEMORY_USAGE}
\pasteAttributeItem{PMIX_QUERY_LOCAL_ONLY}
\pasteAttributeItem{PMIX_QUERY_REPORT_AVG}
\pasteAttributeItem{PMIX_QUERY_REPORT_MINMAX}
\pasteAttributeItem{PMIX_QUERY_ALLOC_STATUS}
\pasteAttributeItem{PMIX_TIME_REMAINING}

\optattrend
%%%%
\descr

Query information from the host environment.
The query will include the namespace/rank of the process that is requesting the info, an array of \refstruct{pmix_query_t} describing the request, and a callback function/data for the return.

\adviceimplstart
The \ac{PMIx} server library should not block in this function as the host environment may, depending upon the information being requested, require significant time to respond.
\adviceimplend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_tool_connection_fn_t}}
\declareapi{pmix_server_tool_connection_fn_t}

%%%%
\summary

Register that a tool has connected to the server.

%%%%
\format

\copySignature{pmix_server_tool_connection_fn_t}{2.0}{
typedef void (*pmix_server_tool_connection_fn_t)( \\
\hspace*{29\sigspace}pmix_info_t info[], size_t ninfo, \\
\hspace*{29\sigspace}pmix_tool_connection_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{info}{Array of \refstruct{pmix_info_t} structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_tool_connection_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\reqattrstart
\ac{PMIx} libraries are required to pass the following attributes in the \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}
\pasteAttributeItemBegin{PMIX_TOOL_NSPACE}This must be included only if the tool already has an assigned namespace.
\pasteAttributeItemEnd{}
\pasteAttributeItemBegin{PMIX_TOOL_RANK}This must be included only if the tool already has an assigned rank.
\pasteAttributeItemEnd{}
\pasteAttributeItem{PMIX_CREDENTIAL}

\reqattrend


\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_FWD_STDOUT}
\pasteAttributeItem{PMIX_FWD_STDERR}
\pasteAttributeItem{PMIX_FWD_STDIN}
\pasteAttributeItem{PMIX_VERSION_INFO}

\optattrend

%%%%
\descr

Register that a tool has connected to the server, possibly requesting that the
tool be assigned a namespace/rank identifier for further interactions.
The \refstruct{pmix_info_t} array is used to pass qualifiers for the
connection request, including the effective uid and gid of the calling tool
for authentication purposes.

If the tool already has an assigned process identifier, then this must be
indicated in the \refarg{info} array. The host is responsible for checking
that the provided namespace does not conflict with any currently known
assignments, returning an appropriate error in the callback function if a conflict is found.

The host environment is solely responsible for authenticating and authorizing
the connection using whatever means it deems appropriate. If certificates or
other authentication information are required, then the tool must provide them.
The conclusion of those operations shall be communicated back to the \ac{PMIx}
server library via the callback function.

Approval or rejection of the connection request shall be returned in the
\refarg{status} parameter of the \refapi{pmix_tool_connection_cbfunc_t}. If
the connection is refused, the \ac{PMIx} server library must terminate the
connection attempt. The host must not execute the callback function prior to
returning from the \ac{API}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Tool connection attributes}

Attributes associated with tool connections.

%
\declareAttribute{PMIX_USERID}{"pmix.euid"}{uint32_t}{
Effective user ID of the connecting process.
}
%
\declareAttribute{PMIX_GRPID}{"pmix.egid"}{uint32_t}{
Effective group ID of the connecting process.
}
%
\declareAttribute{PMIX_VERSION_INFO}{"pmix.version"}{char*}{
\ac{PMIx} version of the library being used by the connecting process.
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{PMIx Tool Connection Callback Function}
\declareapi{pmix_tool_connection_cbfunc_t}

%%%%
\summary

Callback function for incoming tool connections.

%%%%
\format

\copySignature{pmix_tool_connection_cbfunc_t}{2.0}{
typedef void (*pmix_tool_connection_cbfunc_t)( \\
\hspace*{20\sigspace}pmix_status_t status, \\
\hspace*{20\sigspace}pmix_proc_t *proc, void *cbdata);
}

\begin{arglist}
\argin{status}{\refstruct{pmix_status_t} value (handle)}
\argin{proc}{\refstruct{pmix_proc_t} structure containing the identifier assigned to the tool (handle)}
\argin{cbdata}{Data to be passed (memory reference)}
\end{arglist}

%%%%
\descr

Callback function for incoming tool connections.
The host environment shall provide a namespace/rank identifier for the connecting tool.

\advicermstart
It is assumed that \code{rank=0} will be the normal assignment, but allow for the future possibility of a parallel set of tools connecting, and thus each process requiring a unique rank.
\advicermend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_log_fn_t}}
\declareapi{pmix_server_log_fn_t}

%%%%
\summary

Log data on behalf of a client.

%%%%
\format

\copySignature{pmix_server_log_fn_t}{2.0}{
typedef void (*pmix_server_log_fn_t)( \\
\hspace*{20\sigspace}const pmix_proc_t *client, \\
\hspace*{20\sigspace}const pmix_info_t data[], size_t ndata, \\
\hspace*{20\sigspace}const pmix_info_t directives[], size_t ndirs, \\
\hspace*{20\sigspace}pmix_op_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{client}{\refstruct{pmix_proc_t} structure (handle)}
\argin{data}{Array of info structures (array of handles)}
\argin{ndata}{Number of elements in the \refarg{data} array (integer)}
\argin{directives}{Array of info structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{directives} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}


\reqattrstart
\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing. In addition, the following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\divider

Host environments that provide this module entry point are required to support the following attributes:

\pasteAttributeItem{PMIX_LOG_STDERR}
\pasteAttributeItem{PMIX_LOG_STDOUT}
\pasteAttributeItem{PMIX_LOG_SYSLOG}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_LOG_MSG}
\pasteAttributeItem{PMIX_LOG_EMAIL}
\pasteAttributeItem{PMIX_LOG_EMAIL_ADDR}
\pasteAttributeItem{PMIX_LOG_EMAIL_SUBJECT}
\pasteAttributeItem{PMIX_LOG_EMAIL_MSG}

\optattrend

%%%%
\descr

Log data on behalf of a client. This function is not intended for output of computational results, but rather for reporting status and error messages. The host must not execute the callback function prior to returning from the \ac{API}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_alloc_fn_t}}
\declareapi{pmix_server_alloc_fn_t}

%%%%
\summary

Request allocation operations on behalf of a client.

%%%%
\format

\copySignature{pmix_server_alloc_fn_t}{2.0}{
typedef pmix_status_t (*pmix_server_alloc_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *client, \\
\hspace*{29\sigspace}pmix_alloc_directive_t directive, \\
\hspace*{29\sigspace}const pmix_info_t data[], \\
\hspace*{29\sigspace}size_t ndata, \\
\hspace*{29\sigspace}pmix_info_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{client}{\refstruct{pmix_proc_t} structure of process making request (handle)}
\argin{directive}{Specific action being requested (\refstruct{pmix_alloc_directive_t})}
\argin{data}{Array of info structures (array of handles)}
\argin{ndata}{Number of elements in the \refarg{data} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_info_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\ac{PMIx} libraries are required to pass any provided attributes to the host environment for processing. In addition, the following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\divider

Host environments that provide this module entry point are required to support the following attributes:

\pasteAttributeItem{PMIX_ALLOC_ID}
\pasteAttributeItem{PMIX_ALLOC_NUM_NODES}
\pasteAttributeItem{PMIX_ALLOC_NUM_CPUS}
\pasteAttributeItem{PMIX_ALLOC_TIME}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_ALLOC_NODE_LIST}
\pasteAttributeItem{PMIX_ALLOC_NUM_CPU_LIST}
\pasteAttributeItem{PMIX_ALLOC_CPU_LIST}
\pasteAttributeItem{PMIX_ALLOC_MEM_SIZE}
\pasteAttributeItem{PMIX_ALLOC_FABRIC}
\pasteAttributeItem{PMIX_ALLOC_FABRIC_ID}
\pasteAttributeItem{PMIX_ALLOC_BANDWIDTH}
\pasteAttributeItem{PMIX_ALLOC_FABRIC_QOS}

\optattrend

%%%%
\descr

Request new allocation or modifications to an existing allocation on behalf of a client. Several broad categories are envisioned, including the ability to:

\begin{compactitem}
%
\item Request allocation of additional resources, including memory, bandwidth, and compute for an existing allocation. Any additional allocated resources will be considered as part of the current allocation, and thus will be released at the same time.
%
\item Request a new allocation of resources. Note that the new allocation will be disjoint from (i.e., not affiliated with) the allocation of the requestor - thus the termination of one allocation will not impact the other.
%
\item Extend the reservation on currently allocated resources, subject to scheduling availability and priorities.
%
\item Return no-longer-required resources to the scheduler.
This includes the \textit{loan} of resources back to the scheduler with a promise to return them upon subsequent request.
\end{compactitem}

The callback function provides a \refarg{status} to indicate whether or not the request was granted, and to provide some information as to the reason for any denial in the \refapi{pmix_info_cbfunc_t} array of \refstruct{pmix_info_t} structures.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_job_control_fn_t}}
\declareapi{pmix_server_job_control_fn_t}

%%%%
\summary

Execute a job control action on behalf of a client.

%%%%
\format

\copySignature{pmix_server_job_control_fn_t}{2.0}{
typedef pmix_status_t (*pmix_server_job_control_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *requestor, \\
\hspace*{29\sigspace}const pmix_proc_t targets[], \\
\hspace*{29\sigspace}size_t ntargets, \\
\hspace*{29\sigspace}const pmix_info_t directives[], \\
\hspace*{29\sigspace}size_t ndirs, \\
\hspace*{29\sigspace}pmix_info_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{requestor}{\refstruct{pmix_proc_t} structure of requesting process (handle)}
\argin{targets}{Array of proc structures (array of handles)}
\argin{ntargets}{Number of elements in the \refarg{targets} array (integer)}
\argin{directives}{Array of info structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_info_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
\ac{PMIx} libraries are required to pass any attributes provided by the client to the host environment for processing. In addition, the following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\divider

Host environments that provide this module entry point are required to support the following attributes:

\pasteAttributeItem{PMIX_JOB_CTRL_ID}
\pasteAttributeItem{PMIX_JOB_CTRL_PAUSE}
\pasteAttributeItem{PMIX_JOB_CTRL_RESUME}
\pasteAttributeItem{PMIX_JOB_CTRL_KILL}
\pasteAttributeItem{PMIX_JOB_CTRL_SIGNAL}
\pasteAttributeItem{PMIX_JOB_CTRL_TERMINATE}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_JOB_CTRL_CANCEL}
\pasteAttributeItem{PMIX_JOB_CTRL_RESTART}
\pasteAttributeItem{PMIX_JOB_CTRL_CHECKPOINT}
\pasteAttributeItem{PMIX_JOB_CTRL_CHECKPOINT_EVENT}
\pasteAttributeItem{PMIX_JOB_CTRL_CHECKPOINT_SIGNAL}
\pasteAttributeItem{PMIX_JOB_CTRL_CHECKPOINT_TIMEOUT}
\pasteAttributeItem{PMIX_JOB_CTRL_CHECKPOINT_METHOD}
\pasteAttributeItem{PMIX_JOB_CTRL_PROVISION}
\pasteAttributeItem{PMIX_JOB_CTRL_PROVISION_IMAGE}
\pasteAttributeItem{PMIX_JOB_CTRL_PREEMPTIBLE}

\optattrend

%%%%
\descr

Execute a job control action on behalf of a client. The \refarg{targets} array identifies the processes to which the requested job control action is to be applied.
A \code{NULL} value can be used to indicate all processes in the caller's namespace.
The use of \refconst{PMIX_RANK_WILDCARD} can also be used to indicate that all processes in the given namespace are to be included.

The directives are provided as \refstruct{pmix_info_t} structures in the \refarg{directives} array.
The callback function provides a \refarg{status} to indicate whether or not the request was granted, and to provide some information as to the reason for any denial in the \refapi{pmix_info_cbfunc_t} array of \refstruct{pmix_info_t} structures.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_monitor_fn_t}}
\declareapi{pmix_server_monitor_fn_t}

%%%%
\summary

Request that a client be monitored for activity.

%%%%
\format

\copySignature{pmix_server_monitor_fn_t}{2.0}{
typedef pmix_status_t (*pmix_server_monitor_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *requestor, \\
\hspace*{29\sigspace}const pmix_info_t *monitor, \\
\hspace*{29\sigspace}pmix_status_t error, \\
\hspace*{29\sigspace}const pmix_info_t directives[], \\
\hspace*{29\sigspace}size_t ndirs, \\
\hspace*{29\sigspace}pmix_info_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{requestor}{\refstruct{pmix_proc_t} structure of requesting process (handle)}
\argin{monitor}{\refstruct{pmix_info_t} identifying the type of monitor being requested (handle)}
\argin{error}{Status code to use in generating event if alarm triggers (integer)}
\argin{directives}{Array of info structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_info_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the host must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

This entry point is only called for monitoring requests that are not directly supported by the \ac{PMIx} server library itself.

\reqattrstart
If supported by the \ac{PMIx} server library, then the library must not pass any supported attributes to the host environment. Any attributes provided by the client that are not directly supported by the server library must be passed to the host environment if it provides this module entry. In addition, the following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

Host environments are not required to support any specific monitoring attributes.

\reqattrend

\optattrstart
The following attributes may be implemented by a host environment.

\pasteAttributeItem{PMIX_MONITOR_ID}
\pasteAttributeItem{PMIX_MONITOR_CANCEL}
\pasteAttributeItem{PMIX_MONITOR_APP_CONTROL}
\pasteAttributeItem{PMIX_MONITOR_HEARTBEAT}
\pasteAttributeItem{PMIX_MONITOR_HEARTBEAT_TIME}
\pasteAttributeItem{PMIX_MONITOR_HEARTBEAT_DROPS}
\pasteAttributeItem{PMIX_MONITOR_FILE}
\pasteAttributeItem{PMIX_MONITOR_FILE_SIZE}
\pasteAttributeItem{PMIX_MONITOR_FILE_ACCESS}
\pasteAttributeItem{PMIX_MONITOR_FILE_MODIFY}
\pasteAttributeItem{PMIX_MONITOR_FILE_CHECK_TIME}
\pasteAttributeItem{PMIX_MONITOR_FILE_DROPS}

\optattrend

%%%%
\descr

Request that a client be monitored for activity.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%  v3 Module Functions
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_get_cred_fn_t}}
\declareapi{pmix_server_get_cred_fn_t}

%%%%
\summary

Request a credential from the host environment.

%%%%
\format

\copySignature{pmix_server_get_cred_fn_t}{3.0}{
typedef pmix_status_t (*pmix_server_get_cred_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *proc, \\
\hspace*{29\sigspace}const pmix_info_t directives[], \\
\hspace*{29\sigspace}size_t ndirs, \\
\hspace*{29\sigspace}pmix_credential_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure of requesting process (handle)}
\argin{directives}{Array of info structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function to return the credential (\refapi{pmix_credential_cbfunc_t} function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
If the \ac{PMIx} library does not itself provide the requested credential, then it is required to pass any attributes provided by the client to the host environment for processing. In addition, it must include the following attributes in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_CRED_TYPE}
\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Request a credential from the host environment.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Credential callback function}
\declareapi{pmix_credential_cbfunc_t}

%%%%
\summary

Callback function to return a requested security credential

%%%%
\format

\copySignature{pmix_credential_cbfunc_t}{3.0}{
typedef void (*pmix_credential_cbfunc_t)( \\
\hspace*{20\sigspace}pmix_status_t status, \\
\hspace*{20\sigspace}pmix_byte_object_t *credential, \\
\hspace*{20\sigspace}pmix_info_t info[], size_t ninfo, \\
\hspace*{20\sigspace}void *cbdata);
}

\begin{arglist}
\argin{status}{\refstruct{pmix_status_t} value (handle)}
\argin{credential}{\refstruct{pmix_byte_object_t} structure containing the security credential (handle)}
\argin{info}{Array of provided by the system to pass any additional information about the credential - e.g., the identity of the issuing agent. (handle)}
\argin{ninfo}{Number of elements in \refarg{info} (\code{size_t})}
\argin{cbdata}{Object passed in original request (memory reference)}
\end{arglist}

%%%%
\descr

Define a callback function to return a requested security credential. Information provided by the issuing agent can subsequently be used
by the application for a variety of purposes. Examples include:

\begin{itemize}
    \item checking identified authorizations to determine what requests/operations are feasible as a means to steering \refterm{workflows}
    \item compare the credential type to that of the local SMS for compatibility
\end{itemize}

\adviceuserstart
The credential is opaque and therefore understandable only by a service compatible with the issuer. The \refarg{info} array is owned by the \ac{PMIx} library and is not to be released or altered by the receiving party.
\adviceuserend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_validate_cred_fn_t}}
\declareapi{pmix_server_validate_cred_fn_t}

%%%%
\summary

Request validation of a credential.

%%%%
\format

\copySignature{pmix_server_validate_cred_fn_t}{3.0}{
typedef pmix_status_t (*pmix_server_validate_cred_fn_t)( \\
\hspace*{29\sigspace}const pmix_proc_t *proc, \\
\hspace*{29\sigspace}const pmix_byte_object_t *cred, \\
\hspace*{29\sigspace}const pmix_info_t directives[], \\
\hspace*{29\sigspace}size_t ndirs, \\
\hspace*{29\sigspace}pmix_validation_cbfunc_t cbfunc, \\
\hspace*{29\sigspace}void *cbdata);
}

\begin{arglist}
\argin{proc}{\refstruct{pmix_proc_t} structure of requesting process (handle)}
\argin{cred}{Pointer to \refstruct{pmix_byte_object_t} containing the credential (handle)}
\argin{directives}{Array of info structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function to return the result (\refapi{pmix_validation_cbfunc_t} function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
If the \ac{PMIx} library does not itself validate the credential, then it is required to pass any attributes provided by the client to the host environment for processing. In addition, it must include the following attributes in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\divider

Host environments are not required to support any specific attributes.

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pasteAttributeItem{PMIX_TIMEOUT}

\optattrend

%%%%
\descr

Request validation of a credential obtained from the host environment via a prior call to the \refapi{pmix_server_get_cred_fn_t} module entry.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Credential validation callback function}
\declareapi{pmix_validation_cbfunc_t}

%%%%
\summary

Callback function for security credential validation.

%%%%
\format

\copySignature{pmix_validation_cbfunc_t}{3.0}{
typedef void (*pmix_validation_cbfunc_t)( \\
\hspace*{20\sigspace}pmix_status_t status, \\
\hspace*{20\sigspace}pmix_info_t info[], size_t ninfo, \\
\hspace*{20\sigspace}void *cbdata);
}

\begin{arglist}
\argin{status}{\refstruct{pmix_status_t} value (handle)}
\argin{info}{Array of \refstruct{pmix_info_t} provided by the system to pass any additional information about the authentication - e.g., the effective userid and group id of the certificate holder, and any related authorizations (handle)}
\argin{ninfo}{Number of elements in \refarg{info} (\code{size_t})}
\argin{cbdata}{Object passed in original request (memory reference)}
\end{arglist}

The returned status shall be one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request was processed and returned \textit{success} (i.e., the credential was both valid and any information it contained was successfully processed). Details of the result will be returned in the \refarg{info} array
    \item a PMIx error constant indicating either an error in the parsing of the credential or that the request was refused
\end{itemize}

%%%%
\descr

Define a validation callback function to indicate if a provided credential is valid, and any corresponding information regarding authorizations and other security matters.

\adviceuserstart
The precise contents of the array will depend on the host environment and its associated security system. At the minimum, it is expected (but not required) that the array will contain entries for the \refattr{PMIX_USERID} and \refattr{PMIX_GRPID} of the client described in the credential. The \refarg{info} array is owned by the \ac{PMIx} library and is not to be released or altered by the receiving party.
\adviceuserend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_iof_fn_t}}
\declareapi{pmix_server_iof_fn_t}

%%%%
\summary

Request the specified IO channels be forwarded from the given array of processes.

%%%%
\format

\copySignature{pmix_server_iof_fn_t}{3.0}{
typedef pmix_status_t (*pmix_server_iof_fn_t)( \\
\hspace*{24\sigspace}const pmix_proc_t procs[], \\
\hspace*{24\sigspace}size_t nprocs, \\
\hspace*{24\sigspace}const pmix_info_t directives[], \\
\hspace*{24\sigspace}size_t ndirs, \\
\hspace*{24\sigspace}pmix_iof_channel_t channels, \\
\hspace*{24\sigspace}pmix_op_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{procs}{Array \refstruct{pmix_proc_t} identifiers whose \ac{IO} is being requested (handle)}
\argin{nprocs}{Number of elements in \refarg{procs} (\code{size_t})}
\argin{directives}{Array of \refstruct{pmix_info_t} structures further defining the request (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{info} array (integer)}
\argin{channels}{Bitmask identifying the channels to be forwarded (\refstruct{pmix_iof_channel_t})}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the library must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}


\reqattrstart
The following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\divider

Host environments that provide this module entry point are required to support the following attributes:

\pasteAttributeItem{PMIX_IOF_CACHE_SIZE}
\pasteAttributeItem{PMIX_IOF_DROP_OLDEST}
\pasteAttributeItem{PMIX_IOF_DROP_NEWEST}

\reqattrend

\optattrstart
The following attributes may be supported by a host environment.

\pasteAttributeItem{PMIX_IOF_BUFFERING_SIZE}
\pasteAttributeItem{PMIX_IOF_BUFFERING_TIME}

\optattrend

%%%%
\descr

Request the specified IO channels be forwarded from the given array of processes. An error shall be returned in the callback function if the requested service from any of the requested processes cannot be provided.

\adviceimplstart
The forwarding of stdin is a \textit{push} process - processes cannot request that it be \textit{pulled} from some other source. Requests including the \refconst{PMIX_FWD_STDIN_CHANNEL} channel will return a \refconst{PMIX_ERR_NOT_SUPPORTED} error.
\adviceimplend

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{IOF delivery function}
\declareapi{pmix_iof_cbfunc_t}

%%%%
\summary

Callback function for delivering forwarded \ac{IO} to a process.

%%%%
\format

\copySignature{pmix_iof_cbfunc_t}{3.0}{
typedef void (*pmix_iof_cbfunc_t)( \\
\hspace*{20\sigspace}size_t iofhdlr, pmix_iof_channel_t channel, \\
\hspace*{20\sigspace}pmix_proc_t *source, pmix_byte_object_t *payload, \\
\hspace*{20\sigspace}pmix_info_t info[], size_t ninfo);
}

\begin{arglist}
\argin{iofhdlr}{Registration number of the handler being invoked (\code{size_t})}
\argin{channel}{bitmask identifying the channel the data arrived on (\refstruct{pmix_iof_channel_t})}
\argin{source}{Pointer to a \refstruct{pmix_proc_t} identifying the namespace/rank of the process that generated the data (\code{char*})}
\argin{payload}{Pointer to a \refstruct{pmix_byte_object_t} that describes the character array containing the data.}
\argin{info}{Array of \refstruct{pmix_info_t} provided by the source containing metadata about the payload. This could include \refattr{PMIX_IOF_COMPLETE} (handle)}
\argin{ninfo}{Number of elements in \refarg{info} (\code{size_t})}
\end{arglist}

%%%%
\descr

Define a callback function for delivering forwarded \ac{IO} to a process. This function will be called whenever data becomes available, or a
specified buffering size and/or time has been met.

\adviceuserstart
Multiple strings may be included in a given \refarg{payload}, and the \refarg{payload} may \textit{not} be \code{NULL} terminated. The user is responsible for releasing the \refarg{payload} memory. The \refarg{info} array is owned by the \ac{PMIx} library and is not to be released or altered by the receiving party.
\adviceuserend


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_stdin_fn_t}}
\declareapi{pmix_server_stdin_fn_t}

%%%%
\summary

Pass standard input data to the host environment for transmission to specified recipients.

%%%%
\format

\copySignature{pmix_server_stdin_fn_t}{3.0}{
typedef pmix_status_t (*pmix_server_stdin_fn_t)( \\
\hspace*{27\sigspace}const pmix_proc_t *source, \\
\hspace*{27\sigspace}const pmix_proc_t targets[], \\
\hspace*{27\sigspace}size_t ntargets, \\
\hspace*{27\sigspace}const pmix_info_t directives[], \\
\hspace*{27\sigspace}size_t ndirs, \\
\hspace*{27\sigspace}const pmix_byte_object_t *bo, \\
\hspace*{27\sigspace}pmix_op_cbfunc_t cbfunc, void *cbdata);
}

\begin{arglist}
\argin{source}{\refstruct{pmix_proc_t} structure of source process (handle)}
\argin{targets}{Array of \refstruct{pmix_proc_t} target identifiers (handle)}
\argin{ntargets}{Number of elements in the \refarg{targets} array (integer)}
\argin{directives}{Array of info structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{info} array (integer)}
\argin{bo}{Pointer to \refstruct{pmix_byte_object_t} containing the payload (handle)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the library must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
The following attributes are required to be included in the passed \refarg{info} array:

\pasteAttributeItem{PMIX_USERID}
\pasteAttributeItem{PMIX_GRPID}

\reqattrend

%%%%
\descr

Passes stdin to the host environment for transmission to specified recipients. The host environment is responsible for forwarding the data to all locations that host the specified \refarg{targets} and delivering the payload to the \ac{PMIx} server library connected to those clients.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_grp_fn_t}}
\declareapi{pmix_server_grp_fn_t}

%%%%
\summary

Request group operations (construct, destruct, etc.) on behalf of a set of processes.

%%%%
\format

\copySignature{pmix_server_grp_fn_t}{4.0}{
typedef pmix_status_t (*pmix_server_grp_fn_t)( \\
\hspace*{27\sigspace}pmix_group_operation_t op, \\
\hspace*{27\sigspace}char grp[], \\
\hspace*{27\sigspace}const pmix_proc_t procs[], \\
\hspace*{27\sigspace}size_t nprocs, \\
\hspace*{27\sigspace}const pmix_info_t directives[], \\
\hspace*{27\sigspace}size_t ndirs, \\
\hspace*{27\sigspace}pmix_info_cbfunc_t cbfunc, \\
\hspace*{27\sigspace}void *cbdata);
}

\begin{arglist}
\argin{op}{\refstruct{pmix_group_operation_t} value indicating operation the host is requested to perform (integer)}
\argin{grp}{Character string identifying the group (string)}
\argin{procs}{Array of \refstruct{pmix_proc_t} identifiers of participants (handle)}
\argin{nprocs}{Number of elements in the \refarg{procs} array (integer)}
\argin{directives}{Array of info structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_info_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the library must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
The following attributes are required to be supported by a host environment.

\pasteAttributeItem{PMIX_LOCAL_COLLECTIVE_STATUS}
\reqattrend

\optattrstart
The following attributes may be supported by a host environment.

\pasteAttributeItem{PMIX_GROUP_ASSIGN_CONTEXT_ID}
\pasteAttributeItem{PMIX_GROUP_LOCAL_ONLY}
\pasteAttributeItem{PMIX_GROUP_ENDPT_DATA}
\pasteAttributeItem{PMIX_GROUP_OPTIONAL}
\pasteAttributeItem{PMIX_RANGE}


The following attributes may be included in the host's response:

\pasteAttributeItem{PMIX_GROUP_ID}
\pasteAttributeItem{PMIX_GROUP_MEMBERSHIP}
\pasteAttributeItem{PMIX_GROUP_CONTEXT_ID}
\pasteAttributeItem{PMIX_GROUP_ENDPT_DATA}

\optattrend

%%%%
\descr

Perform the specified operation across the identified processes, plus any special actions included in the directives. Return the result of any special action requests in the callback function when the operation is completed. Actions may include a request (\refattr{PMIX_GROUP_ASSIGN_CONTEXT_ID}~) that the host assign a unique numerical (size_t) ID to this group - if given, the \refattr{PMIX_RANGE} attribute will specify the range across which the ID must be unique (default to \refconst{PMIX_RANGE_SESSION}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Group Operation Constants}
\declarestruct{pmix_group_operation_t}

\versionMarker{4.0}
The \refstruct{pmix_group_operation_t} structure is a \code{uint8_t} value for specifying group operations. All values were originally defined in version 4 of the standard unless otherwise marked.

\begin{constantdesc}
%
\declareconstitemvalue{PMIX_GROUP_CONSTRUCT}{0}
Construct a group composed of the specified processes - used by a \ac{PMIx} server library to direct host operation.
%
\declareconstitemvalue{PMIX_GROUP_DESTRUCT}{1}
Destruct the specified group - used by a \ac{PMIx} server library to direct host operation.
%
\end{constantdesc}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\code{pmix_server_fabric_fn_t}}
\declareapi{pmix_server_fabric_fn_t}

%%%%
\summary

Request fabric-related operations (e.g., information on a fabric) on behalf of a tool or other process.

%%%%
\format

\copySignature{pmix_server_fabric_fn_t}{4.0}{
typedef pmix_status_t (*pmix_server_fabric_fn_t)( \\
\hspace*{27\sigspace}const pmix_proc_t *requestor, \\
\hspace*{27\sigspace}pmix_fabric_operation_t op, \\
\hspace*{27\sigspace}const pmix_info_t directives[], \\
\hspace*{27\sigspace}size_t ndirs, \\
\hspace*{27\sigspace}pmix_info_cbfunc_t cbfunc, \\
\hspace*{27\sigspace}void *cbdata);
}

\begin{arglist}
\argin{requestor}{\refstruct{pmix_proc_t} identifying the requestor (handle)}
\argin{op}{\refstruct{pmix_fabric_operation_t} value indicating operation the host is requested to perform (integer)}
\argin{directives}{Array of info structures (array of handles)}
\argin{ndirs}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_info_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns one of the following:

\begin{itemize}
    \item \refconst{PMIX_SUCCESS}, indicating that the request is being processed by the host environment - result will be returned in the provided \refarg{cbfunc}. Note that the library must not invoke the callback function prior to returning from the \ac{API}.
    \item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will not be called
    \item \refconst{PMIX_ERR_NOT_SUPPORTED}, indicating that the host environment does not support the request, even though the function entry was provided in the server module - the \refarg{cbfunc} will not be called
    \item a PMIx error constant indicating either an error in the input or that the request was immediately processed and failed - the \refarg{cbfunc} will not be called
\end{itemize}

\reqattrstart
The following directives are required to be supported by all hosts to aid users in identifying the fabric and (if applicable) the device to whom the operation references:

\pasteAttributeItem{PMIX_FABRIC_VENDOR}
\pasteAttributeItem{PMIX_FABRIC_IDENTIFIER}
\pasteAttributeItem{PMIX_FABRIC_PLANE}
\pasteAttributeItem{PMIX_FABRIC_DEVICE_INDEX}

\reqattrend

%%%%
\descr

Perform the specified operation. Return the result of any requests in the callback function when the operation is completed. Operations may, for example, include a request for fabric information. See \refstruct{pmix_fabric_t} for a list of expected information to be included in the response. Note that requests for device index are to be returned in the callback function's array of \refstruct{pmix_info_t} using the \refattr{PMIX_FABRIC_DEVICE_INDEX} attribute.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%