From 5dd2f622562fe4f7893d37a5fcbdd7c74cfcf8f2 Mon Sep 17 00:00:00 2001 From: David Solt Date: Mon, 24 Apr 2023 12:40:00 -0500 Subject: [PATCH 1/4] First pass at IAWG pass at pack/unpack chapter Signed-off-by: David Solt --- Chap_API_Data_Mgmt.tex | 147 +++++++++++++++++++++-------------------- 1 file changed, 77 insertions(+), 70 deletions(-) diff --git a/Chap_API_Data_Mgmt.tex b/Chap_API_Data_Mgmt.tex index 8bd23acd..1f95818e 100644 --- a/Chap_API_Data_Mgmt.tex +++ b/Chap_API_Data_Mgmt.tex @@ -4,7 +4,7 @@ \chapter{Data Packing and Unpacking} \label{chap:api_data_mgmt} -\ac{PMIx} intentionally does not include support for internode communications in the standard, instead relying on its host \ac{SMS} environment to transfer any needed data and/or requests between nodes. These operations frequently involve PMIx-defined public data structures that include binary data. Many \ac{HPC} clusters are homogeneous, and so transferring the structures can be done rather simply. However, greater effort is required in heterogeneous environments to ensure binary data is correctly transferred. \ac{PMIx} buffer manipulation functions are provided for this purpose via standardized interfaces to ease adoption. +\ac{PMIx} intentionally does not include support for internode communications in the standard, instead relying on its host \ac{SMS} environment to transfer any needed data and/or requests between nodes. However, to support \ac{SMS} environments which must frequently transfer \ac{PMIx} data structures between nodes and client applications that need to store or transfer \ac{PMIx} data structures, \ac{PMIx} provides the \acs{API} presented in this chapter. These operations frequently involve PMIx-defined data structures that include data that may have different binary representations on different hosts. Many \ac{HPC} clusters are homogeneous, and so transferring the structures can be done rather simply. However, greater effort is required in heterogeneous environments to ensure binary data is correctly transferred. \ac{PMIx} buffer manipulation functions are provided for this purpose via standardized interfaces to ease adoption. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Data Buffer Type} @@ -91,7 +91,7 @@ \section{Support Macros} \littleheader{\code{PMIX_DATA_BUFFER_CONSTRUCT}} \declaremacro{PMIX_DATA_BUFFER_CONSTRUCT} -Initialize a statically declared \refstruct{pmix_data_buffer_t} object. +Initialize the fields of a previously allocated \refstruct{pmix_data_buffer_t} object. \copySignature{PMIX_DATA_BUFFER_CONSTRUCT}{2.0}{ PMIX_DATA_BUFFER_CONSTRUCT(buffer); @@ -199,14 +199,16 @@ \subsection{\code{PMIx_Data_pack}} The pack function packs one or more values of a specified type into the specified buffer. The buffer must have already been initialized via the \refmacro{PMIX_DATA_BUFFER_CREATE} or \refmacro{PMIX_DATA_BUFFER_CONSTRUCT} -macros --- otherwise, \refapi{PMIx_Data_pack} will return an error. +macros --- otherwise, \refapi{PMIx_Data_pack} will return an error. +The buffer may have already contain packed data, in which case the new data is appended to the buffer. Providing an unsupported type flag will likewise be reported as an error. -Note that any data to be packed that is not hard type cast (i.e., -not type cast to a specific size) may lose precision when unpacked +Note that packed packing data using a type that +does not explicitly specifies its size may +may lose precision when unpacked by a non-homogeneous recipient. The \refapi{PMIx_Data_pack} function will do its best to deal with heterogeneity issues between the packer and unpacker in such -cases. Sending a number larger than can be handled by the recipient +cases. Sending a value larger than can be handled by the recipient will return an error code (generated upon unpacking) --- the error cannot be detected during packing. @@ -243,11 +245,12 @@ \subsection{\code{PMIx_Data_unpack}} \begin{arglist} -\argin{source}{Pointer to a \refstruct{pmix_proc_t} structure containing the nspace/rank of the process that packed the provided buffer. A NULL value may be used to indicate that the source is based on the same \ac{PMIx} version as the caller. Note that only the source's nspace is relevant. (handle)} +\argin{source}{Pointer to a \refstruct{pmix_proc_t} structure containing the description of the process that packed the provided buffer. A NULL value may be used to indicate that the source is based on the same \ac{PMIx} version as the caller. +i. The rank may be set to \refconst{PMIX_RANK_WILDCARD} as only the namespace is used to determine the packing version as all processes in a namespace are required to be using the same \ac{PMIx} version. (handle)} \argin{buffer}{A pointer to the buffer from which the value will be extracted. (handle)} -\arginout{dest}{A pointer to the memory location into which the data is to be stored. Note that these values will be stored contiguously in memory. For strings, this pointer must be to (char**) to provide a means of supporting multiple string operations. The unpack function will allocate memory for each string in the array - the caller must only provide adequate memory for the array of pointers. (\code{void*})} -\arginout{max_num_values}{The number of values to be unpacked --- upon completion, the parameter will be set to the actual number of values unpacked. In most cases, this should match the maximum number provided in the parameters --- but in no case will it exceed the value of this parameter. Note that unpacking fewer values than are actually available will leave the buffer in an unpackable state --- the function will return an error code to warn of this condition.(\code{int32_t})} -\argin{type}{The type of the data to be unpacked --- must be one of the \ac{PMIx} defined data types (\refstruct{pmix_data_type_t})} +\arginout{dest}{A pointer to the memory location into which the data is to be stored. Note that these values will be stored contiguously in memory. For strings, this pointer must be to (char**) to provide a means of supporting multiple string operations. The unpack function will allocate memory for each string in the array, but the caller must provide adequate memory for the array of pointers. (\code{void*})} +\arginout{max_num_values}{The number of values to be unpacked. Upon completion, the parameter will be set to the actual number of values unpacked. In most cases, this should match the maximum number provided in the parameters, but in no case will it exceed the value of this parameter. Note that unpacking fewer values than are actually available will leave the buffer in an unpackable state and the function will return an error code to warn of this condition.(\code{int32_t})} +\argin{type}{The type of the data to be unpacked. Must be one of the \ac{PMIx} defined data types (\refstruct{pmix_data_type_t})} \end{arglist} \returnstart @@ -261,23 +264,22 @@ \subsection{\code{PMIx_Data_unpack}} %%%% \descr -The unpack function unpacks the next value (or values) of a specified type from the given buffer. The buffer must have already been initialized via an \refmacro{PMIX_DATA_BUFFER_CREATE} or \refmacro{PMIX_DATA_BUFFER_CONSTRUCT} call (and assumedly filled with some data) --- otherwise, the unpack_value function will return an error. Providing an unsupported type flag will likewise be reported as an error, as will specifying a data type that \textit{does not} match the type of the next item in the buffer. An attempt to read beyond the end of the stored data held in the buffer will also return an error. - -Note that it is possible for the buffer to be corrupted and that \ac{PMIx} will \textit{think} there is a proper variable type at the beginning of an unpack region --- but that the value is bogus (e.g., just a byte field in a string array that so happens to have a value that matches the specified data type flag). Therefore, the data type error check is \textit{not} completely safe. +The unpack function unpacks the next value (or values) from the given buffer. Providing an unsupported type flag +or specifying a data type that \textit{does not} match the type of the next item in the buffer will be reported as an error. +An attempt to read an uninitialized buffer or read beyond the end of the stored data held in the buffer will also return an error. Unpacking values is a "nondestructive" process --- i.e., the values are not removed from the buffer. It is therefore possible for the caller to re-unpack a value from the same buffer by resetting the unpack_ptr. Warning: The caller is responsible for providing adequate memory storage for the requested data. The user must provide a parameter indicating the maximum number of values that can be unpacked into the allocated memory. If more values exist in the buffer than can fit into the memory storage, then the function will unpack what it can fit into that location and return an error code indicating that the buffer was only partially unpacked. -Note that any data that was not hard type cast (i.e., not type cast to a specific size) when packed may lose precision when unpacked by a non-homogeneous recipient. \ac{PMIx} will do its best to deal with heterogeneity issues between the packer and unpacker in such cases. Sending a number larger than can be handled by the recipient will return an error code generated upon unpacking --- these errors cannot be detected during packing. +Note that any data that is packed using a type that does not explicitly specifies its size may lose precision when unpacked by a non-homogeneous recipient. \ac{PMIx} will do its best to deal with heterogeneity issues between the packer and unpacker in such cases. Sending a number larger than can be handled by the recipient will return an error code generated upon unpacking --- these errors cannot be detected during packing. The namespace of the process that packed the buffer is used solely to resolve any data type differences between \ac{PMIx} versions. The packer must, therefore, be -known to the user prior to calling the pack function so that the -\ac{PMIx} library is aware of the version the packer is using. Note that +known to the caller prior to calling the unpack function so that the +\ac{PMIx} library is aware of the version the packer used. Note that all processes in a given namespace are \textit{required} to use the same \ac{PMIx} -version --- thus, the caller must only know at least one process from the -packer's namespace. +version --- thus the target rank may be set to \refconst{PMIX_RANK_WILDCARD}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -315,7 +317,7 @@ \subsection{\code{PMIx_Data_copy}} %%%% \descr -Since registered data types can be complex structures, the system needs some way to know how to copy the data from one location to another (e.g., for storage in the registry). This function, which can call other copy functions to build up complex data types, defines the method for making a copy of the specified data type. +\ac{PMIx} provides \refapi{PMIx_Data_copy} to copy \ac{PMIx} data types. If the specified \refarg{type} is composed of multilpe \ac{PMIx} data types, the function will ensure that each of these internal portions are also copied correctly as part of the operation. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -337,7 +339,7 @@ \subsection{\code{PMIx_Data_print}} } \begin{arglist} -\argin{output}{The address of a pointer into which the address of the resulting output is to be stored. (\code{char**})} +\argout{output}{The address of a pointer into which the address of the resulting output is to be stored. (\code{char**})} \argin{prefix}{String to be prepended to the resulting output (\code{char*})} \argin{src}{A pointer to the memory location of the data value to be printed (handle)} \argin{type}{The type of the data value to be printed --- must be one of the PMIx defined data types. (\refstruct{pmix_data_type_t})} @@ -352,7 +354,7 @@ \subsection{\code{PMIx_Data_print}} %%%% \descr -Since registered data types can be complex structures, the system needs some way to know how to print them (i.e., convert them to a string representation). Primarily for debug purposes. +Since registered data types can be complex structures, the \ac{PMIx} provides \refapi{PMIx_Data_print} to print them (i.e., convert them to a string representation). Primarily for debug purposes. Note that the format of the resulting string may vary from one implementation to another. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -390,15 +392,14 @@ \subsection{\code{PMIx_Data_copy_payload}} This function will append a copy of the payload in one buffer into another buffer. Note that this is \textit{not} a destructive procedure --- the source buffer's payload will remain intact, as will any pre-existing payload in the destination's buffer. Only the unpacked portion of the source payload will be copied. - %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsection{\code{PMIx_Data_load}} -\declareapi{PMIx_Data_load} +\subsection{\code{PMIx_Data_unload}} +\declareapi{PMIx_Data_unload} %%%% \summary -Load a buffer with the provided payload +Unload a buffer into a byte object %%%% \format @@ -407,49 +408,49 @@ \subsection{\code{PMIx_Data_load}} \cspecificstart \begin{codepar} pmix_status_t -PMIx_Data_load(pmix_data_buffer_t *dest, - pmix_byte_object_t *src); +PMIx_Data_unload(pmix_data_buffer_t *src, + pmix_byte_object_t *dest); \end{codepar} \cspecificend \begin{arglist} -\argin{dest}{Pointer to the destination \refstruct{pmix_data_buffer_t} (handle)} -\argin{src}{Pointer to the source \refstruct{pmix_byte_object_t} (handle)} +\argin{src}{Pointer to the source \refstruct{pmix_data_buffer_t} (handle)} +\argin{dest}{Pointer to the destination \refstruct{pmix_byte_object_t} (handle)} \end{arglist} -Returns one of the following: +\returnstart \begin{constantdesc} -\item \refconst{PMIX_SUCCESS} The data has been loaded as requested -\item \refconst{PMIX_ERR_BAD_PARAM} The \refarg{dest} structure pointer is \code{NULL} -\item \refconst{PMIX_ERR_NOT_SUPPORTED} The \ac{PMIx} implementation does not support this function. +\item \refconst{PMIX_ERR_BAD_PARAM} The destination and/or source pointer is \code{NULL} \end{constantdesc} +\returnend %%%% \descr -The load function allows the caller to transfer the contents of the \refarg{src} -\refstruct{pmix_byte_object_t} to the \refarg{dest} target buffer. If a payload -already exists in the buffer, the function will "free" the existing data to -release it, and then replace the data payload with the one provided -by the caller. +The unload function provides the caller with a pointer to the +portion of the data payload within the buffer that has not yet been +unpacked, along with the size of that region. Any portion of +the payload that was previously unpacked using the \refapi{PMIx_Data_unpack} +routine will not be included in the unload operation. This operation allows the user to directly access the payload of a \refstruct{pmix_data_buffer_t}. \adviceuserstart -The buffer must be allocated or constructed in advance - failing to do so -will cause the load function to return an error code. - -The caller is responsible for pre-packing the provided -payload. For example, the load function cannot convert to network byte order -any data contained in the provided payload. +This is a destructive operation. While the payload returned in the +destination \refstruct{pmix_byte_object_t} is +undisturbed, the function will clear the \refarg{src}'s pointers to the +payload. Thus, the \refarg{src} and the payload are completely separated, +leaving the caller able to free or destruct the \refarg{src}. \adviceuserend + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsection{\code{PMIx_Data_unload}} -\declareapi{PMIx_Data_unload} +\subsection{\code{PMIx_Data_load}} +\declareapi{PMIx_Data_load} %%%% \summary -Unload a buffer into a byte object +Load a buffer with the provided payload %%%% \format @@ -458,39 +459,40 @@ \subsection{\code{PMIx_Data_unload}} \cspecificstart \begin{codepar} pmix_status_t -PMIx_Data_unload(pmix_data_buffer_t *src, - pmix_byte_object_t *dest); +PMIx_Data_load(pmix_data_buffer_t *dest, + pmix_byte_object_t *src); \end{codepar} \cspecificend \begin{arglist} -\argin{src}{Pointer to the source \refstruct{pmix_data_buffer_t} (handle)} -\argin{dest}{Pointer to the destination \refstruct{pmix_byte_object_t} (handle)} +\argin{dest}{Pointer to the destination \refstruct{pmix_data_buffer_t} (handle)} +\argin{src}{Pointer to the source \refstruct{pmix_byte_object_t} (handle)} \end{arglist} -\returnstart +Returns one of the following: \begin{constantdesc} -\item \refconst{PMIX_ERR_BAD_PARAM} The destination and/or source pointer is \code{NULL} +\item \refconst{PMIX_SUCCESS} The data has been loaded as requested +\item \refconst{PMIX_ERR_BAD_PARAM} The \refarg{dest} structure pointer is \code{NULL} +\item \refconst{PMIX_ERR_NOT_SUPPORTED} The \ac{PMIx} implementation does not support this function. \end{constantdesc} -\returnend %%%% \descr -The unload function provides the caller with a pointer to the -portion of the data payload within the buffer that has not yet been -unpacked, along with the size of that region. Any portion of -the payload that was previously unpacked using the \refapi{PMIx_Data_unpack} -routine will be ignored. This allows the user to directly access the payload. +The load function allows the caller to transfer the contents of the \refarg{src} +\refstruct{pmix_byte_object_t} to the \refarg{dest} target buffer. If a payload +already exists in the buffer, the function will "free" the existing data to +release it, and then replace the data payload with the one provided +by the caller. \adviceuserstart -This is a destructive operation. While the payload returned in the -destination \refstruct{pmix_byte_object_t} is -undisturbed, the function will clear the \refarg{src}'s pointers to the -payload. Thus, the \refarg{src} and the payload are completely separated, -leaving the caller able to free or destruct the \refarg{src}. -\adviceuserend +The buffer must be allocated or constructed in advance - failing to do so +will cause the load function to return an error code. +The caller is responsible for pre-packing the provided +payload. For example, the load function cannot convert to network byte order +any data contained in the provided payload. +\adviceuserend %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -533,7 +535,10 @@ \subsection{\code{PMIx_Data_compress}} Compress the provided data block. Destination memory will be allocated if operation is successfully concluded. Caller is responsible for release of the allocated region. The input -data block will remain unaltered. +data block will remain unaltered. +The method of compressing data is implementation dependent. +Data compressed by one implementation should not be uncompressed by a different implementation unless +the documentation of the two implementations explicitly support doing so. Note: the compress function will return \code{False} if the operation would not result in a smaller data block. @@ -562,9 +567,9 @@ \subsection{\code{PMIx_Data_decompress}} \cspecificend \begin{arglist} -\argout{outbytes}{Address where the pointer to the decompressed data region is to be returned (handle)} -\argout{nbytes}{Address where the number of bytes in the decompressed data region is to be returned (handle)} \argin{inbytes}{Pointer to the source data (handle)} +\argout{nbytes}{Address where the number of bytes in the decompressed data region is to be returned (handle)} +\argout{outbytes}{Address where the pointer to the decompressed data region is to be returned (handle)} \argin{size}{Number of bytes in the source data region (\code{size_t})} \end{arglist} @@ -581,11 +586,13 @@ \subsection{\code{PMIx_Data_decompress}} will be allocated if operation is successfully concluded. Caller is responsible for release of the allocated region. The input data block will remain unaltered. +Data compressed by one implementation should not be uncompressed by a different implementation unless +the documentation of the two implementations explicitly support doing so. Only data compressed by the \refapi{PMIx_Data_compress} \ac{API} can be decompressed by this function. Passing data that has not been compressed by \refapi{PMIx_Data_compress} will lead to -unexpected and potentially catastrophic results. +undefined behavior. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -625,6 +632,6 @@ \subsection{\code{PMIx_Data_embed}} \descr The embed function is identical in operation to \refapi{PMIx_Data_load} -except that it does \emph{not} clear the payload object upon completion. +except that it does \emph{not} clear the payload object upon completion. The data is copied from the payload to the buffer and remains in the payload. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% From dd0acbb1320b882aab1931b800b6b7d67df1e48a Mon Sep 17 00:00:00 2001 From: David Solt Date: Thu, 4 May 2023 15:44:59 -0500 Subject: [PATCH 2/4] Simplify explanation of compressing being implementation dependant + minor grammar issue Signed-off-by: David Solt --- Chap_API_Data_Mgmt.tex | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/Chap_API_Data_Mgmt.tex b/Chap_API_Data_Mgmt.tex index fa575ba6..57edc1a1 100644 --- a/Chap_API_Data_Mgmt.tex +++ b/Chap_API_Data_Mgmt.tex @@ -203,7 +203,7 @@ \subsection{\code{PMIx_Data_pack}} The buffer may have already contain packed data, in which case the new data is appended to the buffer. Providing an unsupported type flag will likewise be reported as an error. -Note that packed packing data using a type that +Note that packing data using a type that does not explicitly specifies its size may may lose precision when unpacked by a non-homogeneous recipient. The \refapi{PMIx_Data_pack} function will do its best to deal @@ -246,7 +246,7 @@ \subsection{\code{PMIx_Data_unpack}} \begin{arglist} \argin{source}{Pointer to a \refstruct{pmix_proc_t} structure containing the description of the process that packed the provided buffer. A NULL value may be used to indicate that the source is based on the same \ac{PMIx} version as the caller. -i. The rank may be set to \refconst{PMIX_RANK_WILDCARD} as only the namespace is used to determine the packing version as all processes in a namespace are required to be using the same \ac{PMIx} version. (handle)} +i. The rank may be set to \refconst{PMIX_RANK_WILDCARD} as only the namespace is used to determine the packing version as all processes in a namespace are required to use the same \ac{PMIx} version. (handle)} \argin{buffer}{A pointer to the buffer from which the value will be extracted. (handle)} \arginout{dest}{A pointer to the memory location into which the data is to be stored. Note that these values will be stored contiguously in memory. For strings, this pointer must be to (char**) to provide a means of supporting multiple string operations. The unpack function will allocate memory for each string in the array, but the caller must provide adequate memory for the array of pointers. (\code{void*})} \arginout{max_num_values}{The number of values to be unpacked. Upon completion, the parameter will be set to the actual number of values unpacked. In most cases, this should match the maximum number provided in the parameters, but in no case will it exceed the value of this parameter. Note that unpacking fewer values than are actually available will leave the buffer in an unpackable state and the function will return an error code to warn of this condition.(\code{int32_t})} @@ -340,7 +340,7 @@ \subsection{\code{PMIx_Data_print}} \begin{arglist} \argout{output}{The address of a pointer into which the address of the resulting output is to be stored. (\code{char**})} -\argin{prefix}{String to be prepended to the resulting output (\code{const char*} +\argin{prefix}{String to be prepended to the resulting output (\code{const char*})} \argin{src}{A pointer to the memory location of the data value to be printed (handle)} \argin{type}{The type of the data value to be printed --- must be one of the PMIx defined data types. (\refstruct{pmix_data_type_t})} \end{arglist} @@ -536,9 +536,7 @@ \subsection{\code{PMIx_Data_compress}} will be allocated if operation is successfully concluded. Caller is responsible for release of the allocated region. The input data block will remain unaltered. -The method of compressing data is implementation dependent. -Data compressed by one implementation should not be uncompressed by a different implementation unless -the documentation of the two implementations explicitly support doing so. +The method of compressing and uncompressing data is implementation dependent. Note: the compress function will return \code{False} if the operation would not result in a smaller data block. @@ -586,8 +584,7 @@ \subsection{\code{PMIx_Data_decompress}} will be allocated if operation is successfully concluded. Caller is responsible for release of the allocated region. The input data block will remain unaltered. -Data compressed by one implementation should not be uncompressed by a different implementation unless -the documentation of the two implementations explicitly support doing so. +The method of compressing and uncompressing data is implementation dependent. Only data compressed by the \refapi{PMIx_Data_compress} \ac{API} can be decompressed by this function. Passing data that has not From a7f83fac6150c87504a5bbb071db3b8f421b60ca Mon Sep 17 00:00:00 2001 From: David Solt Date: Mon, 1 Jul 2024 14:01:28 -0500 Subject: [PATCH 3/4] Changes from first attempted (failed) reading Signed-off-by: David Solt --- Chap_API_Data_Mgmt.tex | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/Chap_API_Data_Mgmt.tex b/Chap_API_Data_Mgmt.tex index 57edc1a1..91925535 100644 --- a/Chap_API_Data_Mgmt.tex +++ b/Chap_API_Data_Mgmt.tex @@ -4,7 +4,7 @@ \chapter{Data Packing and Unpacking} \label{chap:api_data_mgmt} -\ac{PMIx} intentionally does not include support for internode communications in the standard, instead relying on its host \ac{SMS} environment to transfer any needed data and/or requests between nodes. However, to support \ac{SMS} environments which must frequently transfer \ac{PMIx} data structures between nodes and client applications that need to store or transfer \ac{PMIx} data structures, \ac{PMIx} provides the \acs{API} presented in this chapter. These operations frequently involve PMIx-defined data structures that include data that may have different binary representations on different hosts. Many \ac{HPC} clusters are homogeneous, and so transferring the structures can be done rather simply. However, greater effort is required in heterogeneous environments to ensure binary data is correctly transferred. \ac{PMIx} buffer manipulation functions are provided for this purpose via standardized interfaces to ease adoption. +\ac{PMIx} intentionally does not include support for internode communications in the standard, instead relying on its host \ac{SMS} environment to transfer any needed data and/or requests between nodes. However, to support \ac{SMS} environments which must frequently transfer \ac{PMIx} data structures between nodes, \ac{PMIx} provides the \acs{API} presented in this chapter. These operations frequently involve PMIx-defined data structures that include data that may have different binary representations on different hosts. Many \ac{HPC} clusters are homogeneous, and so transferring the structures can be done rather simply. However, greater effort is required in heterogeneous environments to ensure binary data is correctly transferred. \ac{PMIx} buffer manipulation functions are provided for this purpose via standardized interfaces to ease adoption. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Data Buffer Type} @@ -199,16 +199,16 @@ \subsection{\code{PMIx_Data_pack}} The pack function packs one or more values of a specified type into the specified buffer. The buffer must have already been initialized via the \refmacro{PMIX_DATA_BUFFER_CREATE} or \refmacro{PMIX_DATA_BUFFER_CONSTRUCT} -macros --- otherwise, \refapi{PMIx_Data_pack} will return an error. -The buffer may have already contain packed data, in which case the new data is appended to the buffer. +macros. +The buffer may already contain packed data, in which case the new data is appended to the buffer. Providing an unsupported type flag will likewise be reported as an error. Note that packing data using a type that -does not explicitly specifies its size may +does not explicitly specifiy its size may lose precision when unpacked by a non-homogeneous recipient. The \refapi{PMIx_Data_pack} function will do its best to deal with heterogeneity issues between the packer and unpacker in such -cases. Sending a value larger than can be handled by the recipient +cases. Sending a value outside the range of values that can be handled by the recipient will return an error code (generated upon unpacking) --- the error cannot be detected during packing. @@ -245,8 +245,8 @@ \subsection{\code{PMIx_Data_unpack}} \begin{arglist} -\argin{source}{Pointer to a \refstruct{pmix_proc_t} structure containing the description of the process that packed the provided buffer. A NULL value may be used to indicate that the source is based on the same \ac{PMIx} version as the caller. -i. The rank may be set to \refconst{PMIX_RANK_WILDCARD} as only the namespace is used to determine the packing version as all processes in a namespace are required to use the same \ac{PMIx} version. (handle)} +\argin{source}{Pointer to a \refstruct{pmix_proc_t} structure of the process that packed the provided buffer. A NULL value may be used to indicate that the source is based on the same \ac{PMIx} version as the caller. +i. Only the namespace is used to determine the packing version as all processes in a namespace are required to use the same \ac{PMIx} version. (handle)} \argin{buffer}{A pointer to the buffer from which the value will be extracted. (handle)} \arginout{dest}{A pointer to the memory location into which the data is to be stored. Note that these values will be stored contiguously in memory. For strings, this pointer must be to (char**) to provide a means of supporting multiple string operations. The unpack function will allocate memory for each string in the array, but the caller must provide adequate memory for the array of pointers. (\code{void*})} \arginout{max_num_values}{The number of values to be unpacked. Upon completion, the parameter will be set to the actual number of values unpacked. In most cases, this should match the maximum number provided in the parameters, but in no case will it exceed the value of this parameter. Note that unpacking fewer values than are actually available will leave the buffer in an unpackable state and the function will return an error code to warn of this condition.(\code{int32_t})} @@ -264,22 +264,22 @@ \subsection{\code{PMIx_Data_unpack}} %%%% \descr -The unpack function unpacks the next value (or values) from the given buffer. Providing an unsupported type flag -or specifying a data type that \textit{does not} match the type of the next item in the buffer will be reported as an error. -An attempt to read an uninitialized buffer or read beyond the end of the stored data held in the buffer will also return an error. +The unpack function unpacks the next value (or values) from the given buffer. +An attempt to read an uninitialized buffer or read beyond the end of the stored data held in the buffer will return an error. Unpacking values is a "nondestructive" process --- i.e., the values are not removed from the buffer. It is therefore possible for the caller to re-unpack a value from the same buffer by resetting the unpack_ptr. Warning: The caller is responsible for providing adequate memory storage for the requested data. The user must provide a parameter indicating the maximum number of values that can be unpacked into the allocated memory. If more values exist in the buffer than can fit into the memory storage, then the function will unpack what it can fit into that location and return an error code indicating that the buffer was only partially unpacked. -Note that any data that is packed using a type that does not explicitly specifies its size may lose precision when unpacked by a non-homogeneous recipient. \ac{PMIx} will do its best to deal with heterogeneity issues between the packer and unpacker in such cases. Sending a number larger than can be handled by the recipient will return an error code generated upon unpacking --- these errors cannot be detected during packing. +Note that any data that is packed using a type that does not explicitly specify its size may lose precision when unpacked by a non-homogeneous recipient. \ac{PMIx} will do its best to deal with heterogeneity issues between the packer and unpacker in such cases. Sending a value outside the range of values that can be handled by the recipient will return an error code generated upon unpacking --- these errors cannot be detected during packing. The namespace of the process that packed the buffer is used solely to resolve any data type differences between \ac{PMIx} versions. The packer must, therefore, be known to the caller prior to calling the unpack function so that the \ac{PMIx} library is aware of the version the packer used. Note that all processes in a given namespace are \textit{required} to use the same \ac{PMIx} -version --- thus the target rank may be set to \refconst{PMIX_RANK_WILDCARD}. +version --- thus, the caller must only know at least one process from the +packer's namespace %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -354,7 +354,7 @@ \subsection{\code{PMIx_Data_print}} %%%% \descr -Since registered data types can be complex structures, the \ac{PMIx} provides \refapi{PMIx_Data_print} to print them (i.e., convert them to a string representation). Primarily for debug purposes. Note that the format of the resulting string may vary from one implementation to another. +Since registered data types can be complex structures, \ac{PMIx} provides \refapi{PMIx_Data_print} to print them (i.e., convert them to a string representation). Primarily for debug purposes. Note that the format of the resulting string may vary from one implementation to another. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -566,9 +566,9 @@ \subsection{\code{PMIx_Data_decompress}} \begin{arglist} \argin{inbytes}{Pointer to the source data (handle)} -\argout{nbytes}{Address where the number of bytes in the decompressed data region is to be returned (handle)} -\argout{outbytes}{Address where the pointer to the decompressed data region is to be returned (handle)} \argin{size}{Number of bytes in the source data region (\code{size_t})} +\argout{outbytes}{Address where the pointer to the decompressed data region is to be returned (handle)} +\argout{nbytes}{Address where the number of bytes in the decompressed data region is to be returned (handle)} \end{arglist} Returns one of the following: From 2e5d76ed5c36050686436d1547a5e75a724836ef Mon Sep 17 00:00:00 2001 From: David Solt Date: Tue, 16 Jul 2024 11:55:44 -0500 Subject: [PATCH 4/4] Stray "i." characters and restore accidentally deleted '.' Signed-off-by: David G. Solt --- Chap_API_Data_Mgmt.tex | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Chap_API_Data_Mgmt.tex b/Chap_API_Data_Mgmt.tex index 91925535..aba0359e 100644 --- a/Chap_API_Data_Mgmt.tex +++ b/Chap_API_Data_Mgmt.tex @@ -246,7 +246,7 @@ \subsection{\code{PMIx_Data_unpack}} \begin{arglist} \argin{source}{Pointer to a \refstruct{pmix_proc_t} structure of the process that packed the provided buffer. A NULL value may be used to indicate that the source is based on the same \ac{PMIx} version as the caller. -i. Only the namespace is used to determine the packing version as all processes in a namespace are required to use the same \ac{PMIx} version. (handle)} +Only the namespace is used to determine the packing version as all processes in a namespace are required to use the same \ac{PMIx} version. (handle)} \argin{buffer}{A pointer to the buffer from which the value will be extracted. (handle)} \arginout{dest}{A pointer to the memory location into which the data is to be stored. Note that these values will be stored contiguously in memory. For strings, this pointer must be to (char**) to provide a means of supporting multiple string operations. The unpack function will allocate memory for each string in the array, but the caller must provide adequate memory for the array of pointers. (\code{void*})} \arginout{max_num_values}{The number of values to be unpacked. Upon completion, the parameter will be set to the actual number of values unpacked. In most cases, this should match the maximum number provided in the parameters, but in no case will it exceed the value of this parameter. Note that unpacking fewer values than are actually available will leave the buffer in an unpackable state and the function will return an error code to warn of this condition.(\code{int32_t})} @@ -279,7 +279,7 @@ \subsection{\code{PMIx_Data_unpack}} \ac{PMIx} library is aware of the version the packer used. Note that all processes in a given namespace are \textit{required} to use the same \ac{PMIx} version --- thus, the caller must only know at least one process from the -packer's namespace +packer's namespace. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%