What are members in CommMetaData

[maikel]

I try to understand how to customize communication of FabArrays. A commonly used type in synchronization primitives, such as FillBoundary or ParallelCopy is struct CommMetaData

struct CommMetaData
{
    // The cache of local and send/recv per FillBoundary() or ParallelCopy().
    bool m_threadsafe_loc = false;
    bool m_threadsafe_rcv = false;
    std::unique_ptr<CopyComTagsContainer>      m_LocTags;
    std::unique_ptr<MapOfCopyComTagContainers> m_SndTags;
    std::unique_ptr<MapOfCopyComTagContainers> m_RcvTags;
};

where

struct CopyComTag
{
	Box dbox;
        Box sbox;
        int dstIndex;
        int srcIndex;
        /* omit constructors and typedefs */
};
using CopyComTagsContainer = std::vector<CopyComTag>;
using MapOfCopyComTagContainers = std::map<int, std::vector<CopyComTag>>;

Question: Could you explain the member variables of those classes to me? Are LocTags (local tags?) of same shape as local FABs? What about the maps? Does dstIndex and srcIndex mean components?

Can I use CommMetaData To write a free function ParallelCopy ?

[WeiqunZhang]

CopyComTag stores the meta-data for copying the data from a subregion sbox (here s means source) of the source FabArray to a subregion dbox of the destination FabArray. The integers srcIndex and dstIndex specify the global box index in the FabArrays. That is source_fabarray[srcIndex] gives us the source Fab (e.g., FArrayBox).

CommMetaData::m_LocTags are for copying data locally (as opposed to via MPI). The information for sending and receiving using MPI is stored in m_SndTags and m_RcvTags as std::map. The key to the map is the MPI rank (destination rank for m_SndTags and source rank for m_RcvTags). The two boolean flags are used to indicate if there are race conditions (i.e., multiple threads writing to the same location).

Yes, I believe you can use CommMetaData to write a free function to do your own communication. An example of this can be found in AMReX_NonLocalBC.H. Usually you want to cache the communication meta-data. In AMReX, the caches are stored in FabarrayBase::FBCache, FabArrayBase::CPCache, etc.