crystal_router aborts at np >= 256 with ~16 MiB-per-record transfers

[slitvinov]

We want to use sarray_transfer to move records, each about 16 MiB,
between 256 MPI ranks. We expect it to work the same way it does at
smaller np. Instead, inside crystal_router one of the internal
MPI sends grows past 2 GiB. Because MPI's message count is a signed
32-bit integer, gslib's check aborts the run:

https://github.com/Nek5000/gslib/blob/95acf5b/src/crystal.c#L137-L145

Error in crystal_router: rank = 0 send_n = 2147484160 (> INT_MAX)

np = 256 is the smallest power-of-two that triggers it with
16 MiB records. Memory on rank 0 is about np * 16 MiB (~4 GiB at
np=256); other ranks use roughly 16 MiB each. Even if the explicit
check were removed, comm_isend and comm_irecv in src/comm.h
pass the byte count straight to MPI_Isend / MPI_Irecv, which take
a 32-bit int and would truncate it:

https://github.com/Nek5000/gslib/blob/95acf5b/src/comm.h#L242-L256

Reproducer (repro.c):

#include "gslib.h"

/* matches nekRS CB_BUFFER_SIZE (nek5000/core/nekio.c) */
#define RECORD_BYTES (16u * 1024u * 1024u - 8u)
typedef struct { uint p; char data[RECORD_BYTES]; } record_t;

int main(int argc, char **argv)
{
    int np;
    struct comm comm;
    struct crystal crystal;
    struct array A = null_array;
    record_t *row;
    uint i;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &np);
    comm_init(&comm, MPI_COMM_WORLD);
    crystal_init(&crystal, &comm);

    if (comm.id == 0) {
        array_init(record_t, &A, np);
        A.n = np;
        row = A.ptr;
        for (i = 0; i < (uint)np; ++i) row[i].p = i;
    }

    sarray_transfer(record_t, &A, p, 1, &crystal);

    array_free(&A);
    crystal_free(&crystal);
    comm_free(&comm);
    MPI_Finalize();
    return 0;
}

Build and run:

git clone https://github.com/Nek5000/gslib.git
( cd gslib && make CC=mpicc MPI=1 )
mpicc -O2 -Igslib/build/include/gslib repro.c \
      -Lgslib/build/lib -lgs -lm -o repro

mpiexec --oversubscribe -n 224 ./repro    # ok
mpiexec --oversubscribe -n 256 ./repro    # aborts

Reproduced on macOS 15 / Apple Silicon with Open MPI 5.0.9 (Homebrew),
gslib master at 95acf5b. Also
reproduced on ALCF Crux with cray-mpich 9.0.1 and gcc 13.2.

The guard itself was added in #45
("Check if crystal router message size is larger than INT_MAX") whose
body lists one TODO: "Test if the check works using a failed case".
This reproducer is that case.

[yslan]

I’m curious about your usage pattern and what leads to the worst-case scenario where traffic is funneled through rank 0.

In our typical use of crystal_router for all-to-all communication, large data exchanges originate from meshes that are already partitioned to reduce bisection bandwidth. As a result, hitting INT_MAX at np=256 is uncommon.

That said, this limit becomes more relevant at larger scales. A workaround we’ve used is batching the communication to reduce peak volume (roughly 4 MiB per rank), which has scaled to O(100k) ranks in practice:
https://github.com/Nek5000/Nek5000/blob/a07ba8a63dbbe15eb908f0a218d713c640642f7c/core/ic.f#L1988-L2038

I’ve been working on this issue recently by introducing batching directly inside crystal_router when it needs > 2GiB. A work-in-progress implementation is available here:
https://github.com/yslan/gslib/commits/bcr/

It still needs more testing, but so far it has successfully handled ~200 MiB per rank × 24 ranks which peaks at 2.66GiB.