Data corruption trying to run bbview on single node

[denisbertini]

Environment

• OpenMPI version: 6.1.a1 (built from source with BBView submodule)
• BBView commit: [current commit hash]
• System: Rocky Linux 8, Lustre filesystem, NVMe local storage (/tmp)
• MPI Implementation: (Open MPI) 6.1.0a1 with BBView

Data Corruption Due to Unaligned I/O

Description

When writing non-contiguous data with BBView, the data becomes corrupted. The corruption occurs during the initial write to the burst buffer (/tmp/NVMe), not during the flush to Lustre.

Steps to Reproduce

1. Build OpenMPI (git repo master ) with BBView following the instructions
2. Run the provided benchmark with verification:

# Start bbviewd
bbviewd 1 &


# Write test file
mpirun -np 2 ./bbview_benchmark -f /lustre/test.out -s 10 -c 100

### Errors data corrupted 

mpirun -np 2 ./bbview_benchmark -f /lustre/rz/dbertini/test.out -s 10 -c 100 -v
Configuration: 2 ranks, 10 MB/rank, 100 chunks, 104857 bytes/chunk
chunk 0: offset=0, size=104857
chunk 1: offset=209714, size=104857
chunk 2: offset=419428, size=104857
chunk 3: offset=629142, size=104857
chunk 4: offset=838856, size=104857
Not using O_DIRECT for /tmp//lustre-rz-dbertini-test.out-0-1
Not using O_DIRECT for /tmp//lustre-rz-dbertini-test.out-1-1
mca_fbtl_posix_pwritev: error in (p)write(v):Bad address
mca_fbtl_posix_pwritev: error in (p)write(v):Bad address
Flushing file /lustre/rz/dbertini/test.out for view index 1
Flushing file /lustre/rz/dbertini/test.out for view index 1
bbviewd[3898601]: bbviewd: begin /tmp//lustre-rz-dbertini-test.out-0-1 -> /lustre/rz/dbertini/test.out

========================================
BBView Benchmark Results

File: /lustre/rz/dbertini/test.out
Ranks: 2
Data per rank: 10 MB
Total data: 20.00 MB
Chunks/rank: 100
Chunk size: 104857 bytes

Write Performance:
Write time (min/avg/max): 0.001 / 0.001 / 0.001 seconds
Aggregate bandwidth: 31871.04 MB/s (31.12 GB/s)

Close Performance:
Close time (min/avg/max): 0.001 / 0.001 / 0.001 seconds

bbviewd[3898601]: bbviewd: end /tmp//lustre-rz-dbertini-test.out-0-1 -> /lustre/rz/dbertini/test.out
bbviewd[3898601]: bbviewd: begin /tmp//lustre-rz-dbertini-test.out-1-1 -> /lustre/rz/dbertini/test.out
bbviewd[3898601]: bbviewd: end /tmp//lustre-rz-dbertini-test.out-1-1 -> /lustre/rz/dbertini/test.out

Any  idea what these errors means:

Not using O_DIRECT for /tmp//lustre-rz-dbertini-test.out-0-1
Not using O_DIRECT for /tmp//lustre-rz-dbertini-test.out-1-1
mca_fbtl_posix_pwritev: error in (p)write(v):Bad address
mca_fbtl_posix_pwritev: error in (p)write(v):Bad address

responsible of data corruption!

[denisbertini]

really want to use bbview, would be nice if you could help me here !

[KoyamaSohei]

Thank you for using BBView, and I apologize for the delayed response.
Would you be able to share the source code for bbview_benchmark?

[denisbertini]

Thank you for your answer!,
meanwhile i solved the issue , one need to force bbview to use individual for the collective algorithm and then the data corruption is solved.
I am now able to use bbview in a environment using container ( apptainer ) and slurm as a launcher.
I planned to use bbview as a service launched prior to the application, since i notice some memory usage which very fast exceed the cgroup controlled memory in slurm of 2Gbyte/core.
My hope is that using bbview as service will allow me to allocate more memory for the bbviewd daemon.
For now i used very simple MPI I/O collective benchmark, but i would like to give a try on real application such as EPOCH or WARPX that we use here at GSI for plasma physics simulation.
Again i would like to thank you to share your extremely interesting work on MPI I/O bbview plugin and i hope to be able to also contribute !
For example did you think of developping also a king of read ahead prefetching for bbview ? Would be very interesting !