Consistent C++ Memory Alloc Error

[Sanderson1887]

Summary

When attempting to use a deepmd trained model with LAMMPS on a HPC system, using any more than a few computing cores will consistently lead to a "C++ memory allocation failed: std::bad_alloc" error.
If the total processor count is very low (less than about 20) the LAMMPS run will commence fine, but not many more than that.

DeePMD-kit Version

DeePMD-kit v3.0.2

Backend and its version

2.15.0

Python Version, CUDA Version, GCC Version, LAMMPS Version, etc

Python 3.9.2
CUDA 12.2
GCC 9.1.0
LAMMPS 29 Aug 2024 - Update 1

Details

I am using a model with 5 different elements. Type_one_side is set to it's default value of FALSE and, when the model is compressed, the step size is also set to default. These values give a (seemingly) fairly large potential file of ~300 Megabytes. I assume this is a likely cause of the following error.

When attempting to use this trained potential on a LAMMPS script, I cannot use more than ~20 compute nodes without getting a C++ Memory Alloc error. This occurs during the first step when the model would be called (minimization, run, etc. etc.) Might this be a DeePMD error due to file size? Or is it more likely an error on the HPC side.

[wanghan-iapcm]

Have you attempted to decrease the system size while maintaining the same number of processors?

[njzjz]

Is it just due to out-of-memory? How much memory is there?

[Sanderson1887]

Have you attempted to decrease the system size while maintaining the same number of processors?

For incredibly small system sizes (less than ~100) the LAMMPS simulation can proceed, but any more than that and the memory alloc error reappears.

Is it just due to out-of-memory? How much memory is there?

SSHing into the compute node being used and using "free -h" shows that about 180G is available for use.

[wanghan-iapcm]

could you please provide the submission script and the lammps input script?　

[Sanderson1887]

Apologies for the delay responding. Attached are the scripts for the training of the potential, the MWE Lammps script, and the submission script (the latter two have extensions changed so GitHub plays nicely with them).

bohrtrain.json
submit.txt
runFe.txt

Would you also like a potential to run it with?

[njzjz]

Hi @Sanderson1887, please limit the number of OpenMP threads with OMP_NUM_THREADS.

[Sanderson1887]

I tried this but I still got a memory alloc error. I also tried to set DP_INTER_OP_PARALLELISM_THREADS and DP_INTRA_OP_PARALLELISM_THREADS to minimize total thread usage.
The submission script is still the same, except with the addition of the lines

export OMP_NUM_THREADS=1
export DP_INTER_OP_PARALLELISM_THREADS=1
export DP_INTRA_OP_PARALLELISM_THREADS=1

[njzjz]

Did you observe how much memory each rank uses? Loading a large size of model needs a lot of memory on each MPI rank.