Run the ww3_shel part of run_cmake_test through a sbatch job scheduler

[Liu-Jincan]

Hello, everyone.
I'm currently trying to run the ww3_shel part of run_cmake_test through a sbatch job scheduler, but I'm having a lot of problems with MPI in this process.

• Open a terminal, go to the regtests folder, and run the following codes:

  ./bin/run_cmake_test -i input -N -s ST4 -r ww3_shel -p mpirun -n 8 -o netcdf -w work ../model test
  

  The program runs correctly, and outputs the associated out_grd.ww3 file.

• When running the program in the form of a sbatch job submission:

  #!/bin/bash
  #SBATCH --comment=WW3
  #SBATCH -J wavewatch3
  #SBATCH -p normal
  #SBATCH -t 1:00:00
  #SBATCH -n 120
  #SBATCH --mem=60G
  #SBATCH -e wavewatch3_sbatch_error.log
  
  # Load Environment
  # ...
  
  cd WW3-develop-20230213/regtests
  ./bin/run_cmake_test -i input -N -s ST4 -r ww3_shel -p mpirun -n 8 -o netcdf -w work ../model test
  

  There is an error running, ww3_shel.out shows the following error:

  ===================================================================================
  =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
  =   PID 69057 RUNNING AT cmbc0523
  =   EXIT CODE: 9
  =   CLEANING UP REMAINING PROCESSES
  =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
  ===================================================================================
  

• I then go through the run_cmake_test settings, add -b slurm, change the argument after -p to srun, and add a few more commands to the sbatch file:

  #SBATCH -N 4               
  #SBATCH --ntasks-per-node=32
  

  The log of wavewatch3_sbatch_error.log shows the following error:

  srun: Warning: can't honor --ntasks-per-node set to 32 which doesn't match the requested tasks 100 with the number of requested nodes 4. Ignoring --ntasks-per-node.
  srun: error: cmbc0094: task 75: Killed
  srun: Terminating job step 56725533.0
  srun: error: cmbc0091: task 17: Killed
  slurmstepd: error: *** STEP 56725533.0 ON cmbc0091 CANCELLED AT 2023-03-22T08:39:12 ***
  srun: error: cmbc0092: task 33: Killed
  srun: error: cmbc0093: task 68: Killed
  srun: error: cmbc0094: task 77: Killed
  srun: error: cmbc0091: task 22: Killed
  srun: error: cmbc0091: tasks 0-3,9,19,21,24: Terminated
  srun: error: cmbc0091: task 11: Terminated
  srun: error: cmbc0094: tasks 79,82-83,86,90,92,94,97-98: Terminated
  srun: error: cmbc0092: tasks 34,38-39,44,46: Terminated
  srun: error: cmbc0094: tasks 81,89: Terminated
  srun: error: cmbc0092: tasks 36,41,43: Terminated
  srun: error: cmbc0094: tasks 76,78,84,93: Terminated
  srun: error: cmbc0092: tasks 27,29,32,35,40,47: Terminated
  srun: error: cmbc0094: tasks 80,85,87-88,91,95-96,99: Terminated
  srun: error: cmbc0092: tasks 25-26,28,30-31,37,42,45,48-49: Terminated
  srun: error: cmbc0091: task 6: Terminated
  srun: error: cmbc0093: tasks 51-52,54,63,69: Terminated
  srun: error: cmbc0091: tasks 14,18,20: Terminated
  srun: error: cmbc0093: tasks 56,74: Terminated
  srun: error: cmbc0091: tasks 4-5,7-8,10: Terminated
  srun: error: cmbc0093: task 53: Terminated
  srun: error: cmbc0091: tasks 12-13,15-16,23: Terminated
  srun: error: cmbc0093: tasks 64,72: Terminated
  srun: error: cmbc0093: tasks 58-59: Terminated
  srun: error: cmbc0093: tasks 50,55,57,60-62,65-67,70-71,73: Terminated
  srun: Force Terminated job step 56725533.0
  

Any suggestions are sincerely appreciated, thank you very much.

[JessicaMeixner-NOAA]

Have you tried using
#SBATCH -n 8 which matches the number of mpi tasks below?

[Liu-Jincan]

@JessicaMeixner-NOAA Very thanks for your reply. It gives me the motivation to continue again.
Try using #SBATCH -n 8 with just noe node, and the front part of the SBATCH is set：

#!/bin/bash
#SBATCH --comment=WW3
#SBATCH -J wavewatch3
#SBATCH -p normal
#SBATCH -t 1:00:00
#SBATCH -n 8
#SBATCH --mem=20G
#SBATCH -e wavewatch3_sbatch_error.log

• The latter part of the SBATCH is set in case 1：

  ./bin/run_cmake_test -i input -N -s ST4 -b slurm -r ww3_shel -p srun -n 8 -o netcdf -w work ../model test
  

  The error in wavewatch3_sbatch_error.log:

  srun: error: cmbc1380: task 0: Killed
  srun: Terminating job step 56757518.0
  slurmstepd: error: *** STEP 56757518.0 ON cmbc1380 CANCELLED AT 2023-03-22T13:38:34 ***
  srun: error: cmbc1380: tasks 2,4,6: Terminated
  srun: error: cmbc1380: tasks 1,3,5,7: Terminated
  srun: Force Terminated job step 56757518.0
  

• The latter part of the SBATCH is set in case 2：

  ./bin/run_cmake_test -i input -N -s ST4 -b slurm -r ww3_shel -p mpirun -n 8 -o netcdf -w work ../model test
  

  No problem, running correctly.

Try using #SBATCH -n 64 with two nodes, and the front part of the SBATCH is set：

#!/bin/bash
#SBATCH --comment=WW3
#SBATCH -J wavewatch3
#SBATCH -p normal
#SBATCH -t 1:00:00
#SBATCH -n 64
#SBATCH --mem=20G
#SBATCH -e wavewatch3_sbatch_error.log

• The latter part of the SBATCH is set in case 3：

  ./bin/run_cmake_test -i input -N -s ST4 -b slurm -r ww3_shel -p mpirun -n 64 -o netcdf -w work ../model test
  

  The error in wavewatch3_sbatch_error.log:

  Operating system error: Cannot allocate memory
  Allocation would exceed memory limit
  

Try using #SBATCH --mem=50G , and the front part of the SBATCH is set：

#!/bin/bash
#SBATCH --comment=WW3
#SBATCH -J wavewatch3
#SBATCH -p normal
#SBATCH -t 1:00:00
#SBATCH -n 64
#SBATCH --mem=50G
#SBATCH -e wavewatch3_sbatch_error.log

• The latter part of the SBATCH is set in case 4：

  ./bin/run_cmake_test -i input -N -s ST4 -b slurm -r ww3_shel -p mpirun -n 64 -o netcdf -w work ../model test
  

  The error in ww3_shel.out:

  ===================================================================================
  =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
  =   PID 35384 RUNNING AT cmbc0496
  =   EXIT CODE: 9
  =   CLEANING UP REMAINING PROCESSES
  =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
  ===================================================================================
  

  The error in wavewatch3_sbatch_error.log:

  [proxy:0:1@cmbc1460] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): assert (!closed) failed
  [proxy:0:1@cmbc1460] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status
  [proxy:0:1@cmbc1460] main (pm/pmiserv/pmip.c:200): demux engine error waiting for event
  srun: error: cmbc1460: task 1: Exited with exit code 7
  srun: Terminating job step 56761249.0
  [mpiexec@cmbc0496] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:75): one of the processes terminated badly; aborting
  [mpiexec@cmbc0496] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:22): launcher returned error waiting for completion
  [mpiexec@cmbc0496] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:215): launcher returned error waiting for completion
  [mpiexec@cmbc0496] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion
  
  

Well, I really don't know what went wrong.

[JessicaMeixner-NOAA]

I'd encourage you to use the configuration that is running. As for the others, these appear to be a batch system issue, not WW3 a issue and would suggest reaching out to your system admins.