Running on Discover

[cbblanke]

Where can I find an example slurm script for submitting a job on Discover?

[dmocko]

Hi @cbblanke

Sure - happy to help. But please be sure that you are using the latest module on Discover:
https://github.com/NASA-LIS/LISF/blob/master/env/discover/lisf_7_gnu_11.2.0_impi_2021.4.0

The Discover Quick Start guide explains how to use this module:
https://github.com/NASA-LIS/LISF/blob/master/docs/discover_quick_start/discover_quick_start.adoc

Here is an example script. I have the "qos" line commented out, so it will just go to the "general" queue. More on "qos" below.

#!/bin/bash
#SBATCH --job-name lisjob
#SBATCH --ntasks=560
##SBATCH --qos=debug
#SBATCH --time=06:00:00
#SBATCH --output=lisjob.%j.out
#SBATCH --account=[enter your sponsor code]

cd $SLURM_SUBMIT_DIR

ulimit -s unlimited
ulimit -n 4096

mpirun -np 560 ./LIS -f lis.config

560 cores/CPUs is just an example, although it does happen to be the maximum that will run on the "long" queue. The number of core in your job script should exactly match the product of these two lines in your lis.config file:

Number of processors along x:           28
Number of processors along y:           20

Be sure to review these pages from the NCCS on setting up SLURM jobs and the QoS (Quality of Service):
https://www.nccs.nasa.gov/nccs-users/instructional/using-slurm/best-practices
https://www.nccs.nasa.gov/nccs-users/instructional/using-slurm/qos

David

[cbblanke]

This is the output I am getting. (I am not sure the status of my compute hours, I just put in a request to check.)

discover32: bigafrica> runlis.sh
Here
[mpiexec@discover32] check_exit_codes (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:117): unable to run bstrap_proxy on discover32 (pid 4078, exit code 256)
[mpiexec@discover32] poll_for_event (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:159): check exit codes error
[mpiexec@discover32] HYD_dmx_poll_wait_for_proxy_event (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:212): poll for event error
[mpiexec@discover32] HYD_bstrap_setup (../../../../../src/pm/i_hydra/libhydra/bstrap/src/intel/i_hydra_bstrap.c:1062): error waiting for event
[mpiexec@discover32] HYD_print_bstrap_setup_error_message (../../../../../src/pm/i_hydra/mpiexec/intel/i_mpiexec.c:1015): error setting up the bootstrap proxies
[mpiexec@discover32] Possible reasons:
[mpiexec@discover32] 1. Host is unavailable. Please check that all hosts are available.
[mpiexec@discover32] 2. Cannot launch hydra_bstrap_proxy or it crashed on one of the hosts. Make sure hydra_bstrap_proxy is available on all hosts and it has right permissions.
[mpiexec@discover32] 3. Firewall refused connection. Check that enough ports are allowed in the firewall and specify them with the I_MPI_PORT_RANGE variable.
[mpiexec@discover32] 4. slurm bootstrap cannot launch processes on remote host. You may try using -bootstrap option to select alternative launcher.

Run script looks like this:
#!/bin/bash
#SBATCH --job-name testslurm
#SBATCH --ntasks=2
#CBB example was ntasks=560; max for long

##SBATCH --qos=debug
#SBATCH --time=00:01:00
#SBATCH --output=lisjob.%j.out
#SBATCH --account=s5295
##SBATCH --account=s0746

module purge
unset LD_LIBRARY_PATH
module load lisf_7_gnu_11.2.0_impi_2021.4.0

#CBB I assume this is nothing special...
#echo SSD is $SLURM_SUBMIT_DIR
#cd $SLURM_SUBMIT_DIR

ulimit -s unlimited
ulimit -n 4096
mpirun -np 140 ./LIS -f lis.config

[dmocko]

Three things:

1. Change this line to: #SBATCH --ntasks=140

   The number of tasks must match the number on the mpirun line.

2. You can't run this script on the command line. You need to submit it to the job queue.

   From the command line, type: sbatch runlis.sh

3. The "cd $SLURM_SUBMIT_DIR" line should be there to ensure that after you submit the job to SLURM,
   the job will run from the correct directory.

David

[cbblanke]

Thanks. Looks like it is working now.

[cbblanke]

Is this a problem? (Beginning of output from sbatch.)

srun: cluster configuration lacks support for cpu binding

[karsenau]

Hi @cbblanke:

Checked with another team member, and basically that is just a status message from SLURM informing you that discover does not have CPU binding enabled. It is not an actual error and shouldn't cause you any issue.