The most commonly used job scheduler with Singularity or bioinformatics analysis jobs is Slurm. Here are its benefits:
- load balancing between multiple nodes
- limiting ram/process job size
- limiting job time
Implementation -- singularity_job.slurm
#!/bin/bash
#SBATCH --ntasks=1 # number of tasks/processes run in this job
#SBATCH --time=01:00:00 # one hour runtime limit
#SBATCH --mem-per-cpu=4096 # default is 1GB, max memory required per CPU
#SBATCH --nodes=2
#SBATCH --output=slurm.out
#SBATCH --cpus-per-task=1
#SBATCH --requeue # if allocated node hangs, job requeued
#SBATCH --gres=lscratch:500 # max scratch disk space available for use in /lscratch, in GB -- when job is terminated, all data in /lscratch/$SLURM_JOB_ID directory will be automatically deleted (if data needs to be saved, copy to /data before job concludes
CONTAINER=/opt/singularity/orca.simg
OVERLAY=/$HOME/.orca/overlay.simg
module load singularity
# hostname
singularity build ${CONTAINER} ${OVERLAY} # run program/command on resource allocated
Submit the job
$ sbatch singularity_job.slurm
$ cat slurm-<jobid>.out
$ squeue
Etc
mkdir lscratch
export SINGULARITY_BINDPATH="/data/$USER:/data,/fdb:/resources,/lscratch/$SLURM_JOB_ID:/tmp"
singularity shell my-container.simg
"we use symbolic links to refer to our network storage systems. As a result, you will need to bind some additional directories to the ones you know of and use directly to ensure that the symbolic link destinations are also bound into the container." hpc nih
Changing TMPDIR from /tmp (8GB per node) to /lscratch (set your own limit)
- this also is better for multi user accesses
- allocate local scratch disk for jobs
#!/bin/bash
export TMPDIR=/lscratch/$SLURM_JOB_ID
The most commonly used job scheduler with Singularity or bioinformatics analysis jobs is Slurm. Here are its benefits:
Implementation -- singularity_job.slurm
Submit the job
Etc
"we use symbolic links to refer to our network storage systems. As a result, you will need to bind some additional directories to the ones you know of and use directly to ensure that the symbolic link destinations are also bound into the container." hpc nih
Changing TMPDIR from /tmp (8GB per node) to /lscratch (set your own limit)