-
Notifications
You must be signed in to change notification settings - Fork 0
LSF Walkthrough
In this tutorial we'll go through the process of creating and submitting a simple LSF batch script. At the end you should be able to:
- Write an batch script that runs a simple multi-threaded job
- Submit that script and monitor your submission in the queue with
bjobs - Check the output and error log files to diagnose if your job ran correctly
This tutorial assumes you are familiar with navigating on the command line and can edit a file using a command line text editor such as nano
The first thing we're going to do is create a script that will tell the scheduler what resources we'll need for our job and then what commands to run.
This script has two parts: 1) The BSUB directives which we use to tell the scheduler what resources such as CPUs, walltime, and memory that we'll need and 2) The commands that we actually need to run for our job.
Create a new directory called my_project and cd into it. In that directory create a new file and open it in nano with a single command nano test_run.sh. To the top of that script add the following lines:
#! /usr/bin/env bash
#BSUB -J test_run
#BSUB -n 1
#BSUB -R "span[hosts=1]"
#BSUB -W 00:10
#BSUB -o test_%J.out
#BSUB -e test_%J.err
sleep 60
echo "finished with exit code $? at: `date`"
Let's walk through these lines
#! /usr/bin/env bash
This is a line found in many executable scripts and it basically says to run your script using bash. You'll never need to change this and it should always be the first line of your script.
The next 6 lines are called bsub directives and are used to tell the scheduler software, bsub what resources you'll need. All bsub directives start with #BSUB so bsub knows what they are. Each of these directives can alternatively be provided at the command line but for the sake of reproducibility it is recommended that you put these inside you script.
-J test_run
This is name of the job and is optional but pick something sensible
-n 1
The number of cores, also called threads, that you'd like to use for your job. Note that this is total number of cores required across all nodes and should never be more than 56 on this system.
If the software you are running is multithreaded and has an option for specifying the number of threads or cores then you'll need to make sure the number you provide to the -n option is the same as what you specify in your command. You can use the $LSB_DJOB_NUMPROC environmental variable in your script to make sure these values are the same. For example, if you have -n 12 then you can specify the number of cores to your application like so:
blast -num_threads $LSB_DJOBS_NUMPROC ...
-R "span[hosts=1]"
This says to run job on a single node. Most jobs will not be able to run on multiple nodes unless you are using MPI, which you're not, so be sure to always include this.
-W 01:00
Walltime in HH:MM. This is how long your job takes to run and affects scheduling so pay attention to this! Deciding how long to specify is really an art and usually requires trial and error. A good rule of thumb if you have absolutely no idea how long something takes to run is to increase your best guess by an order of magnitude. If you think it should take about 2 hours then start by setting the walltime to 12 hours. If you run the same thing many times you can refine your walltime estimates to be more accurate.
-o test_%J.out
-e test_%J.err
Normally when you run a command on the screen the output and error streams, called stdout and stderr respectively, are merged and printed out to your screen. When your job is run on a compute node the stdout and stderr get redirected to files. The two lines below allow you to specify names for these files. By default they are created in your working directory. The %J gets replaced with the job ID.
sleep 60
echo "finished with exit code $? at: `date`"
These are simply shell commands and this is where you would put your commands for your job. For now these will serve as placeholder commands to test our script.
Ok, now that we've finished editing the script, press CTRL-O to write the file followed by CTRL-X to exit nano. Staying in the current working directory, my_project we can submit our job using the following command.
bsub < test_run.sh
Note that we need the redirect arrow < here as bsub takes input from stdin. You should get a message about job submitted and an job ID.
Now our job is in the queue. Type bjobs to see a list of your jobs in the queue. The "STAT" column tells us the status of our job and will be set to PEND when the job is queued but not running. This will change to RUN when the job starts running. You'll also see the job ID, the name of queue, what node your job is running on (along with the number of cores), and the job name as specified in your script.
By default bjobs shows you only your jobs. If you type bjobs -u all you can get a list of all jobs in the queue.
When your job is finished it will disapear from the queue. If you have no jobs left in the queue you'll get a message stating, "No unfinished jobs found".
Now it's time to sort out if our job ran correctly. The time it took for your job to run is your first clue if everything is hunky dory or not. If you expected your job to take 12 hours and it finished in seconds then the chances are very high that you got an error. If the time was on par with what you expected then you can have a quick look at the output files.
Using ls -l in our project directory have a look at the sizes of the two .err and .out files that should be there. How big are they? Is one empty? less the .out file and have a look through the output. This should give you some info about the job and hopefully you'll see the output of the echo command that was in our script.
If your job produced any errors you'll likely have to look in both the .err and .out files as different programs will write errors to different places. The other way to tell if your job ran correctly is to see if it produced the result files that it was supposed to.