...
This page shows you how to run submit a bash script to Slurm. The runSingle
script, accessible through the rcbio/1.23.3
module, converts an input bash script with sbatch command and submits a single job to the Slurm scheduler for you.
...
Please read below for an example.
Log on to O2
If you need help connecting to O2, please review the Using Slurm Basic and the How to Login to O2 wiki pages.
From Windows, use the graphical PuTTY program to connect to o2.hms.harvard.edu and make sure the port is set to the default value of 22.
...
Code Block |
---|
ssh user123@o2.hms.harvard.edu |
Start interactive job, and create working folder
For example, for user abc123, the working directory will be
Code Block |
---|
srun --pty -p interactive -t 0-12:0:0 --mem 2000MB -c 1 /bin/bash mkdir /n/scratch3/users/${USER:0:1}/${USER}/test cd /n/scratch3/users/${USER:0:1}/${USER}/test |
Load the pipeline related modules
Code Block |
---|
# This will setup the path and environmental variables for the pipeline module load rcbio/1.3.23 |
The bash script
Code Block |
---|
cp /n/app/rcbio/1.3.23/bin/bashScript.sh . # Run cat command to show the content of bashScript.sh cat bashScript.sh #!/bin/sh for i in A B; do echo John >> John.txt; echo Mike >> Mike.txt echo Nick >> Nick.txt; echo Julia >> Julia.txt done cat John.txt Mike.txt Nick.txt Julia.txt > all.txt |
Submit the script to Slurm:
Code Block |
---|
runSingleJob "bash bashScript.sh" "sbatch -p short -t 10:0 -c 1" # Below is the output Running flag/slurmPipeLine.202103311551.run.sh module list: Currently Loaded Modules: 1) gcc/6.2.0 2) python/2.7.12 3) rcbio/1.3.23 depend on no job sbatch -p short -t 10 -c 1 --mail-type=FAIL --nodes=1 -J Job_bash_bashScript.sh -o /home/ld32/rcbio/flag/Job_bash_bashScript.sh.out -e /home/ld32/rcbio/flag/Job_bash_bashScript.sh.out /home/ld32/rcbio/flag/Job_bash_bashScript.sh.sh # Submitted batch job 43976 |
Monitoring the jobs
You can use the command:
Code Block |
---|
squeue -u $USER |
To see the job status (running, pending, etc.). You also get two emails for each step, one at the start of the step, one at the end of the step.
Check job log
You can use the command:
Code Block |
---|
ls -l flag |
...
You also get two emails for each step, one at the start of the step, one at the end of the step.
Re-run the job
You can rerun this command in the same folder
...
This command will check if the earlier run is finished or not. If not, ask user to kill the running jobs or not, then ask user to rerun the successfully finished steps or not. Click 'y', it will rerun, directly press 'enter' key, it will not rerun.
To run your own script as Slurm pipeline
In the example above, we run a bash script. You can run any command you like:
...