Page Comparison

This page shows you how to run submit a bash script to Slurm. The runSingle script, accessible through the rcbio/1.2 module, converts an input bash script with sbatch command and submits a single job to the Slurm scheduler for you.

Features of the new way of submitting job:

• Informative email notifications are sent when job fails or succeeds.

• Get a better log with memory and CPU usage. 

• Auto re-run in case node fails.

• When re-running on the same data folder, the user is asked to confirm to re-run or not if it was done successfully earlier.

Please read below for an example.

...

If you need help connecting to O2, please review the Using Slurm Basic and the How to Login to O2 wiki pages.

From Windows, use the graphical PuTTY program to connect to o2.hms.harvard.edu and make sure the port is set to the default value of 22.

From a Mac Terminal, use the ssh command, inserting your eCommons ID instead of user123:

ssh user123@o2.hms.harvard.edu

Start interactive job, and create working folder

For example, for user abc123, the working directory will be

srun --pty -p interactive -t 0-12:0:0 --mem 2000MB -c 1 /bin/bash
mkdir /n/scratch3/users/${USER:0:1}/${USER}/test
cd /n/scratch3/users/${USER:0:1}/${USER}/test

Load the pipeline related modules

# This will setup the path and environmental variables for the pipeline
module load rcbio/1.2

The bash script

cp /n/app/rcbio/1.2/bin/bashScript.sh .
# Run cat command to show the content of bashScript.sh
cat bashScript.sh

#!/bin/sh
for i in A B; do            

    echo John >> John.txt; echo Mike >>  Mike.txt       
  
    echo Nick >> Nick.txt; echo Julia >>  Julia.txt

done

cat John.txt Mike.txt Nick.txt Julia.txt > all.txt

...

You can use the command:

squeue -u $USER

To see the job status (running, pending, etc.). You also get two emails for each step, one at the start of the step, one at the end of the step.

...

You can use the command:

ls -l flag

This command list all the logs created by the pipeline runner. *.sh files are the slurm scripts for each step, *.out files are output files for each step, *.success files means job successfully finished for each step and *.failed means job failed for each steps.

...

You can rerun this command in the same folder

runSingleJob bashScript.sh "sbatch -p short -t 10:0 -c 1"

...

In the example above, we run a bash script. You can run any command you like:  

runSingleJob "python pythonScript.py" "sbatch -p short -t 10:0 -c 1"
or
runSingleJob "module load bowtie/1.2.2; bowtie -x /n/groups/shared_databases/bowtie_indexes/hg19 -p 2 -1 read1.fq -2 read2.fq --sam > out.bam" "sbatch -p short -t 1:0:0 -c 2 -mem 8G"

...