Page Comparison

Table of Contents

This page shows you how to run a regular bash script as a pipeline. The runAsPipeline script, accessible through the rcbio/1.0 module, converts an input bash script to a pipeline that easily submits jobs to the Slurm scheduler for you.

...

Code Block

srun --pty -p interactive -t 0-12:0:0 --mem 2000MB -n 1 /bin/bash
mkdir /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline  
cd /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline

...

The modified bash script

Code Block

cp /homen/app/ld32rcbio/rcbioDev1.1/bin/bashScriptV2.sh .
cat bashScriptV2.sh

# Below is the conten of bashScriptV2.sh
#!/bin/sh
	
for i in A B; do            
    u=university$i.txt    
            
    #@1,0,find1,u,sbatch -p short -n 1 -t 50:0
    grep -H John $u >>  John.txt; grep -H Mike $u >>  Mike.txt        
     
    #@2,0,find2,u,sbatch -p short -n 1 -t 50:0
    grep -H Nick $u >>  Nick.txt; grep -H Julia $u >>  Julia.txt
                   
done

#@3,1.2,merge     
cat John.txt Mike.txt Nick.txt Julia.txt > all.txt

...

With useTmp, the pipeline runner copy related data to /tmp and all file paths will be automatically updated to reflect a file's location in /tmp when using the useTmp option.

Sample output from the test run

Note that only step 2 used -t 50:0, and all other steps used the default -t 10:0. The default walltime limit was set in the runAsPipeline command, and the walltime parameter for step 2 was set in the bash_script_v2.sh script.

Code Block

runAsPipeline bashScriptV2.sh "sbatch -p short -t 10:0 -n 1" useTmp

# Below is the output: 
converting bashScriptV2.sh to flag/slurmPipeLine.201801161424.sh

find loopStart: for i in A B; do 	

find job marker:
#@1,0,find1,u:     

find job:
grep -H John $u >>  John.txt; grep -H Mike $u >>  Mike.txt        

find job marker:
#@2,0,find2,u,sbatch -p short -n 1 -t 50:0
sbatch options: sbatch -p short -n 1 -t 50:0

find job:
grep -H Nick $u >>  Nick.txt; grep -H Julia $u >>  Julia.txt
find loopend: done                    

find job marker:
#@3,1.2,merge:           

find job:
cat John.txt Mike.txt Nick.txt Julia.txt > all.txt
flag/slurmPipeLine.201801161424.sh .sh is ready to run. Starting to run ...
Running flag/slurmPipeLine.201801161424.sh bashScriptV2.sh
---------------------------------------------------------

step: 1, depends on: 0, job name: find1, flag: find1.A reference: .u
depend on no job
sbatch -p short -t 10:0 -n 1 --nodes=1  -J 1.0.find1.A -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.sh 
# Submitted batch job 123

step: 2, depends on: 0, job name: find2, flag: find2.A reference: .u
depend on no job
sbatch -p short -n 1 -t 50:0 --nodes=1  -J 2.0.find2.A -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.sh 
# Submitted batch job 123

step: 1, depends on: 0, job name: find1, flag: find1.B reference: .u
depend on no job
sbatch -p short -t 10:0 -n 1 --nodes=1  -J 1.0.find1.B -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.sh 
# Submitted batch job 123

step: 2, depends on: 0, job name: find2, flag: find2.B reference: .u
depend on no job
sbatch -p short -n 1 -t 50:0 --nodes=1  -J 2.0.find2.B -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.sh 
# Submitted batch job 123

step: 3, depends on: 1.2, job name: merge, flag: merge reference:
depend on multiple jobs
sbatch -p short -t 10:0 -n 1 --nodes=1 --dependency=afterok:123:123:123:123 -J 3.1.2.merge -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.sh 
# Submitted batch job 123

all submitted jobs:
job_id       depend_on              job_flag  
123         null                  1.0.find1.A
123         null                  2.0.find2.A
123         null                  1.0.find1.B
123         null                  2.0.find2.B
123         ..123.123..123.123    3.1.2.merge
---------------------------------------------------------

...

Code Block

runAsPipeline bashScriptV2.sh "sbatch -p short -t 10:0 -n 1" useTmp run

# Below is the output
converting bashScriptV2.sh to flag/slurmPipeLine.201801101002.run.sh

find loopStart: #loopStart,i

find job marker: for i in A B; do

find job:
grep -H John $u >> John.txt; grep -H Mike $u >> Mike.txt

find job marker:
#@2,0,find2,u,sbatch -p short -n 1 -t 50:0
sbatch options: sbatch -p short -n 1 -t 50:0

find job:
grep -H Nick $u >> Nick.txt; grep -H Julia $u >> Julia.txt
find loopend: done

find job marker:
#@3,1.2,merge:

find job:
cat John.txt Mike.txt Nick.txt Julia.txt > all.txt
flag/slurmPipeLine.201801101002.run.sh is ready to run. Starting to run ...
Running flag/slurmPipeLine.201801101002.run.sh
---------------------------------------------------------

step: 1, depends on: 0, job name: find1, flag: find1.A reference: .u
depend on no job
sbatch -p short -t 10:0 -n 1 --kill-on-invalid-dep=yes --nodes=1 -J 1.0.find1.A -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.sh
# Submitted batch job 8091045

step: 2, depends on: 0, job name: find2, flag: find2.A reference: .u
depend on no job
sbatch -p short -n 1 -t 50:0 --kill-on-invalid-dep=yes --nodes=1 -J 2.0.find2.A -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.sh
# Submitted batch job 8091046

step: 1, depends on: 0, job name: find1, flag: find1.B reference: .u
depend on no job
sbatch -p short -t 10:0 -n 1 --kill-on-invalid-dep=yes --nodes=1 -J 1.0.find1.B -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.sh
# Submitted batch job 8091047

step: 2, depends on: 0, job name: find2, flag: find2.B reference: .u
depend on no job
sbatch -p short -n 1 -t 50:0 --kill-on-invalid-dep=yes --nodes=1 -J 2.0.find2.B -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.sh
# Submitted batch job 8091048

step: 3, depends on: 1.2, job name: merge, flag: merge reference:
depend on multiple jobs
sbatch -p short -t 10:0 -n 1 --kill-on-invalid-dep=yes --nodes=1 --dependency=afterok:8091045:8091047:8091046:8091048 -J 3.1.2.merge -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.sh
# Submitted batch job 8091049
all submitted jobs:
job_id depend_on job_flag
8091045 null 1.0.find1.A
8091046 null 2.0.find2.A
8091047 null 1.0.find1.B
8091048 null 2.0.find2.B
8091049 ..8091045.8091047..8091046.8091048 3.1.2.merge
---------------------------------------------------------

...

For details about the second option: Get more informative slurm email notification and logs through rcbio/1.2

To run your own script as Slurm pipeline

...

In case you wonder how it works, here is a simple example to explain:

For each step per loop, the pipeline runner creates a file looks like this (here it is named flag.sh):

Code Block

#!/bin/bash 
srun -n 1 bash -c "{ echo I am running...; hostname; otherCommands; } && touch flag.success" 
sleep 5 
export SLURM_TIME_FORMAT=relative 
echo Job done. Summary: 
sacct --format=JobID,Submit,Start,End,State,Partition,ReqTRES%30,CPUTime,MaxRSS,NodeList%30 --units=M -j $SLURM_JOBID 
sendJobFinishEmail.sh flag 
[ -f flag.success ] && exit 0 || exit 1

Then submit with:

Code Block
sbatch -p short -t 10:0 -o flag.out -e flag.out flag.sh

...

Versions Compared

Old Version 38

New Version Current

Key

The modified bash script

Sample output from the test run

To run your own script as Slurm pipeline