|
This page shows you how to run a regular bash script as a pipeline. The runAsPipeline script, accessible through the rcbio/1.0 module, converts an input bash script to a pipeline that easily submits jobs to the Slurm scheduler for you.
...
| Code Block |
|---|
srun --pty -p interactive -t 0-12:0:0 --mem 2000MB -n 1 /bin/bash mkdir /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline cd /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline |
...
With useTmp, the pipeline runner copy related data to /tmp and all file paths will be automatically updated to reflect a file's location in /tmp when using the useTmp option.
Sample output from the test run
...
| Code Block |
|---|
runAsPipeline bashScriptV2.sh "sbatch -p short -t 10:0 -n 1" useTmp # Below is the output: converting bashScriptV2.sh to flag/slurmPipeLine.201801161424.sh find loopStart: for i in A B; do find job marker: #@1,0,find1,u: find job: grep -H John $u >> John.txt; grep -H Mike $u >> Mike.txt find job marker: #@2,0,find2,u,sbatch -p short -n 1 -t 50:0 sbatch options: sbatch -p short -n 1 -t 50:0 find job: grep -H Nick $u >> Nick.txt; grep -H Julia $u >> Julia.txt find loopend: done find job marker: #@3,1.2,merge: find job: cat John.txt Mike.txt Nick.txt Julia.txt > all.txt flag/slurmPipeLine.201801161424.sh .sh is ready to run. Starting to run ... Running flag/slurmPipeLine.201801161424.sh bashScriptV2.sh --------------------------------------------------------- step: 1, depends on: 0, job name: find1, flag: find1.A reference: .u depend on no job sbatch -p short -t 10:0 -n 1 --nodes=1 -J 1.0.find1.A -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.sh # Submitted batch job 123 step: 2, depends on: 0, job name: find2, flag: find2.A reference: .u depend on no job sbatch -p short -n 1 -t 50:0 --nodes=1 -J 2.0.find2.A -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.sh # Submitted batch job 123 step: 1, depends on: 0, job name: find1, flag: find1.B reference: .u depend on no job sbatch -p short -t 10:0 -n 1 --nodes=1 -J 1.0.find1.B -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.sh # Submitted batch job 123 step: 2, depends on: 0, job name: find2, flag: find2.B reference: .u depend on no job sbatch -p short -n 1 -t 50:0 --nodes=1 -J 2.0.find2.B -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.sh # Submitted batch job 123 step: 3, depends on: 1.2, job name: merge, flag: merge reference: depend on multiple jobs sbatch -p short -t 10:0 -n 1 --nodes=1 --dependency=afterok:123:123:123:123 -J 3.1.2.merge -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.sh # Submitted batch job 123 all submitted jobs: job_id depend_on job_flag 123 null 1.0.find1.A 123 null 2.0.find2.A 123 null 1.0.find1.B 123 null 2.0.find2.B 123 ..123.123..123.123 3.1.2.merge --------------------------------------------------------- |
...
| Code Block |
|---|
runAsPipeline bashScriptV2.sh "sbatch -p short -t 10:0 -n 1" useTmp run # Below is the output converting bashScriptV2.sh to flag/slurmPipeLine.201801101002.run.sh find loopStart: #loopStart,i find job marker: for i in A B; do find job: grep -H John $u >> John.txt; grep -H Mike $u >> Mike.txt find job marker: #@2,0,find2,u,sbatch -p short -n 1 -t 50:0 sbatch options: sbatch -p short -n 1 -t 50:0 find job: grep -H Nick $u >> Nick.txt; grep -H Julia $u >> Julia.txt find loopend: done find job marker: #@3,1.2,merge: find job: cat John.txt Mike.txt Nick.txt Julia.txt > all.txt flag/slurmPipeLine.201801101002.run.sh is ready to run. Starting to run ... Running flag/slurmPipeLine.201801101002.run.sh --------------------------------------------------------- step: 1, depends on: 0, job name: find1, flag: find1.A reference: .u depend on no job sbatch -p short -t 10:0 -n 1 --kill-on-invalid-dep=yes --nodes=1 -J 1.0.find1.A -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.sh # Submitted batch job 8091045 step: 2, depends on: 0, job name: find2, flag: find2.A reference: .u depend on no job sbatch -p short -n 1 -t 50:0 --kill-on-invalid-dep=yes --nodes=1 -J 2.0.find2.A -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.sh # Submitted batch job 8091046 step: 1, depends on: 0, job name: find1, flag: find1.B reference: .u depend on no job sbatch -p short -t 10:0 -n 1 --kill-on-invalid-dep=yes --nodes=1 -J 1.0.find1.B -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.sh # Submitted batch job 8091047 step: 2, depends on: 0, job name: find2, flag: find2.B reference: .u depend on no job sbatch -p short -n 1 -t 50:0 --kill-on-invalid-dep=yes --nodes=1 -J 2.0.find2.B -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.sh # Submitted batch job 8091048 step: 3, depends on: 1.2, job name: merge, flag: merge reference: depend on multiple jobs sbatch -p short -t 10:0 -n 1 --kill-on-invalid-dep=yes --nodes=1 --dependency=afterok:8091045:8091047:8091046:8091048 -J 3.1.2.merge -o /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out -e /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out /n/scratch3scratch/users/a/abc123/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.sh # Submitted batch job 8091049 all submitted jobs: job_id depend_on job_flag 8091045 null 1.0.find1.A 8091046 null 2.0.find2.A 8091047 null 1.0.find1.B 8091048 null 2.0.find2.B 8091049 ..8091045.8091047..8091046.8091048 3.1.2.merge --------------------------------------------------------- |
...
In case you wonder how it works, here is a simple example to explain:
For each step per loop, the pipeline runner creates a file looks like this (here it is named flag.sh):
| Code Block |
|---|
#!/bin/bash
srun -n 1 bash -c "{ echo I am running...; hostname; otherCommands; } && touch flag.success"
sleep 5
export SLURM_TIME_FORMAT=relative
echo Job done. Summary:
sacct --format=JobID,Submit,Start,End,State,Partition,ReqTRES%30,CPUTime,MaxRSS,NodeList%30 --units=M -j $SLURM_JOBID
sendJobFinishEmail.sh flag
[ -f flag.success ] && exit 0 || exit 1 |
Then submit with:
| Code Block |
|---|
sbatch -p short -t 10:0 -o flag.out -e flag.out flag.sh |
...