...
| Code Block |
|---|
runAsPipeline bashScriptV2.sh "sbatch -p short -t 10:0 -c 1" useTmp Fri Sep 24 09:46:15 EDT 2021 Running: /n/app/rcbio/1.3.12/bin/runAsPipeline bashScriptV2.sh sbatch -p short -t 10:0 -c 1 useTmp Currently Loaded Modules: 1) rcbio/1.3.12 converting bashScriptV2.sh to flag/slurmPipeLine.a855454a70b2198fa5b2643bb1d41762.sh find loop start: for i in A B; do find job marker: #@1,0,find1,u,sbatch -p short -c 1 -t 50:0 sbatch options: sbatch -p short -c 1 -t 50:0 find job: grep -H John $u >> John.txt; grep -H Mike $u >> Mike.txt find job marker: #@2,0,find2,u,sbatch -p short -c 1 -t 50:0 sbatch options: sbatch -p short -c 1 -t 50:0 find job: grep -H Nick $u >> Nick.txt; grep -H Julia $u >> Julia.txt find loop end: done find job marker: #@3,1.2,merge find job: cat John.txt Mike.txt Nick.txt Julia.txt > all.txt flag/slurmPipeLine.a855454a70b2198fa5b2643bb1d41762.sh bashScriptV2.sh is ready to run. Starting to run ... Running flag/slurmPipeLine.a855454a70b2198fa5b2643bb1d41762.sh bashScriptV2.sh Currently Loaded Modules: 1) rcbio/1.3.2 --------------------------------------------------------- step: 1, depends on: 0, job name: find1, flag: find1.A reference: .u depend on no job sbatch -p short -c 1 -t 50:0 --requeue --nodes=1 -J 1.0.find1.A -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.sh # This is testing, so no job is submitted. In real run it should submit job such as: Submitted batch job 1349 step: 2, depends on: 0, job name: find2, flag: find2.A reference: .u depend on no job sbatch -p short -c 1 -t 50:0 --requeue --nodes=1 -J 2.0.find2.A -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.sh # This is testing, so no job is submitted. In real run it should submit job such as: Submitted batch job 1560 step: 1, depends on: 0, job name: find1, flag: find1.B reference: .u depend on no job sbatch -p short -c 1 -t 50:0 --requeue --nodes=1 -J 1.0.find1.B -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.sh # This is testing, so no job is submitted. In real run it should submit job such as: Submitted batch job 1766 step: 2, depends on: 0, job name: find2, flag: find2.B reference: .u depend on no job sbatch -p short -c 1 -t 50:0 --requeue --nodes=1 -J 2.0.find2.B -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.sh # This is testing, so no job is submitted. In real run it should submit job such as: Submitted batch job 1970 step: 3, depends on: 1.2, job name: merge , flag: merge reference: depend on multiple jobs sbatch -p short -t 10:0 -c 1 --requeue --nodes=1 --dependency=afterok:1349:1766:1560:1970 -J 3.1.2.merge -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.sh # This is testing, so no job is submitted. In real run it should submit job such as: Submitted batch job 2172 All submitted jobs: job_id depend_on job_flag 1349 null 1.0.find1.A 1560 null 2.0.find2.A 1766 null 1.0.find1.B 1970 null 2.0.find2.B 2172 ..1349.1766..1560.1970 3.1.2.merge --------------------------------------------------------- Note: This is just a test run, so no job is actually submitted. In real run it should submit jobs and report as above. |
...
| Code Block |
|---|
runAsPipeline bashScriptV2.sh "sbatch -p short -t 10:0 -c 1" useTmp run # Below is the output Fri Sep 24 09:48:12 EDT 2021 Running: /n/app/rcbio/1.3.12/bin/runAsPipeline bashScriptV2.sh sbatch -p short -t 10:0 -c 1 useTmp run Currently Loaded Modules: 1) rcbio/1.3.12 converting bashScriptV2.sh to flag/slurmPipeLine.a855454a70b2198fa5b2643bb1d41762.run.sh find loop start: for i in A B; do find job marker: #@1,0,find1,u,sbatch -p short -c 1 -t 50:0 sbatch options: sbatch -p short -c 1 -t 50:0 find job: grep -H John $u >> John.txt; grep -H Mike $u >> Mike.txt find job marker: #@2,0,find2,u,sbatch -p short -c 1 -t 50:0 sbatch options: sbatch -p short -c 1 -t 50:0 find job: grep -H Nick $u >> Nick.txt; grep -H Julia $u >> Julia.txt find loop end: done find job marker: #@3,1.2,merge find job: cat John.txt Mike.txt Nick.txt Julia.txt > all.txt flag/slurmPipeLine.a855454a70b2198fa5b2643bb1d41762.run.sh bashScriptV2.sh is ready to run. Starting to run ... Running flag/slurmPipeLine.a855454a70b2198fa5b2643bb1d41762.run.sh bashScriptV2.sh Currently Loaded Modules: 1) rcbio/1.3.2 Could not find any jobs to cancel. --------------------------------------------------------- step: 1, depends on: 0, job name: find1, flag: find1.A reference: .u depend on no job sbatch -p short -c 1 -t 50:0 --requeue --nodes=1 -J 1.0.find1.A -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.sh # Submitted batch job 41208893 step: 2, depends on: 0, job name: find2, flag: find2.A reference: .u depend on no job sbatch -p short -c 1 -t 50:0 --requeue --nodes=1 -J 2.0.find2.A -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.sh # Submitted batch job 41208894 step: 1, depends on: 0, job name: find1, flag: find1.B reference: .u depend on no job sbatch -p short -c 1 -t 50:0 --requeue --nodes=1 -J 1.0.find1.B -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.sh # Submitted batch job 41208895 step: 2, depends on: 0, job name: find2, flag: find2.B reference: .u depend on no job sbatch -p short -c 1 -t 50:0 --requeue --nodes=1 -J 2.0.find2.B -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.sh # Submitted batch job 41208898 step: 3, depends on: 1.2, job name: merge , flag: merge reference: depend on multiple jobs sbatch -p short -t 10:0 -c 1 --requeue --nodes=1 --dependency=afterok:41208893:41208895:41208894:41208898 -J 3.1.2.merge -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.sh # Submitted batch job 41208899 All submitted jobs: job_id depend_on job_flag 41208893 null 1.0.find1.A 41208894 null 2.0.find2.A 41208895 null 1.0.find1.B 41208898 null 2.0.find2.B 41208899 ..41208893.41208895..41208894.41208898 3.1.2.merge --------------------------------------------------------- |
...
| Code Block |
|---|
# We are intentionally removing an input file to see a "failed job" email message rm universityB.txt runAsPipeline bashScriptV2.sh "sbatch -p short -t 10:0 -c 1" useTmp run # Here is the output Fri Sep 24 10:00:36 EDT 2021 Running: /n/app/rcbio/1.3.12/bin/runAsPipeline bashScriptV2.sh sbatch -p short -t 10:0 -c 1 useTmp run Currently Loaded Modules: 1) rcbio/1.3.2 This is a re-run with the same command and script is not changed, no need to convert the script. Using the old one: flag/slurmPipeLine.a855454a70b2198fa5b2643bb1d41762.run.sh Running flag/slurmPipeLine.a855454a70b2198fa5b2643bb1d41762.run.sh bashScriptV2.sh Currently Loaded Modules: 1) rcbio/1.3.2 Could not find any jobs to cancel. --------------------------------------------------------- step: 1, depends on: 0, job name: find1, flag: find1.A reference: .u depend on no job 1.0.find1.A was done before, do you want to re-run it? y: To re-run this job, press y, then enter key. ystep: To re-run all jobs for step 3: hisatCount, type yall, then press enter key. yall: To re-run all jobs, type yallall, then press enter key. enter: To not re-run this job, directly press enter key. nstep: To not re-run all successful jobs for step 3: hisatCount, type nall, then press enter key. nall: To not re-run all successful jobs, type nallall, then press enter key. # type enter here to not re-run step: 2, depends on: 0, job name: find2, flag: find2.A reference: .u depend on no job 2.0.find2.A was done before, do you want to re-run it? y: To re-run this job, press y, then enter key. ystep: To re-run all jobs for step 3: hisatCount, type yall, then press enter key. yall: To re-run all jobs, type yallall, then press enter key. enter: To not re-run this job, directly press enter key. nstep: To not re-run all successful jobs for step 3: hisatCount, type nall, then press enter key. nall: To not re-run all successful jobs, type nallall, then press enter key. # type enter here to not re-run job 2.0.find2.A is not submitted step: 1, depends on: 0, job name: find1, flag: find1.B reference: .u depend on no job 1.0.find1.B was done before, do you want to re-run it? y: To re-run this job, press y, then enter key. ystep: To re-run all jobs for step 3: hisatCount, type yall, then press enter key. yall: To re-run all jobs, type yallall, then press enter key. enter: To not re-run this job, directly press enter key. nstep: To not re-run all successful jobs for step 3: hisatCount, type nall, then press enter key. nall: To not re-run all successful jobs, type nallall, then press enter key. # type ‘y’ and enter here to re-run Will re-run the down stream steps even if they are done before. sbatch -p short -c 1 -t 50:0 --requeue --nodes=1 -J 1.0.find1.B -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.sh # Submitted batch job 41209197 step: 2, depends on: 0, job name: find2, flag: find2.B reference: .u depend on no job 2.0.find2.B was done before, do you want to re-run it? y: To re-run this job, press y, then enter key. ystep: To re-run all jobs for step 3: hisatCount, type yall, then press enter key. yall: To re-run all jobs, type yallall, then press enter key. enter: To not re-run this job, directly press enter key. nstep: To not re-run all successful jobs for step 3: hisatCount, type nall, then press enter key. nall: To not re-run all successful jobs, type nallall, then press enter key. # type enter here to not re-run job 2.0.find2.B is not submitted step: 3, depends on: 1.2, job name: merge , flag: merge reference: depend on other jobs sbatch -p short -t 10:0 -c 1 --requeue --nodes=1 --dependency=afterok:41209197 -J 3.1.2.merge -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.sh # Submitted batch job 41209210 # Notice above, rcbio didn’t ask if user wants to re-run step3 or not and directly re-run it. All submitted jobs: job_id depend_on job_flag 41209197 null 1.0.find1.B 41209210 ..41209197. 3.1.2.merge --------------------------------------------------------- |
...
| Code Block |
|---|
cp universityA.txt universityC.txt cp bashScriptV2.sh bashScriptV3.sh nano bashScriptV3.sh # change for i in A B; do to: for i in A B C; do # save the file and run: runAsPipeline bashScriptV3.sh "sbatch -p short -t 10:0 -c 1" useTmp run # Here are the output: Fri Sep 24 10:56:16 EDT 2021 Running: /n/app/rcbio/1.3.12/bin/runAsPipeline bashScriptV3.sh sbatch -p short -t 10:0 -c 1 useTmp run Currently Loaded Modules: 1) rcbio/1.3.12 converting bashScriptV3.sh to flag/slurmPipeLine.b72e7f91da30d312a2c85d0735896f79.run.sh find loop start: for i in A B C; do find job marker: #@1,0,find1,u,sbatch -p short -c 1 -t 50:0 sbatch options: sbatch -p short -c 1 -t 50:0 find job: grep -H John $u >> John.txt; grep -H Mike $u >> Mike.txt find job marker: #@2,0,find2,u,sbatch -p short -c 1 -t 50:0 sbatch options: sbatch -p short -c 1 -t 50:0 find job: grep -H Nick $u >> Nick.txt; grep -H Julia $u >> Julia.txt find loop end: done find job marker: #@3,1.2,merge find job: cat John.txt Mike.txt Nick.txt Julia.txt > all.txt flag/slurmPipeLine.b72e7f91da30d312a2c85d0735896f79.run.sh bashScriptV3.sh is ready to run. Starting to run ... Running flag/slurmPipeLine.b72e7f91da30d312a2c85d0735896f79.run.sh bashScriptV3.sh Currently Loaded Modules: 1) rcbio/1.3.12 Could not find any jobs to cancel. --------------------------------------------------------- step: 1, depends on: 0, job name: find1, flag: find1.A reference: .u depend on no job 1.0.find1.A was done before, do you want to re-run it? y: To re-run this job, press y, then enter key. ystep: To re-run all jobs for step 3: hisatCount, type yall, then press enter key. yall: To re-run all jobs, type yallall, then press enter key. enter: To not re-run this job, directly press enter key. nstep: To not re-run all successful jobs for step 3: hisatCount, type nall, then press enter key. nall: To not re-run all successful jobs, type nallall, then press enter key. # type enter here to not re-run job 1.0.find1.A is not submitted step: 2, depends on: 0, job name: find2, flag: find2.A reference: .u depend on no job 2.0.find2.A was done before, do you want to re-run it? y: To re-run this job, press y, then enter key. ystep: To re-run all jobs for step 3: hisatCount, type yall, then press enter key. yall: To re-run all jobs, type yallall, then press enter key. enter: To not re-run this job, directly press enter key. nstep: To not re-run all successful jobs for step 3: hisatCount, type nall, then press enter key. nall: To not re-run all successful jobs, type nallall, then press enter key. # type enter here to not re-run job 2.0.find2.A is not submitted step: 1, depends on: 0, job name: find1, flag: find1.B reference: .u depend on no job 1.0.find1.B was done before, do you want to re-run it? y: To re-run this job, press y, then enter key. ystep: To re-run all jobs for step 3: hisatCount, type yall, then press enter key. yall: To re-run all jobs, type yallall, then press enter key. enter: To not re-run this job, directly press enter key. nstep: To not re-run all successful jobs for step 3: hisatCount, type nall, then press enter key. nall: To not re-run all successful jobs, type nallall, then press enter key. # type enter here to not re-run job 1.0.find1.B is not submitted step: 2, depends on: 0, job name: find2, flag: find2.B reference: .u depend on no job 2.0.find2.B was done before, do you want to re-run it? y: To re-run this job, press y, then enter key. ystep: To re-run all jobs for step 3: hisatCount, type yall, then press enter key. yall: To re-run all jobs, type yallall, then press enter key. enter: To not re-run this job, directly press enter key. nstep: To not re-run all successful jobs for step 3: hisatCount, type nall, then press enter key. nall: To not re-run all successful jobs, type nallall, then press enter key. # type enter here to not re-run job 2.0.find2.B is not submitted step: 1, depends on: 0, job name: find1, flag: find1.C reference: .u depend on no job sbatch -p short -c 1 -t 50:0 --requeue --nodes=1 -J 1.0.find1.C -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.C.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.C.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.C.sh # Submitted batch job 41211380 step: 2, depends on: 0, job name: find2, flag: find2.C reference: .u depend on no job sbatch -p short -c 1 -t 50:0 --requeue --nodes=1 -J 2.0.find2.C -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.C.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.C.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.C.sh # Submitted batch job 41211381 step: 3, depends on: 1.2, job name: merge , flag: merge reference: depend on multiple jobs sbatch -p short -t 10:0 -c 1 --requeue --nodes=1 --dependency=afterok:41211380:41211381 -J 3.1.2.merge -o /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out -e /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out /home/ld32/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.sh # Submitted batch job 41211382 All submitted jobs: job_id depend_on job_flag 41211380 null 1.0.find1.C 41211381 null 2.0.find2.C 41211382 ..41211380..41211381 3.1.2.merge --------------------------------------------------------- |
...
| Code Block |
|---|
# /working/directory is a placeholder, replace it with your actual working directory path cd /working/directory # all/related/modules is a placeholder, replace it with the actual other modules/versions you need module load rcbio/1.3.12 and all/related/modules # submit job with proper partition, time, number of cores and memory sbatch --requeue --mail-type=ALL -p short -t 2:0:0 -c 2 --mem 2G /working/directory/flag/stepID.loopID.stepName.sh Or: runSingleJob "module load bowtie/1.2.2; bowtie -x /n/groups/shared_databases/bowtie_indexes/hg19 -p 2 -1 read1.fq -2 read2.fq --sam > out.bam" "sbatch -p short -t 1:0:0 -c 2 -mem 8G" |
...
Your analysis commands will be wrapped in an srun so we can monitor if it completed successfully. If your commands worked (meaning exited in 0 status), then we will create the success file. Next, we will run sacct to get stats for the job step, and will send a job completion email with sendJobFinishEmail.sh. The sendJobFinishEmail.sh script is available in /n/app/rcbio/1.3.12/bin/, if you are interested in the contents of that script.
...