Page Comparison

...

With useTmp, the pipeline runner copy related data to /tmp and all file paths will be automatically updated to reflect a file's location in /tmp when using the useTmp option. 

Sample output from the test run

...

You also get two emails for each step, one at the start of the step, one at the end of the step.

Cancel all jobs

You can use the command to cancel running and pending jobs:

cancelAllJobs flag/alljobs.jid

Re-run the pipeline

You can rerun this command in the same folder

...

This command will check if the earlier run is finished or not. If not, ask user to kill the running jobs or not, then ask user to rerun the successfully finished steps or not. Click 'y', it will rerun, directly press 'enter' key, it will not rerun. 

Re-run a single job manually

cd proper/directory
module load rcbio/1.1 and all/related/modules


#submit job with proper partition, time, number of cores and memory
sbatch --requeue --mail-type=ALL -p short -t 2:0:0 -c 2 --mem 2G /working/directory/flag/stepID.loopID.stepName.sh

To run your own script as Slurm pipeline

If you have a bash script with multiple steps and you wish to run it as Slurm pipeline, modify your old script and add the notation to mark the start and end of any loops, and the start of any step for which you want to submit as an sbatch job. Then you can use runAsPipeline with your modified bash script,as detailed above. 

...

In case you wonder how it works, here is a simple example to explain:

For each step per loop, the pipeline runner creates a file looks like this (here it is named flag.sh): 

#!/bin/bash 
srun -n 1 bash -c "{ echo I am running...; hostname; otherCommands; } && touch flag.success" 
sleep 5 
export SLURM_TIME_FORMAT=relative 
echo Job done. Summary: 
sacct --format=JobID,Submit,Start,End,State,Partition,ReqTRES%30,CPUTime,MaxRSS,NodeList%30 --units=M -j $SLURM_JOBID 
sendJobFinishEmail.sh flag 
[ -f flag.success ] && exit 0 || exit 1 

Then submit with: 

sbatch -p short -t 10:0 -o flag.out -e flag.out flag.sh

...