Features of the new pipeline:
Submit each step as a cluster job using sbatch.
Automatically arrange dependencies among jobs.
Email notifications are sent when each job fails or succeeds.
If a job fails, all its downstream jobs automatically are killed.
When re-running the pipeline on the same data folder, if there are any unfinished jobs, the user is asked to kill them or not.
When re-running the pipeline on the same data folder, the user is asked to confirm to re-run or not if a step was done successfully earlier.
You can directly copy and paste the commands to test run the pipeline.
Start an interactive job, with a walltime of 2 hours, 2000MB of memory.
srun --pty -p interactive -t 0-02:0:0 --mem 2000MB -n 1 /bin/bash |
Create a working directory on scratch and change into the newly-created directory. For example, for user abc123, the working directory will be
mkdir /n/scratch/users/${USER:0:1}/${USER}/skewerFastQCHisat2HtseqCount
cd /n/scratch/users/${USER:0:1}/${USER}/skewerFastQCHisat2HtseqCount |
Copy some test data following this page: Build Folder Structures From Sample Sheet for rcbio NGS Workflows
Load necessary modules:
module load gcc/6.2.0 python/2.7.12 rcbio/1.3.3 |
Copy the example skewerFastQCHisat2HtseqCount.sh bash script:
cp /n/app/rcbio/1.3.3/bin/skewerFastQCHisat2HtseqCount.sh . |
Now you can modify the options as needed. For example, if you have single end data, you should add read length. Please reference the Hisat2 user manual if you have any questions.
To edit the Kallisto and Sleuth bash script:
nano skewerFastQCHisat2HtseqCount.sh |
To test the pipeline run the following command. Jobs will not be submitted to the scheduler.
runAsPipeline "skewerFastQCHisat2HtseqCount.sh -s no -r mm10 -a /n/groups/shared_databases/rcbio/skewer_adapters.fa" "sbatch -p short -t 2:0:0 -n 1 --mem 4G" noTmp # this is a test run |
To run the pipeline:
runAsPipeline "skewerFastQCHisat2HtseqCount.sh -s no -r mm10 -a /n/groups/shared_databases/rcbio/skewer_adapters.fa" "sbatch -p short -t 2:0:0 -n 1 --mem 4G" noTmp run 2>&1 | tee output.log # notice here 'run 2>&1 | tee output.log' is added to the command |
To understand how 'runAsPipeline' works, how to check output, how to re-run the pipeline, please visit: Run Bash Script As Slurm Pipeline
To instead run workflow on your own data, transfer the sample sheet to your local machine following this wiki page and modify the sample sheet. Then you can transfer it back to O2 under your account, then go to the build folder structure step.