Versions Compared

Old Version 11

changes.mady.by.user Kathleen Chappell

Saved on

compared with

New Version Current

changes.mady.by.user Alex Truong

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • profiles describes methods by which the pipeline can be invoked. This is specified at execution time via nextflow ... -p profilename1,profilename2,.... At least one profile name must be specified. The profile names in this file are in addition to the default profiles (the singularity profile in this file augments the default singularity profile implemented by Nextflow, etc.).

    • the singularity profile sets parameters to allow usage of Singularity containers on O2 to execute pipeline steps. You shouldn’t need to mess with this profile.

    • the cluster profile sets parameters to allow submission of pipeline steps via O2’s slurm scheduler.

      • the only parameter you may be interested in is the queue parameter, which governs which partition a pipeline step is submitted to.

        • If a pipeline step requires less than 12 hours, it is submitted to short. If less than 5 days, medium. Otherwise, long.

        • If you have access to additional partitions (such as mpi, highmem, contributed partitions, etc.), set queue accordingly.

          • Keep in mind that such special partitions do not have the same time governances (other than the 30 day limit) on them, so if you would like to integrate one or more of these partitions with the existing short / medium / long paradigm, you will likely need to modify one or more of the pipeline-specific configuration files as well. Please contact rchelp@hms.harvard.edu with inquiries about this.

          • If you are planning to use a specialized partition exclusively, then simply overwrite the queue specification with that partition name.

    • the local profile invites the pipeline to be executed within your existing resource allocation (e.g., inside the active interactive session). You need to make sure you have requested the MAXIMUM of cores and memory desired by any one step of the pipeline in order for this profile to execute successfully.

  • executor describes how the pipeline processes will be run (such as on local compute resources, on cloud resources, or by interacting with a cluster compute scheduler). The executor will keep track of each of the processes, and if they succeed or fail.

    • When using the slurm executor, Nextflow can submit each process in the workflow as an sbatch job.

      • Additional parameters that govern the Slurm job submission process are queueSize and submitRateLimit. The queueSize is how many tasks can be processed at one time; here we use 1900 tasks. The submitRateLimitis the maximum number of jobs that will be submitted for a specified time interval. In our file, we limit it to 20 jobs submitted per second.

    • When using the local executor, Nextflow will run each process using the resources available on the current compute node.

Executing Nextflow Pipelines

Once the NXF_SINGULARITY_CACHEDIR environment variable is set (assuming you are using the singularity profile), you have two options for invoking your pipeline:

  1. if the pipeline is an official nf-core pipeline, you can simply paste the command from the website and modify it to use the correct input, output, and profile settings.

  2. Otherwise, use nextflow run. A typical nextflow run command may look something like this:

    Code Block
    nextflow run REPONAME/PIPELINENAME -profile cluster,singularity -c /path/to/slurm.config -input /path/to/input -outdir /path/to/output

    You may need to refer to execution instructions provided by the pipeline maintainer.

To view a list of all pipelines you have ever downloaded/run, you can invoke the nextflow list command. These pipelines are located at $HOME/.nextflow/assets.

Cleaning Up After Execution

...

Modifying the configuration file

As mentioned above, the following variables may be modified depending on your job submission preferences:

  • queue - if you would like to submit to a contributed or other specialized access partition, you can replace this entire string with the appropriate partition name (e.g., highmem or gpu_quad). You will still need to make sure you have access to submit to that partition.

  • clusterOptions - this is a variable that allows you to set additional sbatch parameters (or even other execution parameters for pipeline steps, but that is outside the scope of this section). More information can be found in the Nextflow documentation for the clusterOptions flag. An example of its use here could be something like:

Code Block
profiles {

    ...

    cluster {
        process {
			executor = 'slurm'
			cache = 'lenient'
			queue = gpu_quad
			clusterOptions = '--gres=gpu:1, -x compute-g-17-[166-171]'
        }
    }
    
    ...

here, we are now submitting directly to the gpu_quad partition in all cases instead of dealing with the cpu-only partitions, and are excluding the compute nodes with L40S GPU cards (just as an example).

Do note that if you do plan to use gpu-based partitions, you should make sure that your process runtime limits do not exceed that of the partition’s specified limits (5 days in this case). If that is a concern, add the time field to the above block to override the process limits specified by the workflow:

Code Block
profiles {

    ...

    cluster {
        process {
			executor = 'slurm'
			cache = 'lenient'
			queue = gpu_quad
			clusterOptions = '--gres=gpu:1', '-x compute-g-17-[166-171]'
			time = '5.d'
        }
    }
    
    ...

Executing Nextflow Pipelines

Once the NXF_SINGULARITY_CACHEDIR environment variable is set (assuming you are using the singularity profile), you have two options for invoking your pipeline:

  1. if the pipeline is an official nf-core pipeline, you can simply paste the command from the website and modify it to use the correct input, output, and profile settings.

  2. Otherwise, use nextflow run. A typical nextflow run command may look something like this:

    Code Block
    nextflow run REPONAME/PIPELINENAME -profile cluster,singularity -c /path/to/slurm.config -input /path/to/input -outdir /path/to/output

    You may need to refer to execution instructions provided by the pipeline maintainer.

To view a list of all pipelines you have ever downloaded/run, you can invoke the nextflow list command. These pipelines are located at $HOME/.nextflow/assets.

Cleaning Up After Execution

After your pipeline completes, there will be work and .nextflow directories at the location where you executed the workflow (not to be confused with your output directory). You may find it useful to occasionally delete these directories, especially if you find that you are using far more space than anticipated. You can keep track of your utilization with the quota-v2 tool (see https://harvardmed.atlassian.net/wiki/spaces/O2/pages/1588662343/Filesystem+Quotas#Checking-Usage ).

...

There are also some workflows where when an workflow it fails for some reason, but you do not see an error explaining the failure unless you visit a log file within a subdirectory of the work folder. In such a case, you can refer to the pipeline_info directory where the workflow was started. Under pipeline_info, there will be a text file that named like execution_trace_YEAR_MONTH_DAY_TIME.txt and has details on each process in the workflow. More details on this trace file can be found in the Nextflow documentation here.

Within execution_trace*txt, you can focus on the processes that report statuses of FAILED or ABORTEDwhen troubleshooting. To find the associated folder and log for a process that did not complete successfully, look at the hash column. The relevant work subdirectory will match the hash column. For example, let’s say we have an execution trace file that contains:

...

within a subdirectory of the work folder. In such a case, you can refer to the output from nextflow log runName -f status,name,workdir. In that command, runName is the name that will be automatically assigned when your workflow is executed and the items after -f are columns to display in the output.

Code Block
$ nextflow log deadly_davinci -f status,name,workdir
COMPLETED	NFCORE_DEMO:DEMO:FASTQC (SAMPLE2_PE)	/n/groups/labname/abc123/nextflow_directory/work/7f/c4076aa7ac34ed830920cd6a38b7cc
COMPLETED	NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE2_PE)	/n/groups/labname/abc123/nextflow_directory/work/53/d42b6aed1d402fe707804dae414aba
COMPLETED	NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE3_SE)	/n/groups/labname/abc123/nextflow_directory/work/da/e3e73c94dd61553e52a3325ca025ef
COMPLETED	NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE1_PE)	/n/groups/labname/abc123/nextflow_directory/work/e7/34a6592fe45d20dc4c67ecbac661f1
COMPLETED	NFCORE_DEMO:DEMO:FASTQC (SAMPLE3_SE)	/n/groups/labname/abc123/nextflow_directory/work/4b/7249114061ce5255f622f027e94757
COMPLETED	NFCORE_DEMO:DEMO:FASTQC (SAMPLE1_PE)	FAILED	0	2024-10-21 15:57:59.516	32.1s	5s	247.0%	570 MB	63.7 GB	19.2 MB	4.6 MB
...

...

/n/groups/labname/abc123/nextflow_directory/work/b9/2ee82e060885b2b33900767db61abd
FAILED	    NFCORE_DEMO:DEMO:MULTIQC	/n/groups/labname/abc123/nextflow_directory/work/f4/1b760137eca3bfa11cfd90cba9301b

In the above output, each process that was run is displayed on its own line. The first column has the status for the process, the second column reports the name of the process, and the final column reports the workdir. The available columns that can be reported with nextflow log can be seen via

Code Block
nextflow log -l

Definitions for each of these fields can be found in the Nextflow documentation here.

All but one of our processes had COMPLETED status, meaning everything executed as expected. We would need to troubleshoot steps that report FAILED or ABORTED, of which we have one (the MULTIQC step). To find the associated files for a process, look at the last column; this has location of the associated subdirectory of the work folder. Depending on the workflow, there may or may not be a log file with useful error messages contained within the process directory.

...