...
There are also some workflows where when an workflow fails for some reason, but you do not see an error explaining the failure unless you visit a log file within a subdirectory of the work folder. In such a case, you can refer to the pipeline_info directory where the workflow was started. Under pipeline_info, there will be a text file that named like execution_trace_YEAR_MONTH_DAY_TIME.txt and has details on each process in the workflow. More details on this trace file can be found in the Nextflow documentation here.
Within execution_trace*txt, you can focus on the processes that report statuses of FAILED or ABORTEDwhen troubleshooting. To find the associated folder and log for a process that did not complete successfully, look at the hash column. The relevant work subdirectory will match the hash column. For example, let’s say we have an execution trace file that contains:
...
output from nextflow log runName -f status,name,workdir. In that command, runName is the name that will be automatically assigned when your workflow is executed and the items after -f are columns to display in the output.
| Code Block |
|---|
$ nextflow log deadly_davinci -f status,name,workdir COMPLETED NFCORE_DEMO:DEMO:FASTQC (SAMPLE2_PE) /n/groups/labname/abc123/nextflow_directory/work/7f/c4076aa7ac34ed830920cd6a38b7cc COMPLETED NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE2_PE) /n/groups/labname/abc123/nextflow_directory/work/53/d42b6aed1d402fe707804dae414aba COMPLETED NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE3_SE) /n/groups/labname/abc123/nextflow_directory/work/da/e3e73c94dd61553e52a3325ca025ef COMPLETED NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE1_PE) /n/groups/labname/abc123/nextflow_directory/work/e7/34a6592fe45d20dc4c67ecbac661f1 COMPLETED NFCORE_DEMO:DEMO:FASTQC (SAMPLE3_SE) /n/groups/labname/abc123/nextflow_directory/work/4b/7249114061ce5255f622f027e94757 COMPLETED NFCORE_DEMO:DEMO:FASTQC (SAMPLE1_PE) FAILED 0 2024-10-21 15:57:59.516 32.1s 5s 247.0% 570 MB 63.7 GB 19.2 MB 4.6 MB ... |
...
/n/groups/labname/abc123/nextflow_directory/work/b9/2ee82e060885b2b33900767db61abd
FAILED NFCORE_DEMO:DEMO:MULTIQC /n/groups/labname/abc123/nextflow_directory/work/f4/1b760137eca3bfa11cfd90cba9301b |
In the above output, each process that was run is displayed on its own line. The first column has the status for the process, the second column reports the name of the process, and the final column reports the workdir. The available columns that can be reported with nextflow log can be seen via
| Code Block |
|---|
nextflow log -l |
All but one of our processes had COMPLETED status, meaning everything executed as expected. We would need to troubleshoot steps that report FAILED or ABORTED, of which we have one (the MULTIQC step). To find the associated files for a process, look at the last column; this has location of the associated subdirectory of the work folder. Depending on the workflow, there may or may not be a log file with useful error messages contained within the process directory.
...