Installing Nextflow

Aims:

Install Nextflow in your HPC environment
Run a test Nextflow task to verify the installation process
eResearch portal: https://eresearchqut.atlassian.net/servicedesk/customer/portals

Background

What is nextflow?

Nextflow is a free and open-source pipeline management software that enables scalable and reproducible scientific workflows. It allows the adaptation of pipelines written in the most common scripting languages.
Key features of Nextflow:
- Reproducible → version control and use of containers ensure the reproducibility of nextflow pipelines
- Portable → compute agnostic (i.e., HPC, cloud, desktop)
- Scalable → run from a single to thousands of samples
- Minimal digital literacy → accessible to anyone
- Active global community → more and more nextflow pipelines are available (i.e., https://nf-co.re/pipelines )

Nextflow enables the ability to write, deploy, and share data-intensive, highly scalable, workflows on any infrastructure.

Nextflow is a pipeline engine that can take advantage of the batch nature of the HPC environment to efficiently and quickly run Bioinformatic workflows.

For more information about Nextflow, please visit Nextflow - A DSL for parallel and scalable computational pipelines

Work in the HPC

Work in the HPC

Installing Nextflow

Nextflow is meant to run from your home folder on a Linux machine like the HPC.

A few commands can install Nextflow. Copy and paste the following block of code into your terminal (i.e., PuTTy that is already connected to the terminal) and hit 'enter'

module load java
curl -s https://get.nextflow.io | bash
mv nextflow $HOME/bin
#verify Nextflow is installed
mkdir $HOME/nftemp && cd $HOME/nftemp
nextflow run hello
#check for output of running the short nextflow hello pipeline
cd $HOME
rm -rf nftemp

Line 1: The module load command is necessary to ensure java is available
Line 2: This command downloads and assembles the parts of nextflow - this step might take some time.
Line 3: When finished, the nextflow binary will be in the current folder so it should be moved to your “bin” folder” so it can be found later.
Line 5: Make a temporary folder for Nextflow to create files when it runs.
Line 6: Verify Nextflow is working.
Lines 7 and 8: Clean up

You should see something like this:

Nextflow’s Default Configuration

Once Nextflow is installed, there are some settings that should be applied to take advantage of the HPC environment. Nextflow has a hierarchy of configuration files, where the base configuration that is applied to every workflow you run is here:

$HOME/.nextflow/config

You can create a suitable file for use on the QUT HPC with this.

Copy and paste the following text into your Linux command line and hit ‘enter’. This will make a few small changes to your local account so that Nextflow can run correctly.

[[ -d $HOME/.nextflow ]] || mkdir -p $HOME/.nextflow
cat <<EOF > $HOME/.nextflow/config
singularity {
    cacheDir = '$HOME/NXF_SINGULARITY_CACHEDIR'
    autoMounts = true
    enabled = true
    runOptions = '-B /data1'
}

conda {
    cacheDir = '$HOME/NXF_CONDA_CACHEDIR'
}

process {
    executor = 'pbspro'
    scratch = true
    cleanup = false
}

includeConfig '/work/datasets/reference/nextflow/qutgenome.config' 

plugins {
    id 'nf-validation@1.1.1' // Validation of pipeline parameters and creation of an input channel from a samplesheet
    id 'nf-prov@1.2.1'       // Provenance reports for pipeline runs
}
EOF

check the creation of the folders: