Prepared by the eResearch Office, QUT.
This page provides a guide to QUT users on how to install and run the nextflow nf-core/rnaseq workflow on the HPC.
Pre-requisites
Installed conda3 or miniconda3 ( https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html )
Basic Unix command line knowledge (example: https://researchcomputing.princeton.edu/education/external-online-resources/linux ; https://swcarpentry.github.io/shell-novice/ )
Familiarity with one unix text editors (example Vi/Vim or Nano):
Have an HPC account on QUT’s HPC compute. Apply for a new HPC account here.
R tutorials:
Install Nextflow
The nf-core/rnaseq workflow requires Nextflow to be installed in your account on the HPC. Find details on how to install and test Nextflow here. Prepare a nextflow.config file and run a PBS pro submission script for Nextflow pipelines.
Additional information is available here: https://nf-co.re/usage/installation
Additional details on the workflow can be found at:
Overview: https://nf-co.re/rnaseq/3.0
Usage: https://nf-co.re/rnaseq/3.0/usage
GitHub: https://github.com/nf-core/rnaseq
Pipeline Summary
FastQC - Raw read QC
UMI-tools extract - UMI barcode extraction
TrimGalore - Adapter and quality trimming
SortMeRNA - Removal of ribosomal RNA (optional)
STAR and Salmon - Fast spliced aware genome alignment and transcriptome quantification
STAR via RSEM - Alignment and quantification of expression levels
HISAT2 - Memory efficient splice aware alignment to a reference
SAMtools - Sort and index alignments
UMI-tools dedup - UMI-based deduplication
picard MarkDuplicates - Duplicate read marking
StringTie - Transcript assembly and quantification
BEDTools and bedGraphToBigWig - Create bigWig coverage files
RSeQC - Various RNA-seq QC metrics
Qualimap - Various RNA-seq QC metrics
dupRadar - Assessment of technical/biological read duplication
Preseq - Estimation of library complexity
featureCounts - Read counting relative to gene biotype
DESeq2 - PCA plot and sample pairwise distance heatmap and dendrogram
MultiQC - Present QC for raw reads, alignment, read counting, and sample similarity
Pseudo-alignment and quantification
Salmon - Wicked fast gene and isoform quantification relative to the transcriptome
Workflow reporting and genomes
Reference genome files - Saving reference genome indices/files
Pipeline information - Report metrics generated during the workflow execution
Getting Started
Download and run the workflow using minimal data provided by nf-core/rnaseq. We recommend using singularity as the profile for QUT’s HPC. Another profile option can be ‘conda.’ Note: the profile option ‘docker’ is unavailable on the HPC.
nextflow run nf-core/rnaseq -profile test,singularity --outdir results -r 3.10.1
Running the pipeline using custom data
Prepare a sample sheet file that specifies the input files to be used. To do this, we use an nf-core script to generate the ‘samplesheet.csv’ file as follows:
#download script and make executable
wget -L https://raw.githubusercontent.com/nf-core/rnaseq/master/bin/fastq_dir_to_samplesheet.py
chmod +x fastq_dir_to_samplesheet.py
#generate the samplesheet.csv file
fastq_dir_to_samplesheet.py /path/to/directory/containing/fastq_files/ samplesheet.csv \
--strandedness reverse \
--read1_extension R1.fastq.gz \
--read2_extension R2.fastq.gz
Example of a typical command to run an RNA-seq analysis for mouse samples:
nextflow run nf-core/rnaseq --input samplesheet.csv \
--outdir results \
-r 3.10.1 \
--genome GRCh38 \
-profile singularity \
--aligner star_rsem \
--clip_r1 10 \
--clip_r2 10 \
--three_prime_clip_r1 1 \
--three_prime_clip_r2 1
Note, if the running was interrupted or you did not complete a particular step, or you want to modify a parameter for a particular step, instead of re-running all processes again, nextflow enables you to “-resume” the workflow.
nextflow run nf-core/rnaseq --input samplesheet.csv \
--outdir results \
-r 3.10.1 \
--genome GRCh38 \
-profile singularity \
--aligner star_rsem \
--clip_r1 10 \
--clip_r2 10 \
--three_prime_clip_r1 1 \
--three_prime_clip_r2 1 \
-resume
Preparing a ‘samplesheet.csv’ file
Prepare an index.csv file containing the information of the samples to be processed. See below examples of index.csv files.
Example index.csv (previous versions):
group,replicate,fastq_1,fastq_2,strandedness control,1,/path/to/fastq/control-1_R1.fastq.gz,/path/to/fastq/control-1_R2.fastq.gz,unstranded control,2,/path/to/fastq/control-2_R1.fastq.gz,/path/to/fastq/control-2_R2.fastq.gz,unstranded control,3,/path/to/fastq/control-3_R1.fastq.gz,/path/to/fastq/control-3_R2.fastq.gz,unstranded infected,1,/path/to/fastq/infected-1_R1.fastq.gz,/path/to/fastq/infected-1_R2.fastq.gz,unstranded infected,2,/path/to/fastq/infected-1_R1.fastq.gz,/path/to/fastq/infected-2_R2.fastq.gz,unstranded infected,3,/path/to/fastq/infected-1_R1.fastq.gz,/path/to/fastq/infected-3_R2.fastq.gz,unstranded
Index format for current version 3.3:
group,fastq_1,fastq_2,strandedness control_rep1,/path/to/fastq/control-1_R1.fastq.gz,/path/to/fastq/control-1_R2.fastq.gz,unstranded control_rep2,/path/to/fastq/control-2_R1.fastq.gz,/path/to/fastq/control-2_R2.fastq.gz,unstranded control_rep3,/path/to/fastq/control-3_R1.fastq.gz,/path/to/fastq/control-3_R2.fastq.gz,unstranded infected_rep1,/path/to/fastq/infected-1_R1.fastq.gz,/path/to/fastq/infected-1_R2.fastq.gz,unstranded infected_rep2,/path/to/fastq/infected-1_R1.fastq.gz,/path/to/fastq/infected-2_R2.fastq.gz,unstranded infected_rep3,/path/to/fastq/infected-1_R1.fastq.gz,/path/to/fastq/infected-3_R2.fastq.gz,unstranded
Preparing to run on the HPC
To run this on the HPC a PBS submission script needs to be created using a text editor. For example, create a file called launch.pbs using a text editor of choice (i.e., vi or nano) and then copy and paste the code below:
#!/bin/bash -l #PBS -N nfrna2 #PBS -l select=1:ncpus=2:mem=4gb #PBS -l walltime=24:00:00 cd $PBS_O_WORKDIR module load java NXF_OPTS='-Xms1g -Xmx4g' nextflow run nf-core/rnaseq -profile singularity -r 3.3 --input index.csv --genome GRCm38 --aligner star_salmon
We recommend running the nextflow nf-core/rnaseq pipeline once and then assess the fastqc results folder to assess if sequence biases are present in the 5'-end and 3'-end ends of the sequences. Then, we can use the PBS script below to tell the pipeline to remove a defined number of bases from the 5'-end (--clip_r1 or --clip_r2) or 3'-end (--three_prime_clip_r1 or --three_prime_clip_r2). Also we can specify to remove ribosomal RNA as these sets of sequences are non-informative.
#!/bin/bash -l
#PBS -N nfrna2
#PBS -l select=1:ncpus=2:mem=4gb
#PBS -l walltime=24:00:00
#work on current directory (folder)
cd $PBS_O_WORKDIR
#load java and set up memory settings to run nextflow
module load java
NXF_OPTS='-Xms1g -Xmx4g'
#run the rnaseq pipeline
#with-dag can output files in .png, .pdf, .svg or .html
nextflow run nf-core/rnaseq -profile conda --input samplesheet.csv \
--genome GRCm38 \
--aligner star_salmon \
--min_mapped_reads 5 \
--clip_r1 10 \
--clip_r2 10 \
--three_prime_clip_r1 2 \
--three_prime_clip_r2 2 \
--remove_ribo_rna \
-dump-channels \
-with-dag flowchart.png
Submitting the job
Once you have created the folder for the run, the input.tsv file, nextflow.config and launch.pbs you are ready to submit.
Submit the run with this command (On Lyra)
qsub launch.pbs
Monitoring the Run
You can use the command
qstat -u $USER
Alternatively use the following command:
qjobs
To check on the jobs you are running. Nextflow will launch additional jobs during the run.
You can also check the .nextflow.log file for details on what is going on.
Finally, if you have configured the connection to the NFTower you can logon and check your run.