Page Comparison

https://mediahub.qut.edu.au/media/t/0_d0bsv333

Requesting Resources

Before you can run your software analysis on the HPC, suitable resources must be reserved for you. You must determine the resources needed to run your software before you submit your job. If you do not request enough CPUs, your analysis may run slower than intended. If your request does request enough memory, the PBS scheduler will kill your job if it tries to allocate more than the limit. The PBS scheduler will also kill your job if it has not finished before time you requested in your job.

It can be a chicken and the egg situation as you may not know what resources you need to request. At the end of your job, PBS will create a summary of the resources you used. You can use this summary to tune your job requests. Asking for more resources than you need is good to ensure your job finishes successfully, but the more resources you ask for, the longer it usually takes before your job starts. So if you over estimate, use the results to lower your request for the next job. A good place to start is if you have run the analysis on your laptop, you can start there, typically 4 cpus and 16gb.

Types of Resources

PBS tracks the following resources which you can request in your job:

• ncpus: The number of CPUs you need. Generally, the more CPUs you ask for, the faster your job will run (up to a limit). Specified as a number such as 2

• mem: The amount of memory your analysis needs. Typically requested in gigabytes (gb). Be generous here, if you do not request enough, your job will be killed! Specified as a number with a unit, 8gb

• walltime: How long you need to run your analysis. If you analysis is not finished by this time, it will be killed. Consider using a minimum of 1 hour to reduce the work the scheduler does. Specified in Hours:Minutes:Seconds, eg 4:00:00 for 4 hours, zero minutes and zero Seconds

Other resources can be requested but considered optional:

• ngpus: Select one or more GPUs for your job. Only select if you know your software can use a GPU

• cputype: Select the type of CPU you want to use. You can use Intel CPUs or AMD CPUs

Types of Jobs

When running software on the HPC, we do NOT run the apps on the Login Node. Since the Login node is shared amongst all the connected HPC users, we don’t want you slowing everyone else down, and we don’t want others slowing you down. To run any software, we need to submit a job.

The workhorse of the PBS is the Batch Job. This job type is where you request resources and software that will run without asking questions. This is important because if your software stops to ask you a question like “press Y to continue” there is no one to press Y. When you submit a Batch Job, PBS may run it at 3am the next morning. You do not have to wait, simply submit and move on.

Sometimes it may be difficult to find the right software that will run on the HPC, or you might need to experiment with providing command line options. An Interactive job can help here. When you submit an Interactive job, your terminal stops accepting input until the job is allocated to a node and starts. When it starts, you will be transfered to the node. This session is not shared, you can run apps without effecting others.

Helpful Tools

The command to submit jobs is qsub. qsub has many options and you find out all of them by accessing qsub’s man page.

To check on the status of your jobs, use the qstat command. Another tool written by QUT HPC staff is qjobs.

Launching an Interactive job

To launch an Interactive job, we need to supply the necessary options to qsub. Let's launch a small Interactive job, with 1 CPU and 1gb of memory for 1 hour

qsub -I -S /bin/bash -l select=1:ncpus=1:mem=1gb -l walltime=1:00:00

You will see:

qsub: wating for job {job id} to start
qsub: job {job id} ready

{username}@{node}:~>

You are now connected to the node assigned to your job and you can run commands. Notice how the server name (after the @ symbol) has changed.

Launching a Batch Job - Constructing a Job Script

...

While the name of the file is not important, I like to save my PBS job scripts as {name}.pbs to easily identify them in the file list. Use training01.pbs here.

Let’s use nano to create the file, nano is provided by a module:

module load nano

Create the file:

nano training01.pbs

Launching a Batch Job - Submitting a Job Script

Since all the options are contained in the job script, the qsub line is short:

qsub trainingtraining01.pbs

And you will see a job number printed on the screen. Use qjobs to check on the status of the job.

...

• Be notified when the job starts, use the -m option eg be sent an email if the job is aborted, when it begins, and when it ends: #PBS -m abe

• Give the job a name: To find your job in a long list give it a meaning name with the -N option: #PBS -N MyJob01

• Merge the error file into the standard output file: #PBS -j oe

• Overriding the email address: If you want to send the job notification email to another address, use the -M option, eg #PBS -M bob@bob.com

Example 2

From the Introduction to the Unix Shell for HPC users course lets run the do-stats.sh script as a job.

First, change to the folder:

cd ~/workshop/2024-2/shell-lesson-data/north-pacific-gyre

Then, create the submission script (copy and paste!)

#!/bin/bash -l
#PBS -N GooStatsRun01
#PBS -l select=1:ncpus=1:mem=2gb
#PBS -l walltime=00:30:00
#PBS -m abe
cd $PBS_O_WORKDIR

# Calculate stats for data files.
for datafile in NENE*A.txt NENE*B.txt
do
    echo $datafile
    bash goostats.sh $datafile stats-$datafile
done

Call this do-goostats.pbs

Now submit to the scheduler:

qsub do-goostats.pbs

And check the status:

qjobs

When run, check the output:

ls -ltr
cat GooStatsRun01.o{job_id}
...
CPU time  : 00:00:00
Wall time : 00:00:33
Mem usage : 4648kb
cat GooStatsRun01.e{job_id}
{Empty File}

Tricks and Tips

When the job starts, PBS will logon to the node as you and your working directory will be your home folder. If your data is in a sub folder or in a shared folder, you can use this to automatically change to that folder:

...

$PBS_O_WORKDIR is a special environment variable created by PBS. This will be the folder where you ran the qsub command.

Example 3

We can run commands to check and trim fastq sequences.

Start by changing directory:

cd $HOME/workshop/2024-2

Then, download the data and scripts (Copy and Paste!):

wget https://github.com/eresearchqut/hpc_training_3/archive/refs/tags/1.0.tar.gz

Then, unpack the downloaded file with the tar command

tar xvzf 1.0.tar.gz

Change into the newly create folder (Don’t forget TAB completion):

cd hpc_training_3-1.0

Examine the data and scripts

ls
cat stage1.pbs

stage1.pbs

#!/bin/bash -l
#PBS -N Stage1-FastQC
#PBS -m abe
#PBS -l select=1:ncpus=1:mem=4gb
#PBS -l walltime=1:00:00

# Change to folder where we launched the job
cd $PBS_O_WORKDIR

# Make a directory for the fastqc results
mkdir fastqc

# Load the fastqc module
module load fastqc/0.11.7-java-1.8.0_92

# Run fastqc
fastqc --threads 2  --outdir fastqc data/sample1_R1.fastq.gz data/sample1_R2.fastq.gz

This job script will load the fastqc app from the module system and run with the samples. This job script will produce the fastqc directory that holds the results of the run.

stage2.pbs

#!/bin/bash -l
#PBS -N Stage2-seqtk
#PBS -m abe
#PBS -l select=1:ncpus=2:mem=8gb
#PBS -l walltime=1:00:00

# Change to folder where we launched the job
cd $PBS_O_WORKDIR

# Make a directory for the seqtk results
mkdir trimmed

# Use Singularity to supply trim the files
singularity exec https://depot.galaxyproject.org/singularity/seqtk:1.4--he4a0461_1 bash -c \
  'seqtk trimfq data/sample1_R1.fastq.gz | \
  gzip --no-name > trimmed/sample1_R1.fastq.trimmed.gz'

singularity exec https://depot.galaxyproject.org/singularity/seqtk:1.4--he4a0461_1 bash -c \
  'seqtk trimfq data/sample1_R2.fastq.gz | \
  gzip --no-name > trimmed/sample1_R2.fastq.trimmed.gz'

This job script will run the seqtk app over the sample fastq files using default settings. Since seqtk is not available on the HPC, we can use singularity to supply the app. This script will saved the trimmed fastq files in the trimmed folder.

stage3.pbs

#!/bin/bash -l
#PBS -N Stage3-multiqc
#PBS -m abe
#PBS -l select=1:ncpus=2:mem=8gb
#PBS -l walltime=1:00:00

# Change to folder where we launched the job
cd $PBS_O_WORKDIR

# Make a directory for the seqtk results
mkdir multiqc

# Use Singularity to supply trim the files
singularity exec https://depot.galaxyproject.org/singularity/multiqc:1.21--pyhdfd78af_0 \
  multiqc --force --outdir multiqc data fastqc trimmed

This job script will run multiqc to create a summary of the data and operations of the previous stages. Like stage2, multiqc is not available on the HPC so we can run it via singularity. The results will be saved in the multiqc folder.

These files must be run one after each other, they cannot run at the same time.

Launch stage1.pbs

qsub stage1.pbs
qjobs
cat Stage1.o{tab}
cat Stage1.e{tab}

When finished, launch stage2.pbs

qsub stage2.pbs
qjobs
cat Stage2.o{tab}
cat Stage2.e{tab}

Finally, when stage2 is finished, launch stage3.pbs

qsub stage3.pbs
qjobs
cat Stage3.o{tab}
cat Stage3.e{tab}

These jobs will send you an email when the job starts and finishes. It can be handy to receive the email rather than watching qjobs etc.

Once stage3 is finished, open up the hpc_training_3-1.0 folder in Windows Explorer/Finder and examine the files, especially in the multiqc folder.

Image Added