Software on the HPC

Modules

The HPC is used by many people with diverse software requirements. You might require Python 3.7 for your script, and your colleague requires Python 2.7 for their script. How do we install both on the HPC?

Software Modules are used on the HPC. A software module is a partially installed version of the software. When you need to use this software, you need to activate so it is available to you. Activation “completes” the install in a temporary manner. When no longer required, the module can be removed.

https://mediahub.qut.edu.au/media/t/0_tb63nhf7

To see a list of modules on the system use the command:

module avail

This prints out a list of all the software on the HPC

You can limit the list adding a name (or partial name) to the command:

module avail tensorflow

We now see all the tensorflow modules and versions available to us.

Lets find and load python 3.7

which python
{system python}
module avail python
...
module load python/3.7...
which python
{module python}

We can see which modules are currently active by using:

module list

We can see python loaded other dependent modules.

To deactivate a module we use the unload command

module unload python

Python 3.7 is no longer available

which python
{system python}

To remove all modules, you can unload each one at at time, or use the purge command to remove all of them:

module list
module purge
module list

Module conflicts can occur when loading modules.

You cannot two different versions of the same package:

# Load R version 3.6.2 
module load r/3.6.2-foss-2019b
R --version

# Load R version 4.0.3
module load r/4.0.3-foss-2020b
{Error}

Tool chain conflicts.

In the name r/4.0.3-foss-2020b, the foss-2020b part is the toolchain. This means “Free and Open Source Software” at version 2020b.

Lets try loading samtools and bowtie2:

module purge
module load samtools
module load bowtie2
gcccore/8.3.0(24):ERROR:150: Module 'gcccore/8.3.0' conflicts with the currently loaded module(s) 'gcccore/7.3.0'
...

Let’s check the versions of bowtie2 that are available:

module avail bowtie2
{output}

To load both packages we need to find a common toolchain:

module avail samtools
{output}

It looks like foss-2017a is common in both, lets load them (but first purge existing modules)

module purge
module load samtools/1.9-foss-2017a
module load bowtie2/2.3.4.3-foss-2017a
module list

Installing Software

Conda:

Conda is a package manager and can be used to install packages to your home folder.

Either you can load a conda module or download the miniconda package.

You use environments to separate package versions.

See this for details:

HPC Software - Conda - QUT MediaHub

By Hand

It is possible to download application source code, compile, and install to your home folder.

This is an advanced topic!

Start by module loading the compiler and tool chain you need to build the software…