eResearch - Software on the HPC

Modules

The HPC is used by many people with diverse software requirements. You might require Python 3.7 for your script, and your colleague requires Python 2.7 for their script. How do we install both on the HPC?

Software Modules are used on the HPC. A software module is a partially installed version of the software. When you need to use this software, you need to activate so it is available to you. Activation “completes” the install in a temporary manner. When no longer required, the module can be removed.

HPC Software - Modules - QUT MediaHub

To see a list of modules on the system use the command:

module avail

This prints out a list of all the software on the HPC

You can limit the list adding a name (or partial name) to the command:

module avail tensorflow

We now see all the tensorflow modules and versions available to us.

Lets find and load python 3.7

which python
{system python}
module avail python
...
module load python/3.7...
which python
{module python}

We can see which modules are currently active by using:

We can see python loaded other dependent modules.

To deactivate a module we use the unload command

Python 3.7 is no longer available

To remove all modules, you can unload each one at at time, or use the purge command to remove all of them:

Module conflicts can occur when loading modules.

You cannot two different versions of the same package:

Tool chain conflicts.

In the name r/4.0.3-foss-2020b, the foss-2020b part is the toolchain. This means “Free and Open Source Software” at version 2020b.

Lets try loading samtools and bowtie2:

Let’s check the versions of bowtie2 that are available:

To load both packages we need to find a common toolchain:

It looks like foss-2017a is common in both, lets load them (but first purge existing modules)

Installing Software

Conda:

Conda is a package manager and can be used to install packages to your home folder.

Either you can load a conda module or download the miniconda package.

You use environments to separate package versions.

See this for details:

HPC Software - Conda - QUT MediaHub

By Hand

It is possible to download application source code, compile, and install to your home folder.

This is an advanced topic!

Start by module loading the compiler and tool chain you need to build the software…