Modules
The HPC is used by many people with diverse software requirements. You might require Python 3.7 for your script, and your colleague requires Python 2.7 for their script. How do we install both on the HPC?
Software Modules are used on the HPC to solve this issue of installing conflicting software. A software module is a partially installed version of the software. When you want to use this software, you need to activate so it is available to you. Activation “completes” the install in a temporary manner. When no longer required, the module can be removed.
https://mediahub.qut.edu.au/media/t/0_tb63nhf7
To see a list of modules on the system use the command:
module avail
This prints out a list of all the software on the HPC
You can limit the list adding a name (or partial name) to the command:
module avail tensorflow
We now see all the tensorflow modules and versions available to us.
Lets find and load python 3.7
which python
{system python}
module avail python
...
module load python/3.7...
which python
{module python}
We can see which modules are currently active by using:
module list
We can see python loaded other dependent modules.
To deactivate a module we use the unload command
module unload python
Python 3.7 is no longer available
which python
{system python}
To remove all modules, you can unload each one at at time, or use the purge command to remove all of them:
module list module purge module list
Module conflicts can occur when loading modules.
You cannot two different versions of the same package:
# Load R version 3.6.2
module load r/3.6.2-foss-2019b
R --version
# Load R version 4.0.3
module load r/4.0.3-foss-2020b
{Error}
Tool chain conflicts.
In the name r/4.0.3-foss-2020b, the foss-2020b part is the toolchain. This means “Free and Open Source Software” at version 2020b.
Let's try loading samtools and bowtie2:
module purge module load samtools module load bowtie2 gcccore/8.3.0(24):ERROR:150: Module 'gcccore/8.3.0' conflicts with the currently loaded module(s) 'gcccore/7.3.0' ...
Let’s check the versions of bowtie2 that are available:
module avail bowtie2
{output}
To load both packages, we need to find a common toolchain:
module avail samtools
{output}
It looks like foss-2017a is common in both, lets load them (but first purge existing modules)
module purge module load samtools/1.9-foss-2017a module load bowtie2/2.3.4.3-foss-2017a module list
Installing Software
Conda:
Conda is a package manager and can be used to install packages to your home folder.
Either you can load a conda module or download the miniconda package.
You use environments to separate package versions.
The conda modules can be displayed with:
module avail anaconda
Or a conda module with Mamba is available:
module avail mamba
-------------------------------------------------------------------------------------------- /pkg/suse12/modules/all -------------------------------------------------------------------------------------------- mamba/4.14.0-0
When you use a conda module, it is a good idea to run the 'conda init' command to update your shell files so conda functions correctly.
Alternatively, you can install miniconda to your home folder buy following the instructions. Be sure to choose 'yes' the update shell question.
Once you have run conda init, or installed Minconda, you should logout and log back in again to activate the shell changes.
See here for details:
HPC Software - Conda - QUT MediaHub
Using Singularity
Singularity runs software containers. A container is a packaged collection of software. The advantage of using a container is it is portable, and runs without installation.
By Hand
It is possible to download application source code, compile, and install to your home folder.
This is an advanced topic!
Start by module loading the compiler and tool chain you need to build the software…