Local Software Installs

1 Quick Start
2 Creating Your Own Modules

On O2, users can compile and install software in their /home directories or shared directories (e.g. /n/groups , /n/dataX ) where you have write permission. In general, during the configure step, --prefix can be set to a directory where you have write access to install the software to that directory. Software installs via binary files do not need to be compiled, only uncompressed.

Quick Start

1) Download (wget or curl) or sFTP upload software source files (usually a tar.gz) to a write-accessible directory (like /home/mfk8) from an interactive shell (not "login0X").

Interactive shell:

mfk8@login02:~$ srun --pty -p interactive -t 0-12:00 /bin/bash

wget or curl software from an FTP or HTTP URL:

mfk8@compute-a-01-01:~$ wget ftp://someURL.tar.gz

2) Unpack file using tar from an interactive shell (please do not compile on shell login servers, denoted "loginX")

If the file is a tar.gz, unpack tarball:

mfk8@compute-a-01-01:~$ tar -zxvf yourtarfile.tar.gz

If the file is just .gz, unpack:

mfk8@compute-a-01-01:~$ gunzip yourgzipfile.gz

This creates a directory for the software with its contents.

3) Change to the directory created during the unpack, view README or install guide. This may indicate what compilers/libraries are needed to install. Check out module spider for available compilers and libraries.

mfk8@compute-a-01-01:~$ cd newsoftwarefolder
mfk8@compute-a-01-01:~/newsoftware$ less README

4) Load the compiler and any libraries necessary, as found in README, and make sure to specify --prefix or whatever the equivalent flag is if specified in the README:

mfk8@compute-a-01-01:~/newsoftwarefolder$ module load gcc/14.2.0
mfk8@compute-a-01-01:~/newsoftwarefolder$ ./configure --prefix=/home/mfk8/newsoftwarefolder

5) Usually run make then make install (check README/install notes)

mfk8@compute-a-01-01:~/newsoftwarefolder$ make
mfk8@compute-a-01-01:~/newsoftwarefolder$ make install

6) Software executables are generally now in a "bin" folder. You can add (prepend) these to your $PATH environment variable like so, to make them accessible simply by name:

mfk8@compute-a-01-01:~/newsoftwarefolder$ export PATH=/home/mfk8/newsoftwarefolder/bin:$PATH

If you download and install software into your /home or other directory, you are generally expected to adhere to any software license agreement. If you have questions about this or need help deploying a license, please feel encouraged to contact HMS Research Computing.

Creating Your Own Modules

Lmod allows end users to manage their own software in tandem with O2's software stack. RC encourages this, as it allows greater flexibility in installation requirements. For example, if a lab requires a newer version of an existing software and RC cannot help for whatever reason, the lab can set up their own installation, create a modulefile for it, then add it to their list of available modules. This section will briefly describe how to do this.

If you would like to leverage Lmod to handle your own set of installed applications, you need to know how these applications were built and where relevant components are located.

Once you have your program installed/the information on how it was installed, you need to create a modulefile for this program. A typical modulefile for a program on the O2 application stack will look something like this (using GCC 6.2.0, now defunct, as an example), written in Lua:

help([[
For detailed instructions, go to:

    https://gcc.gnu.org/
]])

whatis("Version: 6.2.0")
whatis("Keywords: compiler, gcc")
whatis("URL: https://gcc.gnu.org/")
whatis("Description: GCC stands for \"GNU Compiler Collection\". GCC is an integrated distribution of compilers for several major programming languages. These languages currently include C, C++, Objective-C, Objective-C++, Java, Fortran, Ada, and Go.")

prepend_path( "PATH",            "/n/app/gcc/6.2.0/bin")
prepend_path( "LD_LIBRARY_PATH", "/n/app/gcc/6.2.0/lib")
prepend_path( "LD_LIBRARY_PATH", "/n/app/gcc/6.2.0/lib64")
prepend_path( "MANPATH",         "/n/app/gcc/6.2.0/share/man")

add_property("state", "experimental")

family("gcc")

-- set up MODULEPATH for packages built by this compiler
local mroot = os.getenv("MODULEPATH_ROOT")
local mdir = pathJoin(mroot, "Compiler/gcc", "6.2.0")
prepend_path("MODULEPATH", mdir)

If your module is for internal consumption only (which will generally be the case), a number of the above fields are optional. For example, you might not need the metadata fields specified by whatis, nor should you need the add_property line (which handles exclusivity management - you cannot have two modules in the same family loaded at the same time). Depending on how complex your local application set is, you might not need the last section either, which exposes a new directory of modulefiles if this module is loaded; this is how Lmod restricts your view of the module tree based on your environment.

For simple programs that are primarily executables, you should only need one variable prepended, the PATH variable. If you need to access libraries as well, you will likely need to also prepend to LD_LIBRARY_PATH. MANPATH is for if you'd like to view the documentation associated with your program (which might not always be available). Where these paths point to is reliant on where your programs are installed.

Notably, in addition to using Lmod modulefile functions, you can also inject logic by writing instructions directly in Lua if necessary (as seen in the the last section of the modulefile above).

A page of the available functions can be found on the Lmod modulefile documentation page, and general information about setting up your personal modulefile directory can be found in TACC's advanced user guide.