|
On O2, we encourage cluster users to install the packages and software they need. One method to install packages and manage environments is to use conda, which is available through the conda2/4.2.13
or miniconda3/4.10.3
moduleour miniconda
modules. Conda manages dependencies by default when you install packages, which can make it easier to install software. Packages that can be installed with conda include Python modules, libraries, or executable programs.
Commonly used commands, examples
Command | Meaning |
---|---|
|
| shows the versions of conda installed on O2 |
| loads an individual conda module (replace version with an actual version) |
| see available conda environments |
| create conda environment named test_env (name the environment whatever you'd like) |
| create conda environment, and install some packages (bwa, bowtie, and star) on the fly |
| "activate" a conda environment |
located at | |
| exit current conda environment |
| delete a conda environment named test_env (may also require full path if installed to nonstandard location) |
| search for a package (replace numpy with the package of your choice) |
| install a package, and must be within a conda environment or this command will fail. (replace numpy with the package of your choice) |
Setup
To install packages on O2 using conda, you must first create a conda environment. Environments are simply directories in ~/.conda/envs/
that contain packages you installed. You "source" an environment to use those packages, and can "deactivate" to exit the environment. You can have multiple environments, and can switch between them.
...
Code Block |
---|
mfk8@compute-a-01-01:~$ module load miniconda3/423.101.30 |
Then the conda
command will be available:
Code Block |
---|
mfk8@compute-a-01-01:~$ which conda /n/app/miniconda3/423.101.30/bin/conda |
Running conda info
will return information about the current conda installation:
...
You can see available conda environments with conda info --envs.
If you have not created any conda environments yet, the only listing you will see is the root environment in /n/app/conda2miniconda3
. Cluster users do not have access to alter this.
...
You can create as many conda environments as you need. Environments are independent (changing one environment won't affect another). They can be used for different analyses, or perhaps if you need more than one version of the same tool. You can run conda info --envs
to list all of your conda environments.
Usage of conda init
At some point in their interactions with conda, users may be prompted to execute conda init
. Executing conda init
results in a block of initialization code being added to the bottom of the the executing user’s $HOME/.bashrc
profile, causing that conda distribution’s (base)
environment to be initialized upon login to O2. This block also enables the use of the conda activate
command.
The block looks something like this (depending on distribution):
Code Block |
---|
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/n/app/miniconda3/23.1.0/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/n/app/miniconda3/23.1.0/etc/profile.d/conda.sh" ]; then
. "/n/app/miniconda3/23.1.0/etc/profile.d/conda.sh"
else
export PATH="/n/app/miniconda3/23.1.0/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<< |
Research Computing does not recommend the use of conda init
on O2. There are two primary reasons for this:
This fixes the use of the specified version of conda (and all that entails) for the user on login, which may become the cause of package dependency mismatches and errors as a user requires either alternative conda or application versions for future projects and environments.
There have been reports of users experiencing high latency when logging in to O2 when they have
conda init
blocks present in their$HOME/.bashrc
files. This is due to an interaction between the login initialization processes and Research Computing’s login node process watchdog, combined with our existing authentication procedures; this interaction can cause logins can take upwards of minutes to resolve.
Research Computing recommends users either comment out or outright delete the conda init
block from their $HOME/.bashrc
files entirely, and get used to the source activate
(and load the appropriate associated conda module beforehand) approach. In particular, while using the O2-available miniconda modules, the source activate
method should always work.
If a user is leveraging their own personal anaconda/miniconda distribution (i.e., not available via O2’s module system), they can choose to ignore this section at their peril. Users that would like to request assistance in maintaining access to their local distributions without the use of conda init
can contact Research Computing at rchelp@hms.harvard.edu.
Installing Packages
To search for available versions of a package that can be installed, use conda search
:
...
Code Block |
---|
(test_env) mfk8@compute-a-01-01:~$ which python /n/app/miniconda3/423.101.30/bin/python |
...
Code Block |
---|
(test_env) mfk8@compute-a-01-01:~$ python --version
Python 3.9.5 |
If you want to use a specific version of Python with conda (strongly recommended), you can create a conda environment and install it. For example to create an environment using Python 3.6.5:
...
Code Block |
---|
mfk8@login01:~$ srun --pty -p interactive -t 0-2 bash mfk8@compute-a-01-01:~$ module load miniconda3/423.101.30 mfk8@compute-a-01-01:~$ module list Currently Loaded Modules: 1) miniconda3/423.101.30 (E) Where: E: Experimental mfk8@compute-a-01-01:~$ conda create -n my_env # truncated mfk8@compute-a-01-01:~$ source activate my_env # install example python package, scipy, which is available through conda: (my_env) mfk8@compute-a-01-01:~$ conda install scipy # truncated # see list of packages available in this conda environment: (my_env) mfk8@compute-a-01-01:~$ conda list # truncated # will report scipy in the list # test importing scipy in python to verify it is installed correctly (my_env) mfk8@compute-a-01-01:~$ python -c "import scipy" (my_env) mfk8@compute-a-01-01:~$ # exit environment (my_env) mfk8@compute-a-01-01:~$ source deactivate mfk8@compute-a-01-01:~$ |
...