Page Comparison

While we have installed a small number of highly-used packages to each version of Python, we encourage users to maintain their own Python packages. This ensures that the user has no discontinuity in their workflow waiting for packages to be installed and that packages work with the user's chosen, loaded Python version module. Python packages are built specifically to a version of Python and may not work properly if a different version of Python is loaded. For example, there is still a great disparity between Python 2 and Python 3. O2's "module" feature allows users to quickly and easily switch between Python versions.

Useful commands:

module spider python

module load python/version

module unload python/version

which python
# or
which python3

pip freeze

You can use module avail python as an alternative to module spider python, but avail will only work if you have some gcc module loaded (avail only searches exposed hierarchies, where spider searches the entire module list).

Setting Up a Virtual Environment

You’ll want to first get an interactive session, especially if you plan on installing scientific packages. compute-*-* is a placeholder for whatever compute node you land on after requesting your interactive session.

Setup

mfk8@login01:~$ module load gcc/9.2.0
mfk8@login01:~$ module avail python
mfk8@login01:~$ module load python/3.8.12       # or whichever version you'd like here
mfk8@login01:~$ which virtualenv
mfk8@login01:~$ virtualenv nameyourenvhere      # Please read on before executing this command as-srun --pty -p interactive -t 0-1:00 --mem=1G bash # adjust srun parameters accordingly as needed; you may need to request more memory if installing large packages
mfk8@compute-*-*:~$ module load gcc/9.2.0
mfk8@compute-*-*:~$ module avail python
mfk8@compute-*-*:~$ module load python/3.8.12       # or whichever version you'd like here
mfk8@compute-*-*:~$ which virtualenv
mfk8@compute-*-*:~$ virtualenv nameyourenvhere      # Please read on before executing this command as-is

These commands create a copy of our Python distribution in whichever directory you ran the command, placing it in a folder named after whatever you specified. You can name your environment however you'd like. The above examples generate a directories located at ~/nameyourenvhere or ~/foobar, respectively.

If you'd like to use the packages that are pre-installed in the Python module, include the --system-site-packages flag when creating the virtual environment:

mfk8@login01mfk8@compute-*-*:~$ virtualenv nameyourenvhere --system-site-packages

...

To uninstall a virtual environment, simply rm -rf the folder containing the directory:

mfk8@login01mfk8@compute-*-*:~$ rm -rf nameyourenvhere

...

To begin using the virtual environment, it needs to be activated. For a virtual environment located at ~/venv:

mfk8@login01mfk8@compute-*-*:~$ source venv/bin/activate

Your prompt will now look (something like) this:

(venv)mfk8@login01mfk8@compute-*-*:~$

From here you an confirm that you're using the python associated with your virtual environment:

(venv)mfk8@login01mfk8@compute-*-*:~$ which python3
~/venv/bin/python3

...

To deactivate the environment, simply type deactivate, to see the corresponding change to your terminal prompt:

(venv)mfk8@login01mfk8@compute-*-*:~$ deactivate
mfk8@login01:~$

...

To install something, simply type:

(venv)mfk8@login01mfk8@compute-*-*:~$ pip3 install nameofpackage

...

mfk8@login01:~$ which python
/usr/bin/python
mfk8@login01srun --pty -p interactive -t 0-1:00 --mem=1G bash
...
mfk8@compute-*-*:~$ which python
/usr/bin/python
mfk8@compute-*-*:~$ module load gcc/6.2.0
mfk8@login01mfk8@compute-*-*:~$ module avail python

--------------- /n/app/lmod/lmod/modulefiles/Compiler/gcc/6.2.0 ----------------

   python/2.7.12 (E)    python/3.6.0 (E,D)  python/3.7.4

  Where:

   E:  Experimental
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".

mfk8@login01mfk8@compute-*-*:~$ module load python/3.7.4
mfk8@login01mfk8@compute-*-*:~$ which python3
/n/app/python/3.7.4/bin/python3
mfk8@login01mfk8@compute-*-*:~$ which virtualenv
/n/app/python/2.7.12/bin/virtualenv
mfk8@login01mfk8@compute-*-*:~$ mkdir mypythonfolder && cd mypythonfolder
mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ virtualenv myvirtualenv
(truncated)
mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ source myvirtualenv/bin/activate
(myvirtualenv)mfk8@login01mfk8@lcompute-*-*:~/mypythonfolder$ pip3 install numpy
(truncated)
(myvirtualenv)mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ echo "print 'hello world'" > myscript.py
(myvirtualenv)mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ python3 myscript.py
hello world
(myvirtualenv)mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ deactivate
mfk8@login01:mfk8@compute-*-*:~/mypythonfolder$

Troubleshooting

...

Relatedly, if you are experiencing execution errors with mismatched package versions, the first step to troubleshooting is to delete the $HOME/.local directory. If the error persists after taking this step, please contact rchelp@hms.harvard.edu for further assistance.

Errors related to openssl

You may run into an error involving openssl >= 1.1.1 versions. The only known workaround will be to downgrade the package in question to a version that is using < 1.1.1. For an example of this, see https://harvardmed.atlassian.net/wiki/spaces/O2/pages/1594262530/Jupyter+on+O2#urllib3-openssl-error .