While we have installed a small number of highly-used packages to each version of Python, we encourage users to maintain their own Python packages. This ensures that the user has no discontinuity in their workflow waiting for packages to be installed and that packages work with the user's chosen, loaded Python version module. Python packages are built specifically to a version of Python and may not work properly if a different version of Python is loaded. For example, there is still a great disparity between Python 2 and Python 3. O2's "module" feature allows users to quickly and easily switch between Python versions.
Useful commands:
Command | Meaning | |
|---|---|---|
|
| shows the versions of Python installed on O2 |
gcc/6.2.0 to be loaded)
| loads an individual module (substitute | ||
| unloads an individual module version | ||
| shows the current version of Python loaded (use | ||
| shows what packages are installed for currently loaded Python module |
Note that in order for You can use module avail python to give any output, you must have gcc/6.2.0 loaded into your environment. Alternately, you can use module spider python to circumvent this, but you will still need gcc/6.2.0 loaded in order to load any python modules that are installed as an alternative to module spider python, but avail will only work if you have some gcc module loaded (avail only searches exposed hierarchies, where spider searches the entire module list).
Setting Up a Virtual Environment
Setup
| Code Block |
|---|
mfk8@login01:~$ module load gcc/6.2.0
mfk8@login01:~$ module avail python
mfk8@login01:~$ module load python/3.7.4 # or whichever version you'd like here
mfk8@login01:~$ which virtualenv
mfk8@login01:~$ virtualenv nameyourenvhere # Please read on before executing this command as-is |
These commands create a copy of our Python distribution in whichever directory you ran the command, placing it in a folder named after whatever you specified. You can name your environment however you'd like. The above examples generate a directories located at ~/nameyourenvhere or ~/foobar, respectively.
If you'd like to use the packages that are pre-installed in the Python module, include the --system-site-packages flag when creating the virtual environment:
| Code Block |
|---|
mfk8@login01:~$ virtualenv nameyourenvhere --system-site-packages |
The --system-site-packages flag will allow your virtual environment to inherit all existing installed packages in that Python distribution (which means you don't need to install your own numpy or scipy). On O2, we have purposely kept the list of installed python modules as light as possible so that the user is afforded the maximum amount of flexibility to install the versions they need. You can create as many virtual environments as you need.
Note that for the python/3.7.4 module, the --system-site-packages flag will NOT work as expected; this is because the packages that have been installed for this module live in an external directory, and are exposed to the module via modification of the PYTHONPATH environment variable. If you wish to install a different version of a library from the one that is installed for 3.7.4, you will need to unset the PYTHONPATH variable first such that this external directory is no longer visible to the module (and to your virtual environment). The command to do so is simply:
| Code Block |
|---|
mfk5@login01:~$ unset PYTHONPATH |
Be aware that if you have other tools or configurations in your environment that leverage the use of PYTHONPATH, the above command will cause you to lose access to all modules that are exposed by this environment variable. If you encounter this situation, we recommend contacting us so that we can advise on alternate procedures.
To uninstall a virtual environment, simply rm -rf the folder containing the directory:
| Code Block |
|---|
mfk8@login01:~$ rm -rf nameyourenvhere |
Basic Usage
To begin using the virtual environment, it needs to be activated. For a virtual environment located at ~/venv:
| Code Block |
|---|
mfk8@login01:~$ source venv/bin/activate |
Your prompt You’ll want to first get an interactive session, especially if you plan on installing scientific packages. compute-*-* is a placeholder for whatever compute node you land on after requesting your interactive session.
Setup
| Code Block |
|---|
mfk8@login01:~$ srun --pty -p interactive -t 0-1:00 --mem=1G bash # adjust srun parameters accordingly as needed; you may need to request more memory if installing large packages
mfk8@compute-*-*:~$ module load gcc/9.2.0
mfk8@compute-*-*:~$ module avail python
mfk8@compute-*-*:~$ module load python/3.8.12 # or whichever version you'd like here
mfk8@compute-*-*:~$ which virtualenv
mfk8@compute-*-*:~$ virtualenv nameyourenvhere # Please read on before executing this command as-is |
These commands create a copy of our Python distribution in whichever directory you ran the command, placing it in a folder named after whatever you specified. You can name your environment however you'd like. The above examples generate a directories located at ~/nameyourenvhere or ~/foobar, respectively.
If you'd like to use the packages that are pre-installed in the Python module, include the --system-site-packages flag when creating the virtual environment:
| Code Block |
|---|
mfk8@compute-*-*:~$ virtualenv nameyourenvhere --system-site-packages |
The --system-site-packages flag will allow your virtual environment to inherit all existing installed packages in that Python distribution (which means you don't need to install your own numpy or scipy). On O2, we have purposely kept the list of installed python modules as light as possible so that the user is afforded the maximum amount of flexibility to install the versions they need. You can create as many virtual environments as you need.
| Note |
|---|
Note that for the
Be aware that if you have other tools or configurations in your environment that leverage the use of |
To uninstall a virtual environment, simply rm -rf the folder containing the directory:
| Code Block |
|---|
mfk8@compute-*-*:~$ rm -rf nameyourenvhere |
Basic Usage
To begin using the virtual environment, it needs to be activated. For a virtual environment located at ~/venv:
| Code Block |
|---|
mfk8@compute-*-*:~$ source venv/bin/activate |
Your prompt will now look (something like) this:
| Code Block |
|---|
(venv)mfk8@login01mfk8@compute-*-*:~$ |
From here you an confirm that you're using the python associated with your virtual environment:
| Code Block |
|---|
(venv)mfk8@login01:mfk8@compute-*-*:~$ which python3 ~/venv/bin/python3 |
...
To deactivate the environment, simply type deactivate, to see the corresponding change to your terminal prompt:
| Code Block |
|---|
(venv)mfk8@login01mfk8@compute-*-*:~$ deactivate mfk8@login01:~$ |
NOTE:
The executable for Python 3.x is python3 and not python, the choice was made by the Developers to mark a clear difference between version 2.x and 3.x, as as old python 2.x scripts might not be fully compatible across the version jump.
A symbolic link python → python3 is created within each python 3.x virtual environment, this link allows to execute python3 scripts using python instead of python3 when the virtual environment is active, however the link does not exist outside of the python virtual environment.
...
To install something, simply type:
| Code Block |
|---|
(venv)mfk8@login01mfk8@compute-*-*:~$ pip3 install nameofpackage |
...
| Code Block | ||
|---|---|---|
| ||
mfk8@login01:~$ which python /usr/bin/python mfk8@login01:~$ module load gcc/6.2.0 mfk8@login01 srun --pty -p interactive -t 0-1:00 --mem=1G bash ... mfk8@compute-*-*:~$ which python /usr/bin/python mfk8@compute-*-*:~$ module load gcc/6.2.0 mfk8@compute-*-*:~$ module avail python --------------- /n/app/lmod/lmod/modulefiles/Compiler/gcc/6.2.0 ---------------- python/2.7.12 (E) python/3.6.0 (E,D) python/3.7.4 Where: E: Experimental D: Default Module Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". mfk8@login01mfk8@compute-*-*:~$ module load python/3.7.4 mfk8@login01mfk8@compute-*-*:~$ which python3 /n/app/python/3.7.4/bin/python3 mfk8@login01mfk8@compute-*-*:~$ which virtualenv /n/app/python/2.7.12/bin/virtualenv mfk8@login01mfk8@compute-*-*:~$ mkdir mypythonfolder && cd mypythonfolder mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ virtualenv myvirtualenv (truncated) mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ source myvirtualenv/bin/activate (myvirtualenv)mfk8@login01mfk8@lcompute-*-*:~/mypythonfolder$ pip3 install numpy (truncated) (myvirtualenv)mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ echo "print 'hello world'" > myscript.py (myvirtualenv)mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ python3 myscript.py hello world (myvirtualenv)mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ deactivate mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ |
Troubleshooting
...
This behavior could arise for any number of reasons. The most common reason that RC sees is that you've forgotten to load the correct python module prior to activating your virtual environment. Because virtual environments (as configured on O2) are necessarily dependent on their source installations, it is required to load the python module from which you created that virtual environment before activating it, every time you use it. If you have made sure you have already done so, and in the correct order (load module THEN activate virtual environment), then please submit a ticket with as many details as possible (including terminal output and command history such that we know where your environment lives and how to replicate your issue) to rchelp@hms.harvard.edu.
| Note |
|---|
Package mismatch errors with environments that use |
The pip install --user flag
You may come across this flag in the installation instructions for a python package. This option is NOT RECOMMENDED on O2. Using this flag results in your python package installing to $HOME/.local . This location does not change if you switch between python versions, so if there is any possibility that you need to use multiple python versions for your work (e.g., for different projects), your $HOME/.local directory can quickly fill up with packages that were installed against mismatching python versions, which can cause incompatibility issues later on. This behavior is why RC recommends usage of virtual environments to keep package lists completely independent of each other.
Relatedly, if you are experiencing execution errors with mismatched package versions, the first step to troubleshooting is to delete the $HOME/.local directory. If the error persists after taking this step, please contact rchelp@hms.harvard.edu for further assistance.
Errors related to openssl
You may run into an error involving openssl >= 1.1.1 versions. The only known workaround will be to downgrade the package in question to a version that is using < 1.1.1. For an example of this, see https://harvardmed.atlassian.net/wiki/spaces/O2/pages/1594262530/Jupyter+on+O2#urllib3-openssl-error .