While we have installed a small number of highly-used packages to each version of Python, we encourage users to maintain their own Python packages. This ensures that the user has no discontinuity in their workflow waiting for packages to be installed and that packages work with the user's chosen, loaded Python version module. Python packages are built specifically to a version of Python and may not work properly if a different version of Python is loaded. For example, there is still a great disparity between Python 2 and Python 3. O2's "module" feature allows users to quickly and easily switch between Python versions.
Useful commands:
Command | Meaning | |
---|---|---|
|
| shows the versions of Python installed on O2 |
gcc/6.2.0
to be loaded)
| loads an individual module (substitute | ||
| unloads an individual module version | ||
| shows the current version of Python loaded (use | ||
| shows what packages are installed for currently loaded Python module |
Note that in order for You can use module avail python
to give any output, you must have gcc/6.2.0
loaded into your environment. Alternately, you can use module spider python
to circumvent this, but you will still need gcc/6.2.0
loaded in order to load any python modules that are installed as an alternative to module spider python
, but avail
will only work if you have some gcc
module loaded (avail
only searches exposed hierarchies, where spider
searches the entire module list).
Setting Up a Virtual Environment
You’ll want to first get an interactive session, especially if you plan on installing scientific packages. compute-*-*
is a placeholder for whatever compute node you land on after requesting your interactive session.
Setup
Code Block |
---|
mfk8@login01:~$ srun --pty -p interactive -t 0-1:00 --mem=1G bash # adjust srun parameters accordingly as needed; you may need to request more memory if installing large packages mfk8@compute-*-*:~$ module load gcc/69.2.0 mfk8@login01mfk8@compute-*-*:~$ module avail python mfk8@login01mfk8@compute-*-*:~$ module load python/3.78.412 # or whichever version you'd like here mfk8@login01mfk8@compute-*-*:~$ which virtualenv mfk8@login01mfk8@compute-*-*:~$ virtualenv nameyourenvhere # Please read on before executing this command as-is |
...
If you'd like to use the packages that are pre-installed in the Python module, include the --system-site-packages
flag when creating the virtual environment:
Code Block |
---|
mfk8@login01mfk8@compute-*-*:~$ virtualenv nameyourenvhere --system-site-packages |
The --system-site-packages
flag will allow your virtual environment to inherit all existing installed packages in that Python distribution (which means you don't need to install your own numpy or scipy). On O2, we have purposely kept the list of installed python modules as light as possible so that the user is afforded the maximum amount of flexibility to install the versions they need. You can create as many virtual environments as you need.
Note |
---|
Note that for the |
Code Block |
Be aware that if you have other tools or configurations in your environment that leverage the use of |
To uninstall a virtual environment, simply rm -rf
the folder containing the directory:
Code Block |
---|
mfk8@login01mfk8@compute-*-*:~$ rm -rf nameyourenvhere |
...
To begin using the virtual environment, it needs to be activated. For a virtual environment located at ~/venv
:
Code Block |
---|
mfk8@login01mfk8@compute-*-*:~$ source venv/bin/activate |
Your prompt will now look (something like) this:
Code Block |
---|
(venv)mfk8@login01mfk8@compute-*-*:~$ |
From here you an confirm that you're using the python associated with your virtual environment:
Code Block |
---|
(venv)mfk8@login01mfk8@compute-*-*:~$ which python3 ~/venv/bin/python3 |
...
To deactivate the environment, simply type deactivate
, to see the corresponding change to your terminal prompt:
Code Block |
---|
(venv)mfk8@login01mfk8@compute-*-*:~$ deactivate mfk8@login01:~$ |
NOTE:
The executable for Python 3.x is python3 and not python, the choice was made by the Developers to mark a clear difference between version 2.x and 3.x, as as old python 2.x scripts might not be fully compatible across the version jump.
A symbolic link python → python3 is created within each python 3.x virtual environment, this link allows to execute python3 scripts using python instead of python3 when the virtual environment is active, however the link does not exist outside of the python virtual environment.
...
To install something, simply type:
Code Block |
---|
(venv)mfk8@login01mfk8@compute-*-*:~$ pip3 install nameofpackage |
...
Code Block | ||
---|---|---|
| ||
mfk8@login01:~$ srun --pty -p interactive -t 0-1:00 --mem=1G bash ... mfk8@compute-*-*:~$ which python /usr/bin/python mfk8@login01mfk8@compute-*-*:~$ module load gcc/6.2.0 mfk8@login01mfk8@compute-*-*:~$ module avail python --------------- /n/app/lmod/lmod/modulefiles/Compiler/gcc/6.2.0 ---------------- python/2.7.12 (E) python/3.6.0 (E,D) python/3.7.4 Where: E: Experimental D: Default Module Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". mfk8@login01mfk8@compute-*-*:~$ module load python/3.7.4 mfk8@login01mfk8@compute-*-*:~$ which python3 /n/app/python/3.7.4/bin/python3 mfk8@login01mfk8@compute-*-*:~$ which virtualenv /n/app/python/2.7.12/bin/virtualenv mfk8@login01mfk8@compute-*-*:~$ mkdir mypythonfolder && cd mypythonfolder mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ virtualenv myvirtualenv (truncated) mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ source myvirtualenv/bin/activate (myvirtualenv)mfk8@login01mfk8@lcompute-*-*:~/mypythonfolder$ pip3 install numpy (truncated) (myvirtualenv)mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ echo "print 'hello world'" > myscript.py (myvirtualenv)mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ python3 myscript.py hello world (myvirtualenv)mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ deactivate mfk8@login01mfk8@compute-*-*:~/mypythonfolder$ |
Troubleshooting
Errors resembling python3: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory
This error might arise if you have not loaded the corresponding python module prior to invoking the python executable in your virtual environment. Due to the nature of virtualenv, it is necessary (on O2) for the parent installation to be present in your session when your virtual environment is activated. This must be done in the correct order as well.
Solution: As an example, the error shown above points to a problem with (presumably) an environment based on the python/3.7.4 module, so the proper procedure would be to
1) deactivate the current virtual environment,
2) make sure the python/3.7.4 module is loaded,
3) re-activate the virtual environment
If it still fails, then it is unlikely that your virtual environment was created from one of the existing modules, and you may need to re-create it from scratch (or debug with whatever source the environment was from). Note that copying python environments, virtual or otherwise, from a local machine to the cluster will likely fail to work as intended. If an existing python module does not fit your needs, you will need to supply your own version of python, either via (ana)conda or installing it yourself. For further/directed guidance on this option, feel free to send a ticket to rchelp@hms.harvard.edu.
Any errors related to setuptools.build_meta
...
This behavior could arise for any number of reasons. The most common reason that RC sees is that you've forgotten to load the correct python module prior to activating your virtual environment. Because virtual environments (as configured on O2) are necessarily dependent on their source installations, it is required to load the python module from which you created that virtual environment before activating it, every time you use it. If you have made sure you have already done so, and in the correct order (load module THEN activate virtual environment), then please submit a ticket with as many details as possible (including terminal output and command history such that we know where your environment lives and how to replicate your issue) to rchelp@hms.harvard.edu.
Note |
---|
Package mismatch errors with environments that use |
The pip install --user
flag
You may come across this flag in the installation instructions for a python package. This option is NOT RECOMMENDED on O2. Using this flag results in your python package installing to $HOME/.local
. This location does not change if you switch between python versions, so if there is any possibility that you need to use multiple python versions for your work (e.g., for different projects), your $HOME/.local
directory can quickly fill up with packages that were installed against mismatching python versions, which can cause incompatibility issues later on. This behavior is why RC recommends usage of virtual environments to keep package lists completely independent of each other.
Relatedly, if you are experiencing execution errors with mismatched package versions, the first step to troubleshooting is to delete the $HOME/.local
directory. If the error persists after taking this step, please contact rchelp@hms.harvard.edu for further assistance.
Errors related to openssl
You may run into an error involving openssl
>= 1.1.1 versions. The only known workaround will be to downgrade the package in question to a version that is using < 1.1.1. For an example of this, see https://harvardmed.atlassian.net/wiki/spaces/O2/pages/1594262530/Jupyter+on+O2#urllib3-openssl-error .