While we have installed a small number of highly-used packages to each version of Python, we encourage users to maintain their own Python packages. This ensures that the user has no discontinuity in their workflow waiting for packages to be installed and that packages work with the user's chosen, loaded Python version module. Python packages are built specifically to a version of Python and may not work properly if a different version of Python is loaded. For example, there is still a great disparity between Python 2 and Python 3. O2's "module" feature allows users to quickly and easily switch between Python versions.
module spider python
shows the versions of Python installed on O2
module load python/version
loads an individual module (substitute version for an actual version)
module unload python/version
unloads an individual module version
shows the current version of Python loaded (use python for a python 2.x module, python3 for a python 3.x module)
shows what packages are installed for currently loaded Python module
You can use module avail python as an alternative to module spider python, but avail will only work if you have some gcc module loaded (avail only searches exposed hierarchies, where spider searches the entire module list).
Setting Up a Virtual Environment
You’ll want to first get an interactive session, especially if you plan on installing scientific packages. compute-*-* is a placeholder for whatever compute node you land on after requesting your interactive session.
mfk8@login01:~$ srun --pty -p interactive -t 0-1:00 --mem=1G bash # adjust srun parameters accordingly as needed; you may need to request more memory if installing large packages
mfk8@compute-*-*:~$ module load gcc/9.2.0
mfk8@compute-*-*:~$ module avail python
mfk8@compute-*-*:~$ module load python/3.8.12 # or whichever version you'd like here
mfk8@compute-*-*:~$ which virtualenv
mfk8@compute-*-*:~$ virtualenv nameyourenvhere # Please read on before executing this command as-is
These commands create a copy of our Python distribution in whichever directory you ran the command, placing it in a folder named after whatever you specified. You can name your environment however you'd like. The above examples generate a directories located at ~/nameyourenvhere or ~/foobar, respectively.
If you'd like to use the packages that are pre-installed in the Python module, include the --system-site-packages flag when creating the virtual environment:
The --system-site-packages flag will allow your virtual environment to inherit all existing installed packages in that Python distribution (which means you don't need to install your own numpy or scipy). On O2, we have purposely kept the list of installed python modules as light as possible so that the user is afforded the maximum amount of flexibility to install the versions they need. You can create as many virtual environments as you need.
Note that for the python/3.7.4 module, the --system-site-packages flag will NOT work as expected; this is because the packages that have been installed for this module live in an external directory, and are exposed to the module via modification of the PYTHONPATH environment variable. If you wish to install a different version of a library from the one that is installed for 3.7.4, you will need to unset the PYTHONPATH variable first such that this external directory is no longer visible to the module (and to your virtual environment). The command to do so is simply:
mfk5@login01:~$ unset PYTHONPATH
Be aware that if you have other tools or configurations in your environment that leverage the use of PYTHONPATH, the above command will cause you to lose access to all modules that are exposed by this environment variable. If you encounter this situation, we recommend contacting us so that we can advise on alternate procedures.
To uninstall a virtual environment, simply rm -rf the folder containing the directory:
mfk8@compute-*-*:~$ rm -rf nameyourenvhere
To begin using the virtual environment, it needs to be activated. For a virtual environment located at ~/venv:
mfk8@compute-*-*:~$ source venv/bin/activate
Your prompt will now look (something like) this:
From here you an confirm that you're using the python associated with your virtual environment:
(venv)mfk8@compute-*-*:~$ which python3
If it points instead to something like /n/app/python/<version>/bin/python or /usr/bin/python, that means your environment is not active. Once you have confirmed that you are indeed inside your virtual environment, you may do whatever you'd like, such as testing code or installing packages.
To deactivate the environment, simply type deactivate, to see the corresponding change to your terminal prompt:
The executable for Python 3.x is python3 and not python, the choice was made by the Developers to mark a clear difference between version 2.x and 3.x, as old python 2.x scripts might not be fully compatible across the version jump.
A symbolic link python → python3 is created within each python 3.x virtual environment, this link allows to execute python3 scripts using python instead of python3 when the virtual environment is active, however the link does not exist outside of the python virtual environment.
Installing Packages (via pip)
PyPI (The Python Package Index) is a repository of software for the Python programming language, with packages available listed at https://pypi.python.org/pypi.
Notice that your virtual environment needs to be active. The above command will search PyPI and install the package to your virtual environment. If it is not active, you will get a permissions error because you will not have write access to the original installation.
You can also install packages manually; please refer to the instructions provided by the corresponding package if this is required (e.g. a README/INSTALL file or a documentation page).
Using Python Packages
In general, you will access packages by importing them at the top of your Python scripts:
# actual code here
This will allow you to access that package (called a "module")'s functions. Some modules will have extended functionality; please refer to the appropriate usage instructions provided by the package for more information on how to access them.
A Full Example
Here is an example of loading the desired version of Python on O2, setting up a virtual environment and activating it, installing a package to this personal environment, running a Python script, and deactivating the virtual environment (2.x chosen here arbitrarily):
mfk8@login01:~$ srun --pty -p interactive -t 0-1:00 --mem=1G bash
mfk8@compute-*-*:~$ which python
mfk8@compute-*-*:~$ module load gcc/6.2.0
mfk8@compute-*-*:~$ module avail python
--------------- /n/app/lmod/lmod/modulefiles/Compiler/gcc/6.2.0 ----------------
python/2.7.12 (E) python/3.6.0 (E,D) python/3.7.4
D: Default Module
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".
mfk8@compute-*-*:~$ module load python/3.7.4
mfk8@compute-*-*:~$ which python3
mfk8@compute-*-*:~$ which virtualenv
mfk8@compute-*-*:~$ mkdir mypythonfolder && cd mypythonfolder
mfk8@compute-*-*:~/mypythonfolder$ virtualenv myvirtualenv
mfk8@compute-*-*:~/mypythonfolder$ source myvirtualenv/bin/activate
(myvirtualenv)mfk8@lcompute-*-*:~/mypythonfolder$ pip3 install numpy
(myvirtualenv)mfk8@compute-*-*:~/mypythonfolder$ echo "print 'hello world'" > myscript.py
(myvirtualenv)mfk8@compute-*-*:~/mypythonfolder$ python3 myscript.py
Errors resembling python3: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory
This error might arise if you have not loaded the corresponding python module prior to invoking the python executable in your virtual environment. Due to the nature of virtualenv, it is necessary (on O2) for the parent installation to be present in your session when your virtual environment is activated. This must be done in the correct order as well.
Solution: As an example, the error shown above points to a problem with (presumably) an environment based on the python/3.7.4 module, so the proper procedure would be to
1) deactivate the current virtual environment,
2) make sure the python/3.7.4 module is loaded,
3) re-activate the virtual environment
If it still fails, then it is unlikely that your virtual environment was created from one of the existing modules, and you may need to re-create it from scratch (or debug with whatever source the environment was from). Note that copying python environments, virtual or otherwise, from a local machine to the cluster will likely fail to work as intended. If an existing python module does not fit your needs, you will need to supply your own version of python, either via (ana)conda or installing it yourself. For further/directed guidance on this option, feel free to send a ticket to email@example.com.
Any errors related to setuptools.build_meta
This error might arise if you are installing software via pip3 using the python/3.6.0 module within a virtualenv which was created using --system-site-packages. This is due to a bug in pip that does not properly resolve the setuptools dependency for older versions (such as the one used in the aforementioned python module). This problem does not exist in the python/3.7.4 module. RC has no plans to fix this in the existing 3.6.0 module due to the possibility that it could break existing user virtual environments cluster-wide.
Solution: create another python/3.6.0 virtual environment without --system-site-packages; it will download and install a newer version of setuptools that is NOT dependent on the older version that is in the global install. Alternatively, use the python/3.7.4 module (for which --system-site-packages does not work as users may expect anyway - see above).
Most errors containing the word "Unicode"
This error might arise if you are using the python/2.7.12 module instead of the python/2.7.12-ucs4 module, or vice versa. These modules are mutually exclusive, and are generally not compatible with each other. This error is due to a Unicode character incompatibility with the package you are trying to access or install.
Solution: use whichever python/2.7.12 module you are not currently using (e.g. if currently using 2.7.12, use 2.7.12-ucs4, or vice versa). Alternatively (and preferably), use a python 3 module (e.g. python/3.6.0 or python/3.7.4), where this problem does not exist at all.
Most errors containing "Permission Denied"
This error might arise if you are attempting to install a package without a virtual environment active.
Solution: make sure a virtual environment is active before proceeding with the installation; usually this is most easily verified by checking if your terminal prompt has been modified with the name of your environment in parentheses (see the example code blocks above). Alternatively, make sure a python module is loaded before you attempt the installation; it is possible to execute both pip and pip3 without having any python modules loaded; these will invoke system pip and pip3, which will attempt to install packages to system directories, to which you assuredly do not have permission to write files.
"My environment isn't working as expected"
This behavior could arise for any number of reasons. The most common reason that RC sees is that you've forgotten to load the correct python module prior to activating your virtual environment. Because virtual environments (as configured on O2) are necessarily dependent on their source installations, it is required to load the python module from which you created that virtual environment before activating it, every time you use it. If you have made sure you have already done so, and in the correct order (load module THEN activate virtual environment), then please submit a ticket with as many details as possible (including terminal output and command history such that we know where your environment lives and how to replicate your issue) to firstname.lastname@example.org.
Package mismatch errors with environments that use python/3.7.4
You may run into a problem where your environment detects a version of a package that lives in /n/app/python/3.7.4-ext and is unable to access/uninstall/upgrade/downgrade to a version that you desire. The quickest way to address this is to execute unset PYTHONPATH after loading the python/3.7.4 module but before activating your virtual environment. If you have personally modified the $PYTHONPATH variable for other purposes, you may need to re-populate it after removing the module package path from the variable (you can check if/how it is populated prior to executing unset by running echo $PYTHONPATH). If /n/app/python/3.7.4-ext/lib/site-packages is not the only path on $PYTHONPATH and unset-ing it is not worthwhile, you may choose to manually excise the path from the variable using whatever methods you find appropriate, though this is far less trivial than simply unset-ing it and repopulating it afterward, and we are unable to provide any general guidance to do so. If you require assistance with this sort of surgical modification of $PYTHONPATH, please contact us at email@example.com and we can assist you with the process.
The pip install --user flag
You may come across this flag in the installation instructions for a python package. This option is NOT RECOMMENDED on O2. Using this flag results in your python package installing to $HOME/.local . This location does not change if you switch between python versions, so if there is any possibility that you need to use multiple python versions for your work (e.g., for different projects), your $HOME/.local directory can quickly fill up with packages that were installed against mismatching python versions, which can cause incompatibility issues later on. This behavior is why RC recommends usage of virtual environments to keep package lists completely independent of each other.
Relatedly, if you are experiencing execution errors with mismatched package versions, the first step to troubleshooting is to delete the $HOME/.local directory. If the error persists after taking this step, please contact firstname.lastname@example.org for further assistance.