Personal Python Packages

While we have installed a small number of highly-used packages to each version of Python, we encourage users to maintain their own Python packages. This ensures that the user has no discontinuity in their workflow waiting for packages to be installed and that packages work with the user's chosen, loaded Python version module. Python packages are built specifically to a version of Python and may not work properly if a different version of Python is loaded. For example, there is still a great disparity between Python 2 and Python 3. O2's "module" feature allows users to quickly and easily switch between Python versions.

Useful commands:

module avail python

module load python/version

module unload python/version

which python

pip freeze

Note that in order for module avail python to give any output, you must have gcc/6.2.0 loaded into your environment. Alternately, you can use module spider python to circumvent this, but you will still need gcc/6.2.0 loaded in order to load any python modules that are installed.

Setting Up a Virtual Environment

Setup

mfk8@login01:~$ module load gcc/6.2.0
mfk8@login01:~$ module avail python
mfk8@login01:~$ module load python/3.7.4       # or whichever version you'd like here
mfk8@login01:~$ which virtualenv
mfk8@login01:~$ virtualenv nameyourenvhere      # Please read on before executing this command as-is

These commands create a copy of our Python distribution in whichever directory you ran the command, placing it in a folder named after whatever you specified. You can name your environment however you'd like. The above examples generate a directories located at ~/nameyourenvhere or ~/foobar, respectively.

If you'd like to use the packages that are pre-installed in the Python module, include the --system-site-packages flag when creating the virtual environment:

mfk8@login01:~$ virtualenv nameyourenvhere --system-site-packages

The --system-site-packages flag will allow your virtual environment to inherit all existing installed packages in that Python distribution (which means you don't need to install your own numpy or scipy). On O2, we have purposely kept the list of installed python modules as light as possible so that the user is afforded the maximum amount of flexibility to install the versions they need. You can create as many virtual environments as you need.

Note that for the python/3.7.4 module, the --system-site-packages flag will NOT work as expected; this is because the packages that have been installed for this module live in an external directory, and are exposed to the module via modification of the PYTHONPATH environment variable. If you wish to install a different version of a library from the one that is installed for 3.7.4, you will need to unset the PYTHONPATH variable first such that this external directory is no longer visible to the module (and to your virtual environment). The command to do so is simply:

mfk5@login01:~$ unset PYTHONPATH

Be aware that if you have other tools or configurations in your environment that leverage the use of PYTHONPATH, the above command will cause you to lose access to all modules that are exposed by this environment variable. If you encounter this situation, we recommend contacting us so that we can advise on alternate procedures.

To uninstall a virtual environment, simply rm -rf the folder containing the directory:

mfk8@login01:~$ rm -rf nameyourenvhere

Basic Usage

To begin using the virtual environment, it needs to be activated. For a virtual environment located at ~/venv:

mfk8@login01:~$ source venv/bin/activate

Your prompt will now look (something like) this:

(venv)mfk8@login01:~$

From here you an confirm that you're using the python associated with your virtual environment:

(venv)mfk8@login01:~$ which python3
~/venv/bin/python3

If it points instead to something like /n/app/python/<version>/bin/python or /usr/bin/python, that means your environment is not active. Once you have confirmed that you are indeed inside your virtual environment, you may do whatever you'd like, such as testing code or installing packages.

To deactivate the environment, simply type deactivate, to see the corresponding change to your terminal prompt:

(venv)mfk8@login01:~$ deactivate
mfk8@login01:~$

NOTE:

The executable for Python 3.x is python3 and not python, the choice was made by the Developers to mark a clear difference between version 2.x and 3.x, as old python 2.x scripts might not be fully compatible across the version jump.

A symbolic link python → python3 is created within each python 3.x virtual environment, this link allows to execute python3 scripts using python instead of python3 when the virtual environment is active, however the link does not exist outside of the python virtual environment.

Installing Packages (via pip)

PyPI (The Python Package Index) is a repository of software for the Python programming language, with packages available listed at https://pypi.python.org/pypi.

To install something, simply type:

(venv)mfk8@login01:~$ pip3 install nameofpackage

Notice that your virtual environment needs to be active. The above command will search PyPI and install the package to your virtual environment. If it is not active, you will get a permissions error because you will not have write access to the original installation.

You can also install packages manually; please refer to the instructions provided by the corresponding package if this is required (e.g. a README/INSTALL file or a documentation page).

Using Python Packages

In general, you will access packages by importing them at the top of your Python scripts:

#!/usr/bin/env python3


import nameofpackage


# actual code here

This will allow you to access that package (called a "module")'s functions. Some modules will have extended functionality; please refer to the appropriate usage instructions provided by the package for more information on how to access them.

A Full Example

Here is an example of loading the desired version of Python on O2, setting up a virtual environment and activating it, installing a package to this personal environment, running a Python script, and deactivating the virtual environment (2.x chosen here arbitrarily):

mfk8@login01:~$ which python
/usr/bin/python
mfk8@login01:~$ module load gcc/6.2.0
mfk8@login01:~$ module avail python

--------------- /n/app/lmod/lmod/modulefiles/Compiler/gcc/6.2.0 ----------------

   python/2.7.12 (E)    python/3.6.0 (E,D)  python/3.7.4

  Where:

   E:  Experimental
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".

mfk8@login01:~$ module load python/3.7.4
mfk8@login01:~$ which python3
/n/app/python/3.7.4/bin/python3
mfk8@login01:~$ which virtualenv
/n/app/python/2.7.12/bin/virtualenv
mfk8@login01:~$ mkdir mypythonfolder && cd mypythonfolder
mfk8@login01:~/mypythonfolder$ virtualenv myvirtualenv
(truncated)
mfk8@login01:~/mypythonfolder$ source myvirtualenv/bin/activate
(myvirtualenv)mfk8@login01:~/mypythonfolder$ pip3 install numpy
(truncated)
(myvirtualenv)mfk8@login01:~/mypythonfolder$ echo "print 'hello world'" > myscript.py
(myvirtualenv)mfk8@login01:~/mypythonfolder$ python3 myscript.py
hello world
(myvirtualenv)mfk8@login01:~/mypythonfolder$ deactivate
mfk8@login01:~/mypythonfolder$

Troubleshooting

Errors resembling python3: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory

This error might arise if you have not loaded the corresponding python module prior to invoking the python executable in your virtual environment. Due to the nature of virtualenv, it is necessary (on O2) for the parent installation to be present in your session when your virtual environment is activated. This must be done in the correct order as well.

Solution: As an example, the error shown above points to a problem with (presumably) an environment based on the python/3.7.4 module, so the proper procedure would be to

1) deactivate the current virtual environment,

2) make sure the python/3.7.4 module is loaded,

3) re-activate the virtual environment

If it still fails, then it is unlikely that your virtual environment was created from one of the existing modules, and you may need to re-create it from scratch (or debug with whatever source the environment was from). Note that copying python environments, virtual or otherwise, from a local machine to the cluster will likely fail to work as intended. If an existing python module does not fit your needs, you will need to supply your own version of python, either via (ana)conda or installing it yourself. For further/directed guidance on this option, feel free to send a ticket to rchelp@hms.harvard.edu.

Any errors related to setuptools.build_meta

This error might arise if you are installing software via pip3 using the python/3.6.0 module within a virtualenv which was created using --system-site-packages. This is due to a bug in pip that does not properly resolve the setuptools dependency for older versions (such as the one used in the aforementioned python module). This problem does not exist in the python/3.7.4 module. RC has no plans to fix this in the existing 3.6.0 module due to the possibility that it could break existing user virtual environments cluster-wide.

Solution: create another python/3.6.0 virtual environment without --system-site-packages; it will download and install a newer version of setuptools that is NOT dependent on the older version that is in the global install. Alternatively, use the python/3.7.4 module (for which --system-site-packages does not work as users may expect anyway - see above).

Most errors containing the word "Unicode"

This error might arise if you are using the python/2.7.12 module instead of the python/2.7.12-ucs4 module, or vice versa. These modules are mutually exclusive, and are generally not compatible with each other. This error is due to a Unicode character incompatibility with the package you are trying to access or install.

Solution: use whichever python/2.7.12 module you are not currently using (e.g. if currently using 2.7.12, use 2.7.12-ucs4, or vice versa). Alternatively (and preferably), use a python 3 module (e.g. python/3.6.0 or python/3.7.4), where this problem does not exist at all.

Most errors containing "Permission Denied"

This error might arise if you are attempting to install a package without a virtual environment active.

Solution: make sure a virtual environment is active before proceeding with the installation; usually this is most easily verified by checking if your terminal prompt has been modified with the name of your environment in parentheses (see the example code blocks above). Alternatively, make sure a python module is loaded before you attempt the installation; it is possible to execute both pip and pip3 without having any python modules loaded; these will invoke system pip and pip3, which will attempt to install packages to system directories, to which you assuredly do not have permission to write files.

"My environment isn't working as expected"

This behavior could arise for any number of reasons. The most common reason that RC sees is that you've forgotten to load the correct python module prior to activating your virtual environment. Because virtual environments (as configured on O2) are necessarily dependent on their source installations, it is required to load the python module from which you created that virtual environment before activating it, every time you use it. If you have made sure you have already done so, and in the correct order (load module THEN activate virtual environment), then please submit a ticket with as many details as possible (including terminal output and command history such that we know where your environment lives and how to replicate your issue) to rchelp@hms.harvard.edu.