Personal Python Packages

While we have installed a small number of highly-used packages to each version of Python, we encourage users to maintain their own Python packages. This ensures that the user has no discontinuity in their workflow waiting for packages to be installed and that packages work with the user's chosen, loaded Python version module. Python packages are built specifically to a version of Python and may not work properly if a different version of Python is loaded. For example, there is still a great disparity between Python 2 and Python 3. O2's "module" feature allows users to quickly and easily switch between Python versions.

Useful commands:

You can use module avail python as an alternative to module spider python, but avail will only work if you have some gcc module loaded (avail only searches exposed hierarchies, where spider searches the entire module list).

Setting Up a Virtual Environment

You’ll want to first get an interactive session, especially if you plan on installing scientific packages. compute-*-* is a placeholder for whatever compute node you land on after requesting your interactive session.

Setup

These commands create a copy of our Python distribution in whichever directory you ran the command, placing it in a folder named after whatever you specified. You can name your environment however you'd like. The above examples generate a directories located at ~/nameyourenvhere or ~/foobar, respectively.

If you'd like to use the packages that are pre-installed in the Python module, include the --system-site-packages flag when creating the virtual environment:

The --system-site-packages flag will allow your virtual environment to inherit all existing installed packages in that Python distribution (which means you don't need to install your own numpy or scipy). On O2, we have purposely kept the list of installed python modules as light as possible so that the user is afforded the maximum amount of flexibility to install the versions they need. You can create as many virtual environments as you need.

To uninstall a virtual environment, simply rm -rf the folder containing the directory:

Basic Usage

To begin using the virtual environment, it needs to be activated. For a virtual environment located at ~/venv:

Your prompt will now look (something like) this:

From here you an confirm that you're using the python associated with your virtual environment:

If it points instead to something like /n/app/python/<version>/bin/python or /usr/bin/python, that means your environment is not active. Once you have confirmed that you are indeed inside your virtual environment, you may do whatever you'd like, such as testing code or installing packages.

To deactivate the environment, simply type deactivate, to see the corresponding change to your terminal prompt:

NOTE:

The executable for Python 3.x is python3 and not python, the choice was made by the Developers to mark a clear difference between version 2.x and 3.x, as old python 2.x scripts might not be fully compatible across the version jump.

A symbolic link python → python3 is created within each python 3.x virtual environment, this link allows to execute python3 scripts using python instead of python3 when the virtual environment is active, however the link does not exist outside of the python virtual environment.

Installing Packages (via pip)

PyPI (The Python Package Index) is a repository of software for the Python programming language, with packages available listed at https://pypi.python.org/pypi.

To install something, simply type:

Notice that your virtual environment needs to be active. The above command will search PyPI and install the package to your virtual environment. If it is not active, you will get a permissions error because you will not have write access to the original installation.

You can also install packages manually; please refer to the instructions provided by the corresponding package if this is required (e.g. a README/INSTALL file or a documentation page).

Using Python Packages

In general, you will access packages by importing them at the top of your Python scripts:

This will allow you to access that package (called a "module")'s functions. Some modules will have extended functionality; please refer to the appropriate usage instructions provided by the package for more information on how to access them.

A Full Example

Here is an example of loading the desired version of Python on O2, setting up a virtual environment and activating it, installing a package to this personal environment, running a Python script, and deactivating the virtual environment (2.x chosen here arbitrarily):

Troubleshooting

Errors resembling python3: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory

This error might arise if you have not loaded the corresponding python module prior to invoking the python executable in your virtual environment. Due to the nature of virtualenv, it is necessary (on O2) for the parent installation to be present in your session when your virtual environment is activated. This must be done in the correct order as well.

Solution: As an example, the error shown above points to a problem with (presumably) an environment based on the python/3.7.4 module, so the proper procedure would be to

1) deactivate the current virtual environment,

2) make sure the python/3.7.4 module is loaded,

3) re-activate the virtual environment

If it still fails, then it is unlikely that your virtual environment was created from one of the existing modules, and you may need to re-create it from scratch (or debug with whatever source the environment was from). Note that copying python environments, virtual or otherwise, from a local machine to the cluster will likely fail to work as intended. If an existing python module does not fit your needs, you will need to supply your own version of python, either via (ana)conda or installing it yourself. For further/directed guidance on this option, feel free to send a ticket to rchelp@hms.harvard.edu.

Any errors related to setuptools.build_meta

This error might arise if you are installing software via pip3 using the python/3.6.0 module within a virtualenv which was created using --system-site-packages. This is due to a bug in pip that does not properly resolve the setuptools dependency for older versions (such as the one used in the aforementioned python module). This problem does not exist in the python/3.7.4 module. RC has no plans to fix this in the existing 3.6.0 module due to the possibility that it could break existing user virtual environments cluster-wide.

Solution: create another python/3.6.0 virtual environment without --system-site-packages; it will download and install a newer version of setuptools that is NOT dependent on the older version that is in the global install. Alternatively, use the python/3.7.4 module (for which --system-site-packages does not work as users may expect anyway - see above).

Most errors containing the word "Unicode"

This error might arise if you are using the python/2.7.12 module instead of the python/2.7.12-ucs4 module, or vice versa. These modules are mutually exclusive, and are generally not compatible with each other. This error is due to a Unicode character incompatibility with the package you are trying to access or install.

Solution: use whichever python/2.7.12 module you are not currently using (e.g. if currently using 2.7.12, use 2.7.12-ucs4, or vice versa). Alternatively (and preferably), use a python 3 module (e.g. python/3.6.0 or python/3.7.4), where this problem does not exist at all.

Most errors containing "Permission Denied"

This error might arise if you are attempting to install a package without a virtual environment active.

Solution: make sure a virtual environment is active before proceeding with the installation; usually this is most easily verified by checking if your terminal prompt has been modified with the name of your environment in parentheses (see the example code blocks above). Alternatively, make sure a python module is loaded before you attempt the installation; it is possible to execute both pip and pip3 without having any python modules loaded; these will invoke system pip and pip3, which will attempt to install packages to system directories, to which you assuredly do not have permission to write files.

"My environment isn't working as expected"

This behavior could arise for any number of reasons. The most common reason that RC sees is that you've forgotten to load the correct python module prior to activating your virtual environment. Because virtual environments (as configured on O2) are necessarily dependent on their source installations, it is required to load the python module from which you created that virtual environment before activating it, every time you use it. If you have made sure you have already done so, and in the correct order (load module THEN activate virtual environment), then please submit a ticket with as many details as possible (including terminal output and command history such that we know where your environment lives and how to replicate your issue) to rchelp@hms.harvard.edu.

The pip install --user flag

You may come across this flag in the installation instructions for a python package. This option is NOT RECOMMENDED on O2. Using this flag results in your python package installing to $HOME/.local . This location does not change if you switch between python versions, so if there is any possibility that you need to use multiple python versions for your work (e.g., for different projects), your $HOME/.local directory can quickly fill up with packages that were installed against mismatching python versions, which can cause incompatibility issues later on. This behavior is why RC recommends usage of virtual environments to keep package lists completely independent of each other.

Relatedly, if you are experiencing execution errors with mismatched package versions, the first step to troubleshooting is to delete the $HOME/.local directory. If the error persists after taking this step, please contact rchelp@hms.harvard.edu for further assistance.

Errors related to openssl

You may run into an error involving openssl >= 1.1.1 versions. The only known workaround will be to downgrade the package in question to a version that is using < 1.1.1. For an example of this, see Jupyter on O2 | urllib3 openssl error .