Versions Compared

Old Version 23

changes.mady.by.user Raffaele Potami

Saved on

compared with

New Version 24

changes.mady.by.user Alex Truong

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Command

Meaning

Code Block
module availspider python

shows the versions of Python installed on O2 (requires gcc/6.2.0 to be loaded)

Code Block
module load python/version

loads an individual module (substitute version for an actual version)

Code Block
module unload python/version

unloads an individual module version

Code Block
which python
# or
which python3

shows the current version of Python loaded (use python for a python 2.x module, python3 for a python 3.x module)

Code Block
pip freeze

shows what packages are installed for currently loaded Python module

Note that in order for You can use module avail python to give any output, you must have gcc/6.2.0 loaded into your environment. Alternately, you can use module spider python to circumvent this, but you will still need gcc/6.2.0 loaded in order to load any python modules that are installed.

Setting Up a Virtual Environment

Setup

...

as an alternative to module spider python, but avail will only work if you have some gcc module loaded (avail only searches exposed hierarchies, where spider searches the entire module list).

Setting Up a Virtual Environment

Setup

Code Block
mfk8@login01:~$ module load gcc/9.2.0
mfk8@login01:~$ module avail python
mfk8@login01:~$ module load gcc/6.2.0
mfk8@login01:~$ module avail python
mfk8@login01:~$ module load python/3.78.412       # or whichever version you'd like here
mfk8@login01:~$ which virtualenv
mfk8@login01:~$ virtualenv nameyourenvhere      # Please read on before executing this command as-is

...

The --system-site-packages flag will allow your virtual environment to inherit all existing installed packages in that Python distribution (which means you don't need to install your own numpy or scipy). On O2, we have purposely kept the list of installed python modules as light as possible so that the user is afforded the maximum amount of flexibility to install the versions they need. You can create as many virtual environments as you need.

Note

Note that for the python/3.7.4 module, the --system-site-packages flag will NOT work as expected; this is because the packages that have been installed for this module live in an external directory, and are exposed to the module via modification of the PYTHONPATH environment variable. If you wish to install a different version of a library from the one that is installed for 3.7.4, you will need to unset the PYTHONPATH variable first such that this external directory is no longer visible to the module (and to your virtual environment). The command to do so is simply:

Code Block

mfk5@login01:~$

unset

PYTHONPATH

Be aware that if you have other tools or configurations in your environment that leverage the use of PYTHONPATH, the above command will cause you to lose access to all modules that are exposed by this environment variable. If you encounter this situation, we recommend contacting us so that we can advise on alternate procedures.

To uninstall a virtual environment, simply rm -rf the folder containing the directory:

...

Note

Package mismatch errors with environments that use python/3.7.4

You may run into a problem where your environment detects a version of a package that lives in /n/app/python/3.7.4-ext and is unable to access/uninstall/upgrade/downgrade to a version that you desire. The quickest way to address this is to execute unset PYTHONPATH after loading the python/3.7.4 module but before activating your virtual environment. If you have personally modified the $PYTHONPATH variable for other purposes, you may need to re-populate it after removing the module package path from the variable (you can check if/how it is populated prior to executing unset by running echo $PYTHONPATH). If /n/app/python/3.7.4-ext/lib/site-packages is not the only path on $PYTHONPATH and unset-ing it is not worthwhile, you may choose to manually excise the path from the variable using whatever methods you find appropriate, though this is far less trivial than simply unset-ing it and repopulating it afterward, and we are unable to provide any general guidance to do so. If you require assistance with this sort of surgical modification of $PYTHONPATH, please contact us at rchelp@hms.harvard.edu and we can assist you with the processit afterward, and we are unable to provide any general guidance to do so. If you require assistance with this sort of surgical modification of $PYTHONPATH, please contact us at rchelp@hms.harvard.edu and we can assist you with the process.

The pip install --user flag

You may come across this flag in the installation instructions for a python package. This option is NOT RECOMMENDED on O2. Using this flag results in your python package installing to $HOME/.local . This location does not change if you switch between python versions, so if there is any possibility that you need to use multiple python versions for your work (e.g., for different projects), your $HOME/.local directory can quickly fill up with packages that were installed against mismatching python versions, which can cause incompatibility issues later on. This behavior is why RC recommends usage of virtual environments to keep package lists completely independent of each other.

Relatedly, if you are experiencing execution errors with mismatched package versions, the first step to troubleshooting is to delete the $HOME/.local directory. If the error persists after taking this step, please contact rchelp@hms.harvard.edu for further assistance.