Date: Fri, 29 Mar 2024 09:11:12 +0000 (UTC) Message-ID: <406503623.67.1711703472069@b9dc1cea980d> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_66_1009077701.1711703472069" ------=_Part_66_1009077701.1711703472069 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
While we have installed a small number of highly-used packages t= o each version of Python, we encourage users to maintain their own Python p= ackages. This ensures that the user has no discontinuity in their workflow = waiting for packages to be installed and that packages work with the user's= chosen, loaded Python version module. Python packages are built specifical= ly to a version of Python and may not work properly if a different version = of Python is loaded. For example, there is still a great disparity between = Python 2 and Python 3. O2's "module" feature allows users to quickly and ea= sily switch between Python versions.
Useful commands:
Command |
Meaning |
---|---|
module = spider python |
shows the versions of Python installed on O2 |
module = load python/version |
loads an individual module (substitute |
module = unload python/version |
unloads an individual module version |
which p= ython # or which python3 |
shows the current version of Python loaded (use |
pip fre= eze |
shows what packages are installed for currently loaded Python module = |
You can use module avail python
as an alternative to =
module spider python
, but avail
will only work if you h=
ave some gcc
module loaded (avail
only searches e=
xposed hierarchies, where spider
searches the entire module li=
st).
You=E2=80=99ll want to first get an interactive session, especially if y=
ou plan on installing scientific packages. compute-*-*
is a pl=
aceholder for whatever compute node you land on after requesting your inter=
active session.
mfk8@lo= gin01:~$ srun --pty -p interactive -t 0-1:00 --mem=3D1G bash # adjust srun = parameters accordingly as needed; you may need to request more memory if in= stalling large packages mfk8@compute-*-*:~$ module load gcc/9.2.0 mfk8@compute-*-*:~$ module avail python mfk8@compute-*-*:~$ module load python/3.8.12 # or whichever version = you'd like here mfk8@compute-*-*:~$ which virtualenv mfk8@compute-*-*:~$ virtualenv nameyourenvhere # Please read on before= executing this command as-is
These commands create a copy of our Python distribution in whichever dir=
ectory you ran the command, placing it in a folder named after whatever you=
specified. You can name your environment however you'd like. The above exa=
mples generate a directories located at ~/nameyourenvhere
or&n=
bsp;~/foobar
, respectively.
If you'd like to use the packages that are pre-installed in the Python m=
odule, include the --system-site-packages
flag when creat=
ing the virtual environment:
mfk8@co= mpute-*-*:~$ virtualenv nameyourenvhere --system-site-packages
The --system-site-packages
flag will allow your virtua=
l environment to inherit all existing installed packages in that Python dis=
tribution (which means you don't need to install your own numpy or scipy). =
On O2, we have purposely kept the list of installed python modules as light=
as possible so that the user is afforded the maximum amount of flexibility=
to install the versions they need. You can create as many virtual environm=
ents as you need.
Note that for the python/3.7.4
module, the --system-s=
ite-packages
flag will NOT work as expected; this i=
s because the packages that have been installed for this module live in an =
external directory, and are exposed to the module via modification of the <=
code>PYTHONPATH environment variable. If you wish to install a diffe=
rent version of a library from the one that is installed for 3.7.4, you wil=
l need to unset the PYTHONPATH
variable first such that this e=
xternal directory is no longer visible to the module (and to your virtual e=
nvironment). The command to do so is simply:
mfk5@login01:~$ unset PYTHONPATH
Be aware that if you have other tools or configurations in your environm=
ent that leverage the use of PYTHONPATH
, the above command wil=
l cause you to lose access to all modules that are exposed by this environm=
ent variable. If you encounter this situation, we recommend contacting us s=
o that we can advise on alternate procedures.
To uninstall a virtual environment, simply rm -rf
the folde=
r containing the directory:
mfk8@co= mpute-*-*:~$ rm -rf nameyourenvhere
To begin using the virtual environment, it needs to be activated. For a =
virtual environment located at ~/venv
:
mfk8@co= mpute-*-*:~$ source venv/bin/activate
Your prompt will now look (something like) this:
(venv)m= fk8@compute-*-*:~$
From here you an confirm that you're using the python associated with yo= ur virtual environment:
(venv)m= fk8@compute-*-*:~$ which python3 ~/venv/bin/python3
If it points instead to something like /n/app/python/<version&g=
t;/bin/python
or /usr/bin/python
, that means your envir=
onment is not active. Once you have confirmed that you are indeed inside yo=
ur virtual environment, you may do whatever you'd like, such as testing cod=
e or installing packages.
To deactivate the environment, simply type deactivate
, to s=
ee the corresponding change to your terminal prompt:
(venv)m= fk8@compute-*-*:~$ deactivate mfk8@login01:~$
NOTE:
The executable for Python 3.x is python3 and not python, the choice = was made by the Developers to mark a clear difference between version = 2.x and 3.x, as old python 2.x scripts might not be fully compatible a= cross the version jump.
A symbolic link python =E2=86=92 python3 is created w= ithin each python 3.x virtual environment, this link&nbs= p;allows to execute python3 scripts using python instead of p= ython3 when the virtual environment is active, however the link d= oes not exist outside of the python virtual environment.
pip
)PyPI (The Python Package Index) is a repository of software for the Pyth= on programming language, with packages available listed at http= s://pypi.python.org/pypi.
To install something, simply type:
(venv)m= fk8@compute-*-*:~$ pip3 install nameofpackage
Notice that your virtual environment needs to be active. The above comma= nd will search PyPI and install the package to your virtual environment. If= it is not active, you will get a permissions error because you will not ha= ve write access to the original installation.
You can also install packages manually; please refer to the instructions= provided by the corresponding package if this is required (e.g. a README/I= NSTALL file or a documentation page).
In general, you will access packages by importing them at the top of you= r Python scripts:
#!/usr/bi= n/env python3 import nameofpackage # actual code here
This will allow you to access that package (called a "module")'s functio= ns. Some modules will have extended functionality; please refer to the appr= opriate usage instructions provided by the package for more information on = how to access them.
Here is an example of loading the desired version of Python on O2, setti= ng up a virtual environment and activating it, installing a package to this= personal environment, running a Python script, and deactivating the virtua= l environment (2.x chosen here arbitrarily):
mfk8@lo= gin01:~$ srun --pty -p interactive -t 0-1:00 --mem=3D1G bash ... mfk8@compute-*-*:~$ which python /usr/bin/python mfk8@compute-*-*:~$ module load gcc/6.2.0 mfk8@compute-*-*:~$ module avail python --------------- /n/app/lmod/lmod/modulefiles/Compiler/gcc/6.2.0 -----------= ----- python/2.7.12 (E) python/3.6.0 (E,D) python/3.7.4 Where: E: Experimental D: Default Module Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules match= ing any of the "keys". mfk8@compute-*-*:~$ module load python/3.7.4 mfk8@compute-*-*:~$ which python3 /n/app/python/3.7.4/bin/python3 mfk8@compute-*-*:~$ which virtualenv /n/app/python/2.7.12/bin/virtualenv mfk8@compute-*-*:~$ mkdir mypythonfolder && cd mypythonfolder mfk8@compute-*-*:~/mypythonfolder$ virtualenv myvirtualenv (truncated) mfk8@compute-*-*:~/mypythonfolder$ source myvirtualenv/bin/activate (myvirtualenv)mfk8@lcompute-*-*:~/mypythonfolder$ pip3 install numpy (truncated) (myvirtualenv)mfk8@compute-*-*:~/mypythonfolder$ echo "print 'hello world'"= > myscript.py (myvirtualenv)mfk8@compute-*-*:~/mypythonfolder$ python3 myscript.py hello world (myvirtualenv)mfk8@compute-*-*:~/mypythonfolder$ deactivate mfk8@compute-*-*:~/mypythonfolder$
python3: error while loading shared libr=
aries: libpython3.7m.so.1.0: cannot open shared object file: No such file o=
r directory
This error might arise if you have not loaded the corresponding python m= odule prior to invoking the python executable in your virtual environment. = Due to the nature of virtualenv, it is necessary (on O2) for the parent ins= tallation to be present in your session when your virtual environment is ac= tivated. This must be done in the correct order as well.
Solution: As an example, the error shown above points to a problem with = (presumably) an environment based on the python/3.7.4 module, so the proper= procedure would be to
1) deactivate the current virtual environment,
2) make sure the python/3.7.4 module is loaded,
3) re-activate the virtual environment
If it still fails, then it is unlikely that your virtual environment was= created from one of the existing modules, and you may need to re-create it= from scratch (or debug with whatever source the environment was from). Not= e that copying python environments, virtual or otherwise, from a local mach= ine to the cluster will likely fail to work as intended. If an existing pyt= hon module does not fit your needs, you will need to supply your own versio= n of python, either via (ana)conda or installing it yourself. For further/d= irected guidance on this option, feel free to send a ticket to rche= lp@hms.harvard.edu.
setuptools.build_meta
This error might arise if you are installing software via pip3 using the
python/3.6.0
module within a virtualenv which was created using
--system-site-packages
. This is d=
ue to a bug in pip that does not properly resolve the setuptools dependency=
for older versions (such as the one used in the aforementioned python modu=
le). This problem does not exist in the python/3.7.4 module. RC has no plan=
s to fix this in the existing 3.6.0 module due to the possibility that it c=
ould break existing user virtual environments cluster-wide.
Solution: create another python/3.6.0
virt=
ual environment without --system-site-packages
; it will downlo=
ad and install a newer version of setuptools that is NOT dependent on the o=
lder version that is in the global install. Alternatively, use the py=
thon/3.7.4
module (for which --system-site-packages
doe=
s not work as users may expect anyway - see above).
This error might arise if you are using the python/2.7.12
m=
odule instead of the python/2.7.12-ucs4
module, or vice versa.=
These modules are mutually exclusive, and are generally not compatible wit=
h each other. This error is due to a Unicode character incompatibility with=
the package you are trying to access or install.
Solution: use whichever python/2.7.12
modu=
le you are not currently using (e.g. if currently using 2.7.12, use 2.7.12-=
ucs4, or vice versa). Alternatively (and preferably), use a python 3 module=
(e.g. python/3.6.0
or python/3.7.4
), where this =
problem does not exist at all.
This error might arise if you are attempting to install a package withou= t a virtual environment active.
Solution: make sure a virtual environment is active bef=
ore proceeding with the installation; usually this is most easily verified =
by checking if your terminal prompt has been modified with the name of your=
environment in parentheses (see the example code blocks above). Alternativ=
ely, make sure a python module is loaded before you attempt the installatio=
n; it is possible to execute both pip
and pip3
wi=
thout having any python modules loaded; these will invoke system pip<=
/code> and
pip3
, which will attempt to install packages to sys=
tem directories, to which you assuredly do not have permission to write fil=
es.
This behavior could arise for any number of reasons. The most common rea= son that RC sees is that you've forgotten to load the correct python module= prior to activating your virtual environment. Because virtual environments= (as configured on O2) are necessarily dependent on their source installati= ons, it is required to load the python module from which you created that v= irtual environment before activating it, every time you use it. If you have= made sure you have already done so, and in the correct order (load module = THEN activate virtual environment), then please submit a ticket with as man= y details as possible (including terminal output and command history such t= hat we know where your environment lives and how to replicate your issue) t= o rchelp@hms.harvard.edu.
py=
thon/3.7.4
You may run into a problem where your environment detects a version of a=
package that lives in /n/app/python/3.7.4-ext
and is unable t=
o access/uninstall/upgrade/downgrade to a version that you desire. The quic=
kest way to address this is to execute unset PYTHONPATH
after =
loading the python/3.7.4
module but before activating your vir=
tual environment. If you have personally modified the $PYTHONPATH variable for other purposes, you may need to re-populate it after remov=
ing the module package path from the variable (you can check if/how it is p=
opulated prior to executing
unset
by running echo $PYTHO=
NPATH
). If /n/app/python/3.7.4-ext/lib/site-packages
is=
not the only path on $PYTHONPATH
and unset
-ing i=
t is not worthwhile, you may choose to manually excise the path from the va=
riable using whatever methods you find appropriate, though this is far less=
trivial than simply unset
-ing it and repopulating it afterwar=
d, and we are unable to provide any general guidance to do so. If you requi=
re assistance with this sort of surgical modification of $PYTHONPATH<=
/code>, please contact us at rchelp@hms.harvard.edu and we can =
assist you with the process.
pip ins=
tall --user
flagYou may come across this flag in the installation instructions for a pyt=
hon package. This option is NOT RECOMMENDED on O2. Using t=
his flag results in your python package installing to $HOME/.local . This location does not change if you switch between python versions,=
so if there is any possibility that you need to use multiple python versio=
ns for your work (e.g., for different projects), your
$HOME/.local directory can quickly fill up with packages that were installed agains=
t mismatching python versions, which can cause incompatibility issues later=
on. This behavior is why RC recommends usage of virtual environments to ke=
ep package lists completely independent of each other.
Relatedly, if you are experiencing execution errors with mismatched pack=
age versions, the first step to troubleshooting is to delete the $HOM=
E/.local
directory. If the error persists after taking this step, pl=
ease contact rchelp@hms.harvard.edu for further assistance.
openssl
You may run into an error involving openssl
>=3D 1.1.1 v=
ersions. The only known workaround will be to downgrade the package in ques=
tion to a version that is using < 1.1.1. For an example of this, see https://harvardmed.atlassian.net/wiki/spaces/O2/pages/1594262530/J=
upyter+on+O2#urllib3-openssl-error .