Versions Compared

Old Version 28

changes.mady.by.user Raffaele Potami

Saved on

compared with

New Version Current

changes.mady.by.user Alex Truong

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Panel
panelIconIdatlassian-info
panelIcon:info:
bgColor#DEEBFF

Our recommended approach is to use the O2 Portal - Jupyter App, which simplifies file transfer and has support for graphical applications. However, you may still find it useful to read through the Troubleshooting section below.

Table of Contents

Due to implementation choices by the Jupyter developers, Jupyter and its associated dependencies are not installed by default on O2. However, we acknowledge that many users leverage Jupyter notebooks to great effect in their research, so we provide detailed instructions on how to set up a working Jupyter notebook here on O2.

...

First, create your virtual environment. We use version 3.79.4 14 to demonstrate in this example. Additionally, we install the virtual environment to our hypothetical home directory in this example, but you may create and use virtual environments wherever is convenient. You might also decide to use Conda to manage your virtual environment; refer to Conda on O2 for generic information about using Conda on O2.

Code Block
ECOMMONS@login01HMSID@login01:~$ module load gcc/69.2.0 python/3.79.414
ECOMMONS@login01HMSID@login01:~$ virtualenv jupytervenv

Recall from Using Applications on O2 and exploration via module avail that the python/3.79.414 module will not be visible nor loadable until gcc/69.2.0 is loaded.

Now, source the environment, and install jupyter:

Code Block
ECOMMONS@login01HMSID@login01:~$ source jupytervenv/bin/activate
(jupytervenv)ECOMMONS@login01HMSID@login01:~$ pip3 install jupyter jupyterlab

At this point, a number of packages will attempt to install, but hopefully will not throw any errors.

...

The above steps only need to be taken once (unless you need to recreate the virtual environment, or build another one). After the installation finishes, open

The simplest way to open a notebook is to use the O2Portal - Jupyter App to set up your session.

Expand
titleIf your installation is non-standard, you can try using the manual approach to configure a notebook session (expand this block to view these instructions).

Open a new LOCAL terminal. The following instructions assume you will be connecting from OS X or some other native *nix terminal (e.g. via Debian/CentOS, etc.). If on Windows, use some sort of terminal emulator such as Cygwin or MobaXterm (further configuration may be required, e.g. for X11 forwarding).

Make sure you have X11 forwarding active (e.g. XQuartz is running if on a Mac). You can find additional informations about X11 forwarding at Using X11 Applications Remotely

Pick a port on your local machine that is empty using whatever methods you like (e.g. netstat). Generally, somewhere in the 50000 range is safe if you just want to guess. SSH to O2 with that port (to be mentioned as PORT):

Code Block
me@localhost:~$ ssh -Y -L PORT:127.0.0.1:PORT
ECOMMONS@o2
HMSID@o2.hms.harvard.edu

Now, request an interactive session (for illustrative purposes, we've landed at login01 as our login node):

Code Block
ECOMMONS@login01
HMSID@login01:~$ srun -t 0-3:00 --pty -p interactive --x11 --tunnel PORT:PORT /bin/bash

If you might require multiple cores and more than 1GB of memory for your notebook, also specify that here, via -n and --mem=. See Using Slurm Basic or the sbatch man page for more sbatch flags. Here, --pty , and --x11 are mandatory. --tunnel is mandatory to complete the tunnel.

Let's pretend we landed on compute-a-16-20. Start the virtual environment and notebook (Jupyter and your virtual environment will require compiler libraries, so you'll need to load GCC as well):

Code Block
ECOMMONS@compute
HMSID@compute-a-16-20:~$ module load gcc/
6
9.2.0 python/3.
7
9.
4
14
ECOMMONS@compute
HMSID@compute-a-16-20:~$ source jupytervenv/bin/activate
(jupytervenv)
ECOMMONS@compute
HMSID@compute-a-16-20:~$ jupyter notebook --port=PORT --browser='none'

Alternately, to open an existing notebook:

Code Block
(jupytervenv)
ECOMMONS@compute
HMSID@compute-a-16-20:~$ jupyter notebook NOTEBOOKFILE --port=PORT --browser='none'

On newer versions of Jupyter (notebook >= 4.1), the developers implemented token authentication, which is on by default. If you have a freshly installed version of Jupyter, your notebook will have a token associated with it. When you run the above command, you should see a URL show up in the terminal that contains your session token. Click it, then it should open in your local machine's browser. You're all done!

Using other programming languages / Jupyter kernels

...

For example, if you want to run R through a Jupyter notebook on O2, you need to install the IRkernel package to a personal R library. This should be done after the Installing Jupyter instructions, and prior to the Opening a Notebook instructions. First, get into an interactive session:

Code Block
ECOMMONS@login01HMSID@login01:~$ srun --pty -p interactive -t 0-2 bash

While in an interactive session, set up the personal R library. Here we're using the R-3.4.1 module, so the R library reflects this version number in its name:

Code Block
ECOMMONS@computeHMSID@compute-a-16-68:~$ mkdir -p ~/R-3.4.1-IRkernel/library
ECOMMONS@computeHMSID@compute-a-16-68:~$ echo 'R_LIBS_USER="~/R-3.4.1-IRkernel/library"' > $HOME/.Renviron
ECOMMONS@computeHMSID@compute-a-16-68:~$ export R_LIBS_USER="~/R-3.4.1-IRkernel/library"

Next, install IRkernel and dependencies while your virtual environment is sourced:

Code Block
ECOMMONS@computeHMSID@compute-a-16-68:~$ module load gcc/69.2.0 python/3.79.414 R/3.4.1
ECOMMONS@computeHMSID@compute-a-16-68:~$ source jupytervenv/bin/activate
(jupytervenv) ECOMMONS@computeHMSID@compute-a-16-68:~$ R
> install.packages(c('repr', 'IRdisplay', 'evaluate', 'crayon', 'pbdZMQ', 'devtools', 'uuid', 'digest'))
# select mirror
> devtools::install_github('IRkernel/IRkernel')
> IRkernel::installspec()

...

Opening an Rshiny application with Jupyter

You may encounter an R package that deploys an Rshiny app that you can access; on your desktop, this is as easy as pasting the provided link into the browser. On O2, an extra step needs to be taken. The easiest way to access Rshiny will be via O2portal - Shiny on O2 .

Expand
titleIf O2Portal does not meet your needs, you can try manually configuring your application for that session (expand this block to view these instructions).

When the URL is generated, it should specify a port number at the end. You need to make sure Rshiny knows about this new port. For example, say your RShiny URL looks something like 127.0.0.1:7875 on compute-a-16-168. You'll need to open a tunnel using this specified port when you request the interactive session, just as above:

Code Block
me@localhost:~$ ssh -Y -L 7875:127.0.0.1:7875
ECOMMONS@o2
HMSID@o2.hms.harvard.edu
...
ECOMMONS@login01
HMSID@login01:~$ srun -t 0-1:00 --pty -p interactive --mem=10G --x11 --tunnel 7875:7875 /bin/bash
...
ECOMMONS@compute
HMSID@compute-a-16-168:~$ R
> library(shiny)
> options(shiny.port=7875)
> Sys.setlocale('LC_ALL','C') #to solve warnings related to "...invalid in this locale"

You can now paste the RShiny app URL into your local browser and navigate the app accordingly.

Note that each time you generate the RShiny app URL, it is possible that the port will change each time (e.g. across sessions). This means you need to re-execute these instructions every time (don't forget to close the previous session). Don't forget to clean up after yourself once you're done (e.g. close open sessions) if you'd like to use the same port later on.

Troubleshooting Jupyter

This section will be expanded as cases are encountered and solved.

urllib3 openssl error

If you see an error along the lines of:

Code Block
    raise ImportError(
ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'OpenSSL 1.0.2k-fips  26 Jan 2017'. See: https://github.com/urllib3/urllib3/issues/2168

You will need to downgrade your version of urllib3 (hopefully you are dealing with a virtual environment you have write access to; if you are accessing a colleague’s, or lab’s communal environment, bring it up with the appropriate person). Specifically, you will need to downgrade to a version that is less than 2.0. As of this writing, the latest non-2.0 release is 1.26.16, so you can run something like

Code Block
pip3 install --force-reinstall urllib3==1.26.16

or even just

Code Block
pip3 install --force-reinstall urllib3<2.0

Alternatively, you can uninstall/reinstall the correct version:

Code Block
pip3 uninstall urllib3
pip3 install urllib3<2.0

Any iteration of these steps should allow you to resolve your urllib3 openssl error.