NOTICE: FULL O2 Cluster Outage, January 3 - January 10th

O2 will be completely offline for a planned HMS IT data center relocation from Friday, Jan 3, 6:00 PM, through Friday, Jan 10

  • on Jan 3 (5:30-6:00 PM): O2 login access will be turned off.
  • on Jan 3 (6:00 PM): O2 systems will start being powered off.

This project will relocate existing services, consolidate servers, reduce power consumption, and decommission outdated hardware to improve efficiency, enhance resiliency, and lower costs.

Specifically:

  • The O2 Cluster will be completely offline, including O2 Portal.
  • All data on O2 will be inaccessible.
  • Any jobs still pending when the outage begins will need to be resubmitted after O2 is back online.
  • Websites on O2 will be completely offline, including all web content.

More details at: https://harvardmed.atlassian.net/l/cp/1BVpyGqm & https://it.hms.harvard.edu/news/upcoming-data-center-relocation

Jupyter on O2

Owned by Alex Truong
Last updated: Mar 15, 2024
6 min read

Our recommended approach is to use the O2 Portal - Jupyter App, which simplifies file transfer and has support for graphical applications. However, you may still find it useful to read through the Troubleshooting section below.

Due to implementation choices by the Jupyter developers, Jupyter and its associated dependencies are not installed by default on O2. However, we acknowledge that many users leverage Jupyter notebooks to great effect in their research, so we provide detailed instructions on how to set up a working Jupyter notebook here on O2.

Please note: We provide instructions for setting up Jupyter on O2, though we are only available for limited support of this use case. We are investigating the feasibility of offering a more robust solution in the future.

Installing Jupyter

As mentioned above, Jupyter is not installed into any of the Python installations available via the LMOD module system. However, it is very straightforward to install locally, via the use of a virtual environment. For detailed instructions on setting up a virtual environment, see Personal Python Packages, but the instructions will be reiterated here specifically for installing Jupyter.

First, create your virtual environment. We use version 3.9.14 to demonstrate in this example. Additionally, we install the virtual environment to our hypothetical home directory in this example, but you may create and use virtual environments wherever is convenient. You might also decide to use Conda to manage your virtual environment; refer to Conda on O2 for generic information about using Conda on O2.

HMSID@login01:~$ module load gcc/9.2.0 python/3.9.14 HMSID@login01:~$ virtualenv jupytervenv

Recall from Using Applications on O2 and exploration via module avail that the python/3.9.14 module will not be visible nor loadable until gcc/9.2.0 is loaded.

Now, source the environment, and install jupyter:

HMSID@login01:~$ source jupytervenv/bin/activate (jupytervenv)HMSID@login01:~$ pip3 install jupyter jupyterlab

At this point, a number of packages will attempt to install, but hopefully will not throw any errors.

Opening a Notebook

The above steps only need to be taken once (unless you need to recreate the virtual environment, or build another one).

The simplest way to open a notebook is to use the O2Portal - Jupyter App to set up your session.

Open a new LOCAL terminal. The following instructions assume you will be connecting from OS X or some other native *nix terminal (e.g. via Debian/CentOS, etc.). If on Windows, use some sort of terminal emulator such as Cygwin or MobaXterm (further configuration may be required, e.g. for X11 forwarding).

Make sure you have X11 forwarding active (e.g. XQuartz is running if on a Mac). You can find additional informations about X11 forwarding at Using X11 Applications Remotely

Pick a port on your local machine that is empty using whatever methods you like (e.g. netstat). Generally, somewhere in the 50000 range is safe if you just want to guess. SSH to O2 with that port (to be mentioned as PORT):

me@localhost:~$ ssh -Y -L PORT:127.0.0.1:PORT HMSID@o2.hms.harvard.edu

Now, request an interactive session (for illustrative purposes, we've landed at login01 as our login node):

If you might require multiple cores and more than 1GB of memory for your notebook, also specify that here, via -n and --mem=. See Using Slurm Basic or the sbatch man page for more sbatch flags. Here, --pty , and --x11 are mandatory. --tunnel is mandatory to complete the tunnel.

Let's pretend we landed on compute-a-16-20. Start the virtual environment and notebook (Jupyter and your virtual environment will require compiler libraries, so you'll need to load GCC as well):

Alternately, to open an existing notebook:

On newer versions of Jupyter (notebook >= 4.1), the developers implemented token authentication, which is on by default. If you have a freshly installed version of Jupyter, your notebook will have a token associated with it. When you run the above command, you should see a URL show up in the terminal that contains your session token. Click it, then it should open in your local machine's browser. You're all done!

Using other programming languages / Jupyter kernels

The instructions above will enable you to create a Juypter notebook that supports running Python. There are a number of other programming languages that can be used with Jupyter, which simply require installing the appropriate kernel to include support for a given programming language. (Python support is automatically included when the jupyter package is installed to your virtual environment, as the IPython kernel is the default kernel for Jupyter). 

If you want to use a Jupyter-supported programming language other than Python, you will need to manually install the appropriate kernel. 

Please note that we have only tested the use case of non-standard kernels with IRkernel, which allows you to run R notebooks using Jupyter. Using any other kernels for support of programming languages other than Python or R through Jupyter on O2 may be done without any implied support or guarantee of functionality.

R kernel for Jupyter

For example, if you want to run R through a Jupyter notebook on O2, you need to install the IRkernel package to a personal R library. This should be done after the Installing Jupyter instructions, and prior to the Opening a Notebook instructions. First, get into an interactive session:

While in an interactive session, set up the personal R library. Here we're using the R-3.4.1 module, so the R library reflects this version number in its name:

Next, install IRkernel and dependencies while your virtual environment is sourced:

Installing the IRkernel and dependent R packages to a personal R library will only need to be done once. At this point, you can proceed to the Opening a Notebook instructions. When your notebook is opened, you should now see an option for creating an R notebook under the "New" button.

In the future, you'll should ensure that your ~/.Renviron file and R_LIBS_USER environment variable point to the correct R personal library before trying to open an R notebook in Jupyter. 

Opening an Rshiny application

You may encounter an R package that deploys an Rshiny app that you can access; on your desktop, this is as easy as pasting the provided link into the browser. On O2, an extra step needs to be taken. The easiest way to access Rshiny will be via O2portal - Shiny on O2 .

When the URL is generated, it should specify a port number at the end. You need to make sure Rshiny knows about this new port. For example, say your RShiny URL looks something like 127.0.0.1:7875 on compute-a-16-168. You'll need to open a tunnel using this specified port when you request the interactive session, just as above:

You can now paste the RShiny app URL into your local browser and navigate the app accordingly.

Note that each time you generate the RShiny app URL, it is possible that the port will change each time (e.g. across sessions). This means you need to re-execute these instructions every time (don't forget to close the previous session). Don't forget to clean up after yourself once you're done (e.g. close open sessions) if you'd like to use the same port later on.

 

Troubleshooting Jupyter

This section will be expanded as cases are encountered and solved.

urllib3 openssl error

If you see an error along the lines of:

You will need to downgrade your version of urllib3 (hopefully you are dealing with a virtual environment you have write access to; if you are accessing a colleague’s, or lab’s communal environment, bring it up with the appropriate person). Specifically, you will need to downgrade to a version that is less than 2.0. As of this writing, the latest non-2.0 release is 1.26.16, so you can run something like

or even just

Alternatively, you can uninstall/reinstall the correct version:

Any iteration of these steps should allow you to resolve your urllib3 openssl error.

Digital Accessibility