NOTICE: FULL O2 Cluster Outage, January 3 - January 10th

O2 will be completely offline for a planned HMS IT data center relocation from Friday, Jan 3, 6:00 PM, through Friday, Jan 10

  • on Jan 3 (5:30-6:00 PM): O2 login access will be turned off.
  • on Jan 3 (6:00 PM): O2 systems will start being powered off.

This project will relocate existing services, consolidate servers, reduce power consumption, and decommission outdated hardware to improve efficiency, enhance resiliency, and lower costs.

Specifically:

  • The O2 Cluster will be completely offline, including O2 Portal.
  • All data on O2 will be inaccessible.
  • Any jobs still pending when the outage begins will need to be resubmitted after O2 is back online.
  • Websites on O2 will be completely offline, including all web content.

More details at: https://harvardmed.atlassian.net/l/cp/1BVpyGqm & https://it.hms.harvard.edu/news/upcoming-data-center-relocation

Self-Install Singularity Containers

Owned by Gaby Rodriguez
Last updated: Aug 30, 2024 by Kathleen Chappell
3 min read

This new tool is in a beta state. Please share your feedback with us by emailing rchelp@hms.harvard.edu.

The cl-submit-container is a new tool to submit requests for installing a Singularity container on the O2 cluster. An image source is used as an argument to generate an install request, which will build and deploy a Singularity container to an authorized container directory.

Table of Contents

Pilot Program Limitations

Access is limited to those who are part of the pilot program. If you would like to join this group of selected researchers, please email rchelp@hms.harvard.edu.

During this pilot phase, users may encounter bugs with and technical difficulties associated with the self-submission mechanism. HMS IT is developing and improving the process. The main goal of the pilot program is to identify and remedy bugs associated with production usage. We encourage users to email HMS Research Computing when encountering errors, and to be aware that these services are being offered on a preliminary basis.

Type of Image Source

An image source can be any of the following:

  • A path to a pre-built Singularity/Apptainer image file

  • A path to a Singularity/Apptainer definition file

  • A path to a Dockerfile with '.dockerfile' as extension

  • A Docker URI (e.g., docker://alpine:latest)

  • A Singularity hub URI (e.g., shub://GodloveD/lolcow)

  • A container Library URI (e.g., library://alpine:latest)

  • A docker image exported to tarball (via docker save image_name > image_name.tar). Accepts gzipped tarballs (image_name.tar.gz or image_name.tgz) as well.

Usage on o2

To use the cl-submit-container, you must be on a compute node and run the following command with the --help flag to verify functionality:

/n/cluster/bin/cl-submit-container --help


General Form

Basic cl-submit-container tool usage takes the following form:

$ /n/cluster/bin/cl-submit-container [image_source]


Command / Flag Reference

-l, --list-submissions List active submissions -s, --status Show status of all submissions -r, --last-log Show the log file of most recently submitted container. -g SHOW_LOG, --show-log SHOW_LOG Show the log file for specified job. -f FOLLOW_LOG, --follow-log FOLLOW_LOG Follow the log file for specified job one line at a time. --logs List all log files. -i INFO, --info INFO Get submission info for given job. --info-all Dump all submission info for user. -a, --list-all-images List active submissions and installed images. -n DEST_NAME, --dest-name DEST_NAME Name the destination image differently from the source image. -b BUILD_DIRECTORY, --build-directory BUILD_DIRECTORY Specify alternate working directory for building an image. -d INSTALL_SUBDIRECTORY, --install-subdirectory INSTALL_SUBDIRECTORY Add subdirectories to installation path. -v, --verbose Extra output. -y, --assume-yes Assume yes to warning questions. --debug Enable debug logging

 

Check the status of a submitted project

Specifying the status flag will retrieve all your submitted projects. It will show you the job name (image_source name), status, and path to the log file. For example:

If a container fails to install, you can run the command /n/cluster/bin/cl-submit-container -r to see a detailed log of the process.

Please make sure to include the content of this log when submitting a ticket to rchelp@hms.harvard.edu about the failed container install.

Running Approved Containers

Once a container is submitted and successfully created, it will be available under a user-specific directory, i.e. /n/app/singularity/containers/<HMSID>. The user has read and execute permissions on this folder, but cannot modify any of its contents. For more information on running Singularity containers in o2, see Running Singularity Containers in O2.

Common Examples

  1. Create a Singularity image called nginx with the nginx container pulled from Docker Hub URI:

  2. Create a Singularity container called genomics from a Dockerfile (note: -n genomics is used to name the image genomics instead of genomics_workflow)

  3. Create a Singularity container from a Singularity definition file:

Digital Accessibility