The cl-submit-container is a new tool to submit requests for installing a Singularity container on the O2 cluster. An image source is used as an argument to generate an install request, which will build and deploy a Singularity container to an authorized container directory.

Table of Contents

Pilot Program Limitations

During this pilot phase, users may encounter bugs with and technical difficulties associated with the self-submission mechanism. HMS IT is developing and improving the process. The main goal of the pilot program is to identify and remedy bugs associated with production usage. We encourage users to email HMS Research Computing when encountering errors, and to be aware that these services are being offered on a preliminary basis.

Type of Image Source

An image source can be any of the following:

• A path to a pre-built Singularity/Apptainer image file

• A path to a Singularity/Apptainer definition file

• A path to a Dockerfile with '.dockerfile' as extension

• A Docker URI (e.g., docker://alpine:latest)

• A Singularity hub URI (e.g., shub://GodloveD/lolcow)

• A container Library URI (e.g., library://alpine:latest)

• A docker image exported to tarball (via docker save image_name > image_name.tar). Accepts gzipped tarballs (image_name.tar.gz or image_name.tgz) as well.

Usage on o2

To use the cl-submit-container, you must be on a compute node and run the following command with the --help flag to verify functionality:

/n/cluster/container-scanning/bin/cl-submit-container --help

General Form

Basic cl-submit-container tool usage takes the following form:

$ cl-submit-container [image_source]

Command / Flag Reference

  -l, --list-submissions                   List active submissions
  -s, --status                             Show status of all submissions
  -r, --last-log                           Show the log file of most recently submitted container.
  -g SHOW_LOG, --show-log SHOW_LOG         Show the log file for specified job.
  -f FOLLOW_LOG, --follow-log FOLLOW_LOG   Follow the log file for specified job one line at a time.
  --logs                                   List all log files.
  -i INFO, --info INFO                     Get submission info for given job.
  --info-all                               Dump all submission info for user.
  -a, --list-all-images                    List active submissions and installed images.
  -n DEST_NAME, --dest-name DEST_NAME      Name the destination image differently from the source image.
  -b BUILD_DIRECTORY, --build-directory BUILD_DIRECTORY   Specify alternate working directory for building an image. 
  -d INSTALL_SUBDIRECTORY, --install-subdirectory INSTALL_SUBDIRECTORY    Add subdirectories to installation path.
  -v, --verbose         Extra output.
  -y, --assume-yes      Assume yes to warning questions.
  --debug               Enable debug logging

Check the status of a submitted project

Specifying the status flag will retrieve all your submitted projects. It will show you the job name (image_source name), status, and path to the log file. For example:

$ /n/cluster/container-scanning/bin/cl-submit-container --status
Job Name         Status          Log file
nginx.dockerfile FAILED          /n/app/singularity/userlogs/<HMSID>/nginx.dockerfile-2024-07-25T09:51:19.log

If a container fails to install, you can run the command /n/cluster/container-scanning/bin/cl-submit-container -r to see a detailed log of the process.

Please make sure to include the content of this log when submitting a ticket to rchelp@hms.harvard.edu about the failed container install.

Running Approved Containers

Once a container is submitted and successfully created, it will be available under a user-specific directory, i.e. /n/app/singularity/containers/<HMSID>. The user has read and execute permissions on this folder, but cannot modify any of its contents. For more information on running Singularity containers in o2, see Running Singularity Containers in O2.

Common Examples

1. Create a Singularity image called nginx with the nginx container pulled from Docker Hub URI:

   $ /n/cluster/container-scanning/bin/cl-submit-container docker://nginx:latest

2. Create a Singularity container called genomics from a Dockerfile (note: -n genomics is used to name the image genomics instead of genomics_workflow)

   $ /n/cluster/container-scanning/bin/cl-submit-container -n genomics ~/CONTAINERS/working_dir/genomics_workflow.dockerfile 

3. Create a Singularity container from a Singularity definition file:

   $ /n/cluster/container-scanning/bin/cl-submit-container ~/ollama.def