Csubmitter User Documentation

Csubmitter is an o2 tool for submitting Docker or Singularity containers to our scanning pipeline for deployment on the o2 cluster.

Submitted containers that earn a "Passing" grade are automatically deployed to o2 and can be run within the o2 singularity environment



Table of Contents

Pilot Program Limitations and Known Bugs

During the pilot phase, users may encounter bugs with and technical difficulties associated with the scanning system. This system is being actively developed by the DevOps team, and one of the goals of the pilot program is to identify and remedy bugs associated with production usage. We encourage users to submit tickets in STAT when encountering errors, and to be aware that these services are being offered on a preliminary basis.



The section below will document all Known Bugs so users can be aware of them when proceeding.

Known Bugs

  • When attempting to create a new project, the following error is encountered:

    1 2 [ac636@compute-a-16-120 ~]$ csubmitter --image-uri docker://nginx --name newtest 400 Client Error: Bad Request for url: http://clair.rc.hms.harvard.edu:8000/api/submission/

    This bug is due to a race condition in the database that sometimes causes new projects that share a name with another users' project to fail with the above Error.

    • E.g. In the example above, another user has already created a project with name newtest, so the submission fails. 

  • Workaround Solution: Resubmit with a different project name

Usage on o2

Loading the Module

csubmitter can be loaded in an o2 session with the lmod module loading system. It has no other dependencies. To load it, simply run:

1 module load csubmitter


Using the Module

NOTE: Cannot be run from Login or Transfer nodes

As of this writing (10/28/20), the csubmitter tool should only be run from compute nodes. Attempting to run from login or transfer nodes will yield an Error that will prompt you to run the tool within an interactive session

Please run the tool from an interactive session within o2 only. Running the tool from login or transfer nodes is not supported.



Once the module is loaded, you can invoke csubmitter with the --help flag to verify functionality

1 csubmitter --help


General Form

Basic csubmitter tool usage takes the following form:

1 $ csubmitter [STATUS/IMAGE-PATH/IMAGE-URI] [NAME]


Command / Flag Reference

1 2 3 4 5 --status [ID] : displays the status of all your submitted containers (based on the user you are logged in as). Can be followed by a project id to display the status of that container project --image-path [PATH] : specifies the fully qualified path to the container image for scanning. Must be a valid dockerfile, .def file, or .sif file --image-uri [URI] : specifies the URI resource to pull the container image from. Supports dockerhub and singularityhub. (e.g. docker:// , shub:// ) --name [NAME] : specifies the project name for version control + status tracking



Note: You cannot specify an Image URI AND an Image Path



Check the status of a submitted project

Specifying the status flag will retrieve all your submitted projects in a table.

It will show you the name of the container project (Container Name), the container source, and the latest scanning information for that project

1 2 3 4 5 6 7 $ csubmitter --status +----+----------------+----------------+-----------+---------------------+------------+--------------+ | id | Container Name | Source | Status | Submitted date | Scan Grade | Type | +----+----------------+----------------+-----------+---------------------+------------+--------------+ | 1 | nginx | docker://nginx | processed | 2020-10-08T10:19:12 | Pass | ContainerUri | +----+----------------+----------------+-----------+---------------------+------------+--------------+



Each submission in the table has a numerical id in the first column. Using the status flag with this table index as an argument will retrieve a detailed report for that specific project, e.g.

1 2 3 4 5 6 7 8 9 10 11 12 $ csubmitter --status 3 { "Remote Connections":[], "Detected Malware":[], "Open Ports":[], "Scanning Grade":"Pass", "Vulnerability Count":0, "Highest Vulnerability":null, "High CVEs":{}, "Container Name":"nginx", "Grade":"Pass" }



Running Approved Containers

Each user that submits a container with csubmitter is allocated a user-specific directory in /n/app/singularity/containers. The user has read and execute permissions on this folder, but cannot modify any of its contents.

If a container Passes the scanning pipeline, it's placed into their user-specific directory.

Containers are named according to the project name (specified with the --name flag), and are always in .sif format


For example, if user ac636 submits a container with the project name nginx that then passes, it's placed into /n/app/singularity/containers/ac636/nginx.sif

1 2 3 4 5 6 $ csubmitter --image-uri docker://nginx --name nginx Message: ac636 submitted a Uri: nginx with name: nginx for scanning $ ls /n/app/singularity/containers/ac636 nginx.sif



For more information on running Singularity containers in o2, see: Running Singularity Containers in O2


Examples:

  1. Start a new project called nginx with the nginx container pulled from Docker Hub URI:

    1 $ csubmitter --image-uri docker://nginx:latest --name nginx



  2. Start a new project called genomics-workflow using a local definition/Dockerfile

    1 $ csubmitter --image-path /home/$USER/genomics_workflow_foo_lab.def --name g_flow_foo_lab