The O2 cluster includes a set of lab-contributed GPU nodes. These nodes were purchased by individual labs, but are made available to the rest of the O2 community within the re-queue GPU partition called gpu_requeue.
This partition currently comprises 28 Nvidia RTX6000 single precision cards and 2 Nvidia M40 Tesla cards. It is not suitable for GPU double precision jobs.
To see the currently available resources under the gpu_requeue partition you can use the command below:
sinfo --Format=nodehost,cpusstate,memory,statelong,gres -p gpu_requeue
HOSTNAMES CPUS(A/I/O/T) MEMORY STATE GRES
compute-g-16-197 0/20/0/20 257548 idle gpu:teslaM40:2
compute-gc-17-245 0/48/0/48 385218 idle gpu:rtx6000:10
compute-gc-17-246 0/48/0/48 385218 idle gpu:rtx6000:10
How Preemption Works
The labs that purchased these nodes have preemption priority on their own hardware. If the nodes are full and a researcher from one of those labs submits a job, one or more GPU jobs running on the gpu_requeue partition might be killed and re-queued in order to free resources for the Lab's job. That is, the gpu_requeue job will be cancelled, as if you ran the scancel command, and re-submitted (as long as you initially submitted with the flag --requeue).
Please note this may happen at any time without any warning and multiple times.
Preempted jobs will show on Slurm sacct database as PR or PREEMPTED until they are requeued. The scheduler will not replace the jobid of a preempted job but instead it will resubmit it with the same jobid. In order to see the history of a preempted and requeued job you need to add the flag -D to the sacct query.
Jobs that fail for other reasons - like running longer than the reserved time, an error in the code or data, or a node failure - will not automatically be requeued.
How to Submit to the gpu_requeue Partition
To submit jobs on gpu_requeue you need to specify that partition with the flag "-p", and add the flag --requeue. Without the requeue flags jobs will still get killed but will not be automatically requeued.
#SBATCH -p gpu_requeue
#SBATCH -t 6:00:00
<your job goes here>
The gpu_requeue partition currently has a maximum Wall-Time of 24 hours and a fairshare consumption equivalent to 10% of the regular gpu partition.
How to Efficiently Use the gpu_requeue Partition
In order to work properly, any job submitted to gpu_requeue that writes intermediate files must either be restartable from the beginning (overwriting partially completed files) or from a last saved checkpoint. Researchers are responsible to choose jobs that can be run in this way.
- The gpu_requeue partition can be efficiently used to run interactive GPU jobs while developing or testing code and when an interruption won't cause a significant loss of data/work.
- Some researchers run large batches of "embarrassingly parallel", independent jobs, which take less than a couple hours each and can be performed in any order. Using the gpu_requeue partition provides access to substantial extra resources, and not too much work will be lost if a small percentage of these jobs are killed. The requeue functionality also means the jobs will be restarted automatically when preempted.
- Non-interactive jobs can be built with the capability of auto-restarting from a last saved checkpoint. A properly designed job would follow an algorithm to auto determine if a restart is possible, for example:
if a <restart_file> is present
restart from <restart_file>
start from beginning
while job is executing
periodically save a <restart_file>
if job completes
remove temporary files, including <restart_file>