The O2 cluster includes a set of lab-contributed GPU nodes. These nodes were purchased by individual labs, but are made available to the rest of the O2 community within the re-queue GPU partition called gpu_requeue.
...
| Code Block |
|---|
sinfo --Format=nodehost,cpusstate,memory,statelong,gres -p gpu_requeue HOSTNAMES CPUS(A/I/O/T) MEMORY STATE GRES compute-ggc-1617-197245 046/202/0/2048 257548383000 idle mixed gpu:teslaM40rtx6000:210 compute-gc-17-245247 1/47/0/48 383000 mixed gpu:rtx6000:8 compute-gc-17-249 16/32/0/48 1000000 mixed gpu:a100:2 compute-gc-17-246 48/0/0/48 385218 383000 idle allocated gpu:rtx6000:10 compute-gcg-1716-246197 0/4820/0/4820 385218257548 idle gpu:rtx6000teslaM40:102 |
How Preemption Works
The labs that purchased these nodes have preemption priority on their own hardware. If the nodes are full and a researcher from one of those labs submits a job, one or more GPU jobs running on the gpu_requeue partition might be killed and re-queued in order to free resources for the Lab's job. That is, the gpu_requeue job will be cancelled, as if you ran the scancel command, and re-submitted (as long as you initially submitted with the flag --requeue).
...
How to Submit to the gpu_requeue Partition
To submit jobs on gpu_requeue you need to specify that partition with the flag "-p", and add the flag --requeue. Without the requeue flags jobs will still get killed but will not be automatically requeued.
...
How to Efficiently Use the gpu_requeue Partition
IMPORTANT:
In order to work properly, any job submitted to gpu_requeue that writes intermediate files must either be restartable from the beginning (overwriting partially completed files) or from a last saved checkpoint. Researchers are responsible to choose jobs that can be run in this way.
...