The srun Slurm command allows running a job step using the allows you to connect, with a standard terminal shell, to the nodes where your jobs are running. From those shell connections you will be able to use the same computational resources allocated for your jobs. You can also run directly from login nodes a command against resources allocated to a separate already running job.
This feature can be useful for:
monitoring in real time your sbatch running jobs (you can use commands like "top" or “ps” to check your processes or look at the content of local /tmp folders on the compute node)
using resources allocated to a
...
running sbatch
...
job that you know might be idles at a given time. For example use GPU or CPU computing power when temporarily idle.
Get immediate access to some computational resources if you are in a urgent need. For example, you might be running a GPU jobs and still have enough VRAM (GPU memory) free on that card that could be used to run a separate process.
The syntax to use is srun --jobid=<jobid_number> ... followed by your desired command.
The shell or the commands started via the srun --jobid will be constrained to use only the resources available to the sbatch jobid job.
The syntax to use is srun --jobid=<jobid_number> ...
First, submit your This below is an example of how you can connect from a login node to a running sbatch job.
First, we had submitted a standard sbatch job, in this example called my_sbatch_job.sh:
...
| Code Block |
|---|
@login03:SLURM sbatch my_sbatch_job.sh Submitted batch job 10610731 |
Once the sbatch job is starts running, it is possible to start a shell as a slurm jobstep using the same resources allocated for the sbatch job (10610731 in this example).
...
Everything executed within that srun shell will run using sharing the same resource already allocated for the sbatch job.
It is possible to connect multiple times to the same sbatch job, as long as the sbatch job is in “RUNNING” state. Note that it is not possible to have concurrent srun --jobid connections to the same sbatch running job.
It is also possible to run directly non-interactive commands using the srun jobstep, for example:
...
Where the Linux command hostname was executed on the compute node where job 10610731 was dispatched and using the resources allocated for that job.
What can you do with srun --jobid=<jobid_number> ... :
monitor in real time your sbatch running jobs
use resources allocated to a running sbatch job that you know might be idles at a given time. For example use GPU or CPU computing power when temporarily idle.
Get immediate access to some computational resources if you are in a urgent need. For example, you might be running a GPU jobs and still have enough VRAM (GPU memory) free on that card that could be used to run a separate process.
| Warning |
|---|
Note: Any command you run from |