RC created two simplified commands, O2squeue and O2sacct based on slurm squeue and sacct, that can be used to gather information about your active (pending or running) jobs and your past jobs.
O2squeue
This command is based on slurm command squeue and will return information about your pending and running jobs. For example:
login05:~ O2squeue JOBID PARTITION STATE TIME_LIMIT TIME NODELIST(REASON) ELIGIBLE_TIME START_TIME TRES_ALLOC 21801263 interactive RUNNING 12:00:00 2:09:52 compute-a-16-160 2020-11-09T11:35:49 2020-11-09T11:36:19 cpu=1,mem=2G,node=1,billing=1
The field STATE describes the states of your jobs and it will normally be either PENDING or RUNNING. When a job is pending NODELIST(REASON) describes the reason why the job is pending, most common reasons are:
BadConstraints: The job's constraints can not be satisfied.
Dependency: This job is waiting for a dependent job to complete.
InvalidQOS: The job's QOS is invalid.
JobHeldAdmin: The job is held by a system administrator.
JobHeldUser: The job is held by the user.
None: The job has not been evaluated yet by the scheduler
Priority: One or more higher priority jobs exist for this partition or advanced reservation.
QOSJobLimit: The job's QOS has reached its maximum job count.
QOSResourceLimit: The job's QOS has reached some resource limit.
QOSTimeLimit: The job's QOS has reached its time limit.
ReqNodeNotAvail: Some node specifically required by the job is not currently available.
Reservation: The job is waiting its advanced reservation to become available.
Resources: The job is waiting for resources to become available.
The field ELIGIBLE_TIME indicates the time when a job becomes eligible to be dispatched, this is usually the submit time unless there are reason why the job cannot be dispatched such as job dependencies or unavailable resources requested.
For running jobs START_TIME indicates the time when the job was dispatched. For pending jobs it indicates the expected start time. Note that expected start time is only calculated for the first few pending jobs of each user and it is in general an upper bound value.
TRES indicates the resources requested by the job
O2squeue can take as inputs the string R or PD to selectively list only running or pending jobs
O2sacct
This command is based on slurm command sacct and will return information about your past jobs. For example:
login05:~ O2sacct
JobID Partition State Nodelist Start TimeLimit Elapsed CPUefficiency_% AllocTRES MaxMemoryUsed
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
21769333 priority COMPLETED compute-a-16-162 2020-11-09T00:15:11 00:05:00 00:00:43 67.44 billing=1,cpu=1,mem=0.98G,node=1
21769333.batch COMPLETED compute-a-16-162 2020-11-09T00:15:11 00:00:43 67.44 cpu=1,mem=0.98G,node=1 0.53G
21769333.extern COMPLETED compute-a-16-162 2020-11-09T00:15:11 00:00:45 0.00 billing=1,cpu=1,mem=0.98G,node=1 0
21775057 priority COMPLETED compute-a-16-168 2020-11-09T01:17:10 00:05:00 00:00:50 58.00 billing=1,cpu=1,mem=0.98G,node=1
21775057.batch COMPLETED compute-a-16-168 2020-11-09T01:17:10 00:00:50 58.00 cpu=1,mem=0.98G,node=1 0.50G
21775057.extern COMPLETED compute-a-16-168 2020-11-09T01:17:10 00:00:50 0.00 billing=1,cpu=1,mem=0.98G,node=1 0
The field CPUefficiency_% indicates how efficiently the job used the CPU cores allocated. If this number is less than 75% and the job is requesting more than one cpu core, then your job is probably requesting more cores than it can use.
AllocTRES reports the total amount of resources (cpu, memory, etc.) allocated for the job.
MaxMemoryUsed reports the maximum amount of memory used by the job, if this value is significantly smaller than the allocated memory reported by AllocTRES you should reduce the memory requested by your job. (Note: for mpi jobs this is the max amount of memory used in each node)
O2sacct can take as arguments a jobid or a start-time, an end-time and a job state. Times can be used to define a search interval and must be in the format YYYY-MM-DD or YYYY-MM-DDThh-mm-ss, if a time range is not specify only recent jobs are searched. If a end-time is not specified end-time is set to now, if a job state is specified a search time window must be provided and the comand will return jobs that were in the specified state during the given time interval.
Possible job states are:
CA = job cancelled
CD = job completed
F = job failed
NF = job failed due to Node failure
TO = job timeout
R = job running
OOM = job out of memory
PD = job pending
PR = job preempted
-------------------
Examples:
O2sacct
O2sacct 8964563
O2sacct CD 2018-04-01
O2sacct 2018-04-01 R
O2sacct 2018-04-01 2018-04-10
O2sacct 2018-04-01 2018-04-10 R
Note:
If you are running a large batch of jobs, and a limited number of them require more memory and/or wall time than the average, it is almost always better to request resources for the average jobs. The few jobs that fail for timeout and/or exceeded memory can then be resubmitted with increased memory and/or wall time. Limiting the amount of requested memory and wall time can result in a significant reduction of the pending time and a much faster processing time for the overall workflow.