<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I have defined a partition and corresponding QOS in Slurm. This is the serial queue to which we route jobs that require up to (and including) 20 cpus. The nodes controlled by serial are shared. I've set the QOS like so..</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
[djb1@cyan53 slurm]$ <b>sacctmgr show qos serial format=name,maxtresperuser</b><br>
<div> Name MaxTRESPU <br>
</div>
<div>---------- ------------- <br>
</div>
serial cpu=120<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The max cpus/user is set high to try to ensure (as often as possible) that the nodes are all busy and not in mixed states. Obviously this cannot be the case all the time -- depending upon memory requirements, etc.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I noticed that a number of jobs were pending with the reason QOSMaxNodePerUserLimit. I've tried firing test jobs to the queue myself and noticed that I can never have more than 32 jobs running (each requesting 1 cpu) and the rest are pending as per the reason
above. Since the QOS cpu/user limit is set to 120 I would expect to be able to run more jobs -- given that some serial nodes are still not fully occupied. Furthermore, I note that other users appear not to be able to use more then 32 cpus in the queue. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The 32 limit does make a degree of sense. The "normal" QOS is set to cpus/user=1280, nodes/user=32. It's almost like the 32 cpus in the serial queue are being counted as nodes -- as per the pending reason. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Could someone please help me understand this issue and how to avoid it?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Best regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
David</div>
</body>
</html>