<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,</p>
<p>I struggle with configuring job preempting. I have nodes with 8
Nvidia A100 GPUs. I have two partitions: short (lower priority)
and sfglab (higher priority). I want to allow higher priority jobs
to preempt (REQUEUE mode) lower priority job. It looks like it
works, however it works too good.</p>
<p>Job from higher priority partition preempts entire host instead
of only single job which would be enough to release resources for
higher priority partition. Whats more it lock the rest of
resources until high-prio job will end. What am I doing wrong?</p>
<p>Here is example:</p>
<p><font face="monospace">$ srun --test-only -G1 -c1 --mem 1M -p
sfglab<br>
srun: Job 501151 to start at 2023-01-17T12:46:01 using 1
processors on nodes dgx-1 in partition sfglab<br>
srun: Preempts:
363278,501001,501029,501075,501076,501077,501120,501121</font></p>
<p>To release these resources it would be enough to preempt one job
instead of all.<br>
</p>
<p><br>
Here is my config:<br>
</p>
<p>slurm.conf</p>
<p>(...)<br>
</p>
<p><font face="monospace">DefMemPerCPU = 100<br>
JobAcctGatherFrequency = 30<br>
JobAcctGatherType = jobacct_gather/linux<br>
PreemptMode = REQUEUE<br>
PreemptType = preempt/partition_prio<br>
PreemptExemptTime = 00:00:00<br>
SelectType = select/cons_tres<br>
SelectTypeParameters = CR_CORE_MEMORY</font><br>
</p>
<p>(...)<br>
</p>
<p><font face="monospace">PartitionName=short
Nodes=dgx-[1-4],sr-[1-3] MaxTime=1-0 State=UP PriorityTier=10000
Default=YES DefaultTime=0-01:00:00 OverSubscribe=NO
PreemptMode=requeue<br>
</font></p>
<p><font face="monospace">PartitionName=sfglab Nodes=dgx-1
MaxTime=10-0 State=UP PriorityTier=20000 PreemptMode=off
OverSubscribe=NO AllowAccounts=sfglab</font><br>
</p>
<div class="moz-signature">-- <br>
best regards | pozdrawiam serdecznie<br>
<b>Michał Kadlof</b><br>
<table style="font-size:9pt;border: 1px solid
transparent;padding:0 10px; border-collapse: collapse;">
<tbody>
<tr>
<td style="font-style: italic;border: 1px solid
transparent;padding:0 10px;">Head of the high performance
computing center</td>
</tr>
<tr>
<td style="font-style: italic;border: 1px solid
transparent;padding:0 10px;">Eden<sup>N</sup> cluster
administrator</td>
</tr>
<tr>
<td style="border: 1px solid transparent;padding:0
10px;opacity:0.5;">Faculty of Mathematics and Computer
Science</td>
</tr>
<tr>
<td style="border: 1px solid transparent;padding:0
10px;opacity:0.5;">Warsaw University of Technology</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>