<html style="direction: ltr;">
<head>
<meta http-equiv="content-type" content="text/html; charset=windows-1255">
<style type="text/css">body p { margin-bottom: 0cm; margin-top: 0pt; } </style>
</head>
<body style="direction: ltr;"
bidimailui-detected-decoding-type="preferred-charset"
bgcolor="#FFFFFF" text="#000000">
Hey everyone,<br>
Perhaps I am asking a basic question, but I really dont understand
how the preemption works.<br>
The scenario(simplified for the example) is like this: <br>
<br>
Nodes:<br>
NodeName=A1 CPUS=2 RealMemory=128906 TmpDisk=117172 <br>
NodeName=A2 CPUS=30 RealMemory=128906 TmpDisk=117172 Gres=gpu:3<br>
<br>
Partitions:<br>
PartitionName=lab1 Nodes=A2 QOS=lab Default=No State=UP<br>
PartitionName=all Nodes=A2,A1 QOS=normal Default=Yes State=UP<br>
<br>
Users:<br>
u1 : qos=lab<br>
u2: qos=normal<br>
<br>
<u>commands</u>(in this order):<br>
u2: srun --gres=gpu:2 --pty bash<br>
u1: srun --gres=gpu:2 --pty bash<br>
<br>
<u>result</u><br>
<u>squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R %Q"</u><br>
<br>
JOBID PARTITION NAME USER ST TIME NODES
NODELIST(REASON) PRIORITY<br>
318 lab1 bash
u1 PD 0:00 1 (Resources)
101177<br>
317 all bash
u2 R 0:21 1 A2
20<br>
<br>
<br>
As you can see u1 didnt get his resources because(I believe) qos
cannot preempt another qos which run on different partition, oven
though they use the same resources.<br>
<br>
How should i configure the cluster so that all users with specific
qos(lab), can suspend jobs in all other qos(not lab) for specific
partition(lab1)?<br>
<br>
<br>
<br>
<u>
sacctmgr show qos</u><br>
Name Priority GraceTime Preempt PreemptMode<br>
lab1 1000 00:01:00 normal suspend<br>
normal 0 00:00:00 <br>
<br>
<br>
<u>slurm.conf:</u><br>
<br>
<br>
PreemptType=preempt/qos<br>
PreemptMode=suspend,gang<br>
<br>
PriorityType=priority/multifactor<br>
PriorityDecayHalfLife=30-0<br>
PriorityMaxAge=10000<br>
PriorityWeightFairshare=10000<br>
PriorityWeightQOS=100000<br>
<br>
AccountingStorageEnforce=associations,limits,qos<br>
<br>
<br>
<br>
Thanks in advance, Nadav<br>
</body>
</html>