<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Could you give me an idea of what your partition and qos settings are? I've tried the following and I'm getting odd results:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
slurm.conf
<div class="ContentPasted0">PreemptType: preempt/qos</div>
<div class="ContentPasted0">PreemptMode: 'SUSPEND,GANG'</div>
<div class="ContentPasted0">PreemptExemptTime: '00:00:00'</div>
<div><br class="ContentPasted0">
</div>
<div>Partitions:</div>
<div class="ContentPasted0">PartitionName=DEFAULT OverSubscribe=FORCE:1 Nodes=slurm[2-4]</div>
<div class="ContentPasted0">PartitionName=active Default=YES QOS=normal</div>
<div class="ContentPasted0">PartitionName=hipri Default=NO QOS=expedite</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">QOS:</div>
<div class="ContentPasted0">sacctmgr -i modify qos where name=normal set <span style="">PreemptExemptTime=00:03:00 PreemptMode=SUSPEND</span></div>
<div class="ContentPasted0">sacctmgr -i modify qos where name=expedite set PreemptExemptTime=-1 <span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">PreemptMode=OFF</span></div>
</div>
<div id="appendonsend"></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I took these settings directly from the google group I linked before, and I'm seeing what he's seeing...that no preemption happens. What I see is that, even if a job from "active" is already running, slurm will let me submit jobs from hipri to take up resources
as though the active job wasn't there. In other words, there's some time sharing going on, even though the jobs are different partitions. The online docs indicate that jobs from different partitions should NOT be time sharing. I also do not see the "active"
job getting preempted once it has run through its preempexempttime. It never gets preempted.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thanks.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Rob</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Christopher Samuel <chris@csamuel.org><br>
<b>Sent:</b> Tuesday, March 7, 2023 3:40 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] PreemptExemptTime</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">On 3/7/23 6:46 am, Groner, Rob wrote:<br>
<br>
> Over global settings are PreemptMode=SUSPEND,GANG and <br>
> PreemptType=preempt/partition_prio. We have a high priority partition <br>
> that nothing should ever preempt, and an open partition that is always <br>
> preemptable. In between is a burst partition. It can be preempted if <br>
> the high priority partition needs the resources. That's the partition <br>
> we'd like to guarantee a 1 hour run time on. Looking at the sacctmgr <br>
> man page, it gives this info on QOS<br>
<br>
Just a quick comment, here you're talking about both partitions and <br>
QOS's with respect to preemption, I think for this you need to pick just <br>
one of those options and only use those configs. For instance we just <br>
use QOS's for preemption and our exempt time works in that case.<br>
<br>
Hope this helps!<br>
<br>
All the best,<br>
Chris<br>
-- <br>
Chris Samuel : <a href="https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&data=05%7C01%7Crug262%40psu.edu%7C0ed8f546e70843e7266208db1f4c5471%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638138184919726959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=md1MaA%2FIj%2Fb%2B27e7JLkVwSY5IUSVnsJIdx6PADl0JF4%3D&reserved=0" data-auth="NotApplicable">
https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&data=05%7C01%7Crug262%40psu.edu%7C0ed8f546e70843e7266208db1f4c5471%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638138184919726959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=md1MaA%2FIj%2Fb%2B27e7JLkVwSY5IUSVnsJIdx6PADl0JF4%3D&reserved=0</a>
: Berkeley, CA, USA<br>
<br>
<br>
</div>
</span></font></div>
</body>
</html>