<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">After trying to approach this with preempt/partition_prio, we ended up moving to QOS based preemption due to some issues with suspend/requeue, and also wanting to use QOS for quicker/easier tweaks than changing partitions as a whole.<div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class=""><font face="Menlo" class="">PreemptType=preempt/qos</font></div><div class=""><font face="Menlo" class="">PreemptMode=SUSPEND,GANG</font></div></blockquote><blockquote type="cite" class=""><div class=""><font face="Menlo" class="">PartitionName=part-lopri Nodes=nodes[000-NNN] Default=NO MaxTime=INFINITE OverSubscribe=FORCE:1 PriorityTier=10 State=UP</font></div></blockquote><blockquote type="cite" class=""><div class=""><font face="Menlo" class="">PartitionName=part-hipir Nodes=nodes[000-NNN] Default=NO MaxTime=INFINITE OverSubscribe=NO PriorityTier=100 State=UP PreemptMode=OFF</font></div></blockquote><div class=""><br class=""></div>We then have a few QOS that have different Priority values, as well as PreemptMode, QOS it can preempt, etc.</div><div class=""><div class=""><blockquote type="cite" class=""><div class=""><font face="Menlo" class=""> Name Priority Preempt PreemptMode</font></div><div class=""><font face="Menlo" class="">---------- ---------- ---------- -----------</font></div><div class=""><span style="font-family: Menlo;" class=""> rq 10 requeue</span></div><div class=""><font face="Menlo" class=""> susp 11 suspend </font></div><div class=""><font face="Menlo" class=""> hipri 100 rq,susp cluster</font></div></blockquote><blockquote type="cite" class=""><font face="Menlo" class=""> test 50 rq requeue</font></blockquote></div></div><div class=""><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">The rq qos is stateless and can be requeued, susp qos is stateful and needs to be suspended.</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Hipri can preempt rq and susp.</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">We also have a test qos with very strict limits (wall clock, job count, tres count, etc) that allows small jobs to jump the queue, for quick testing before submitting into the full queue.</div></div><div class=""><br class=""></div><div class="">The tricky part for us was that we have some stateful jobs that need to be suspended, and some stateless jobs that can just be requeued without issue.</div><div class="">But we want the hipri partition to take precedent, on the same hardware pool.</div><div class="">We also didn’t want gang scheduling to flip flop jobs running, which if memory serves me correctly, was how/why we ended up going with duplicative partitions for the purpose of priority, because we couldn’t get preemption to work intra-partition correctly.</div><div class="">In a perfect world, we would have just the single partition and everything handled in QOS, but it’s working, and that’s what mattered.</div><div class=""><br class=""></div><div class="">I’m not sure how any of this would work with FORCE:20 oversubscribe, but hopefully it offers something useful to try next.</div><div class=""><br class=""></div><div class="">Reed</div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On May 24, 2023, at 8:42 AM, Groner, Rob <<a href="mailto:rug262@psu.edu" class="">rug262@psu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">What you are describing is definitely doable. We have our system setup similarly. All nodes are in the "open" partition and "prio" partition, but a job submitted to the "prio" partition will preempt the open jobs.</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">I don't see anything clearly wrong with your slurm.conf settings. Ours are very similar, though we use only FORCE:1 for oversubscribe. You might try that just to see if there's a difference.</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">What are the sbatch settings you are using when you submit the jobs?</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Do you have PreemptExemptTime set to anything in slurm.conf?</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">What is the reason squeue gives for the high priority jobs to be pending?</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">For your "run regularly" goal, you might consider scrontab. If we can figure out priority and preemption, then that will start the job at a regular time.</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Rob</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 908.453125px;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" class="">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Fabrizio Roccato <<a href="mailto:f.roccato@isac.cnr.it" class="">f.roccato@isac.cnr.it</a>><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, May 24, 2023 7:17 AM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:slurm-users@lists.schedmd.com" class="">slurm-users@lists.schedmd.com</a><span class="Apple-converted-space"> </span><<a href="mailto:slurm-users@lists.schedmd.com" class="">slurm-users@lists.schedmd.com</a>><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>[slurm-users] hi-priority partition and preemption</font><div class=""> </div></div><div class="BodyFragment" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><font size="2" class=""><span style="font-size: 11pt;" class=""><div class="PlainText">[You don't often get email from<span class="Apple-converted-space"> </span><a href="mailto:f.roccato@isac.cnr.it" class="">f.roccato@isac.cnr.it</a>. Learn why this is important at<span class="Apple-converted-space"> </span><a href="https://aka.ms/LearnAboutSenderIdentification" class="">https://aka.ms/LearnAboutSenderIdentification</a><span class="Apple-converted-space"> </span>]<br class=""><br class="">Hi all,<br class=""> i'm trying to have two overlapping partition, say normal and hi-pri,<br class="">so that when jobs are launched in the second one they can preempt the jobs<br class="">allready running in the first one, automatically putting them in suspend<br class="">state. After completition, the jobs in the normal partition must be<br class="">automatically resumed.<br class=""><br class="">here are my (relevant) slurm.conf settings:<br class=""><br class="">> PreemptMode=suspend,gang<br class="">> PreemptType=preempt/partition_prio<br class="">><br class="">> PartitionName=normal Nodes=node0[01-08] MaxTime=1800 PriorityTier=100 AllowAccounts=group1,group2 OverSubscribe=FORCE:20 PreemptMode=suspend<br class="">> PartitionName=hi-pri Nodes=node0[01-08] MaxTime=360 PriorityTier=500 AllowAccounts=group2 OverSubscribe=FORCE:20 PreemptMode=off<br class=""><br class="">But so, jobs in the hi-pri partition where put in PD state and the ones<br class="">allready running in the normal partition continue in their R status.<br class="">What i'm wrong? What i'm missing?<br class=""><br class="">Since i have jobs thath must run at specific time and must have priority over<br class="">all others, is this the correct way to do?<br class=""><br class=""><br class="">Thanks<br class=""><br class="">FR</div></span></font></div></div></blockquote></div><br class=""></div></body></html>