<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
What you are describing is definitely doable. We have our system setup similarly. All nodes are in the "open" partition and "prio" partition, but a job submitted to the "prio" partition will preempt the open jobs.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I don't see anything clearly wrong with your slurm.conf settings. Ours are very similar, though we use only FORCE:1 for oversubscribe. You might try that just to see if there's a difference.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
What are the sbatch settings you are using when you submit the jobs?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Do you have PreemptExemptTime set to anything in slurm.conf?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
What is the reason squeue gives for the high priority jobs to be pending?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
For your "run regularly" goal, you might consider scrontab. If we can figure out priority and preemption, then that will start the job at a regular time.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Rob</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Fabrizio Roccato <f.roccato@isac.cnr.it><br>
<b>Sent:</b> Wednesday, May 24, 2023 7:17 AM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [slurm-users] hi-priority partition and preemption</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">[You don't often get email from f.roccato@isac.cnr.it. Learn why this is important at
<a href="https://aka.ms/LearnAboutSenderIdentification">https://aka.ms/LearnAboutSenderIdentification</a> ]<br>
<br>
Hi all,<br>
i'm trying to have two overlapping partition, say normal and hi-pri,<br>
so that when jobs are launched in the second one they can preempt the jobs<br>
allready running in the first one, automatically putting them in suspend<br>
state. After completition, the jobs in the normal partition must be<br>
automatically resumed.<br>
<br>
here are my (relevant) slurm.conf settings:<br>
<br>
> PreemptMode=suspend,gang<br>
> PreemptType=preempt/partition_prio<br>
><br>
> PartitionName=normal Nodes=node0[01-08] MaxTime=1800 PriorityTier=100 AllowAccounts=group1,group2 OverSubscribe=FORCE:20 PreemptMode=suspend<br>
> PartitionName=hi-pri Nodes=node0[01-08] MaxTime=360 PriorityTier=500 AllowAccounts=group2 OverSubscribe=FORCE:20 PreemptMode=off<br>
<br>
But so, jobs in the hi-pri partition where put in PD state and the ones<br>
allready running in the normal partition continue in their R status.<br>
What i'm wrong? What i'm missing?<br>
<br>
Since i have jobs thath must run at specific time and must have priority over<br>
all others, is this the correct way to do?<br>
<br>
<br>
Thanks<br>
<br>
FR<br>
<br>
</div>
</span></font></div>
</body>
</html>