<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">I can confirm that we do preemption based on partition for one of our clusters. I will say that we are not using time-based partitions, ours are always up and they are based on group node ownership. I wonder if Slurm is refusing to preempt
a job in a DOWN partition. Maybe try leaving the partition UP, but just change the priority of the partition.
<o:p></o:p></p>
<p class="MsoNormal">One other suggestion would be to turn up the debugging on the Slurm controller and/or use DebugFlags. I don’t know for sure which flag would give the best data, but I would start with the priority flag. With the right debugging turned on
the slurmctld.log should give you more data on how or why it is making its decisions.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Mike<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Russell Jones <arjones85@gmail.com><br>
<b>Date: </b>Tuesday, August 24, 2021 at 10:36<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>[External] Re: [slurm-users] Preemption not working for jobs in higher priority partition<o:p></o:p></span></p>
</div>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="line-height:12.0pt;background:#FFEB9C"><b><span style="font-size:10.0pt;color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt;color:black"> This email originated from outside of the Colorado School of Mines organization.
Do not click on links or open attachments unless you recognize the sender and know the content is safe.<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">I have been researching this further and I see other systems that appear to be set up the same way we are. Example: <a href="https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhpcrcf.atlassian.net%2Fwiki%2Fspaces%2FTCP%2Fpages%2F733184001%2FHow-to%2BUse%2Bthe%2Bpreempt%2BPartition&data=04%7C01%7Cmrobbert%40mines.edu%7Ca69c3fac8af3497278b508d9671d4700%7C997209e009b346239a4d76afa44a675c%7C0%7C0%7C637654197710622568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qqslNuf2I4DxfD4%2FVB53WS0%2FfMUAbEndrsBl9Y2OaL8%3D&reserved=0">https://hpcrcf.atlassian.net/wiki/spaces/TCP/pages/733184001/How-to+Use+the+preempt+Partition</a>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Any further insight into what may be wrong with our setup is appreciated. I am not seeing what is wrong with my config, but it also isn't working anymore to allow preemption.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Aug 20, 2021 at 9:46 AM Russell Jones <<a href="mailto:arjones85@gmail.com">arjones85@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">I could have swore I had tested this before implementing it and it worked as expected.
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If I am dreaming that testing - is there a way of allowing preemption across partitions? <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Aug 20, 2021 at 8:40 AM Brian Andrus <<a href="mailto:toomuchit@gmail.com" target="_blank">toomuchit@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p>IIRC, Preemption is determined by partition first, not node.<o:p></o:p></p>
<p>Since your pending job is in the 'day' partition, it will not preempt something in the 'night' partition (even if the node is in both).<o:p></o:p></p>
<p>Brian Andrus<o:p></o:p></p>
<div>
<p class="MsoNormal">On 8/19/2021 2:49 PM, Russell Jones wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hi all, <o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
I could use some help to understand why preemption is not working for me properly. I have a job blocking other jobs that doesn't make sense to me. Any assistance is appreciated, thank you!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I have two partitions defined in slurm, a day time and a night time pariition:<o:p></o:p></p>
</div>
<blockquote style="margin-left:30.0pt;margin-right:0in">
<div>
<p class="MsoNormal">Day partition - PriorityTier of 5, always Up. Limited resources under this QOS.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Night partition - PriorityTier of 5 during night time, during day time set to Down and PriorityTier changed to 1. Jobs can be submitted to night queue for an unlimited QOS as long as resources are available. <br>
<br>
The thought here is jobs can continue to run in the night partition, even during the day time, until resources are requested from the day partition. Jobs would then be requeued/canceled in the night partition to satisfy those requirements.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
Current output of "scontrol show part" :<o:p></o:p></p>
</div>
<blockquote style="margin-left:30.0pt;margin-right:0in">
<div>
<p class="MsoNormal">PartitionName=day<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> AllocNodes=ALL Default=NO QoS=part_day<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> MaxNodes=UNLIMITED MaxTime=1-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Nodes=cluster-r1n[01-13],cluster-r2n[01-08]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> PriorityJobFactor=1 PriorityTier=5 RootOnly=NO ReqResv=NO OverSubscribe=NO<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> OverTimeLimit=NONE PreemptMode=REQUEUE<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> State=UP TotalCPUs=336 TotalNodes=21 SelectTypeParameters=NONE<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> JobDefaults=(null)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-left:30.0pt;margin-right:0in">
<div>
<p class="MsoNormal">PartitionName=night<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> AllocNodes=ALL Default=NO QoS=part_night<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> MaxNodes=22 MaxTime=7-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Nodes=cluster-r1n[01-13],cluster-r2n[01-08]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> OverTimeLimit=NONE PreemptMode=REQUEUE<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> State=DOWN TotalCPUs=336 TotalNodes=21 SelectTypeParameters=NONE<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> JobDefaults=(null)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<br>
I currently have a job in the night partition that is blocking jobs in the day partition, even though the day partition has a PriorityTier of 5, and night partition is Down with a PriorityTier of 1.<br>
<br>
My current slurm.conf preemption settings are:<o:p></o:p></p>
</div>
<blockquote style="margin-left:30.0pt;margin-right:0in">
<div>
<p class="MsoNormal">PreemptMode=REQUEUE<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">PreemptType=preempt/partition_prio<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
The blocking job's scontrol show job output is:<o:p></o:p></p>
</div>
<blockquote style="margin-left:30.0pt;margin-right:0in">
<div>
<p class="MsoNormal">JobId=105713 JobName=jobname<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Priority=1986 Nice=0 Account=xxx QOS=normal<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> JobState=RUNNING Reason=None Dependency=(null)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> RunTime=17:49:39 TimeLimit=7-00:00:00 TimeMin=N/A<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> SubmitTime=2021-08-18T22:36:36 EligibleTime=2021-08-18T22:36:36<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> AccrueTime=2021-08-18T22:36:36<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> StartTime=2021-08-18T22:36:39 EndTime=2021-08-25T22:36:39 Deadline=N/A<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> PreemptEligibleTime=2021-08-18T22:36:39 PreemptTime=None<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> SuspendTime=None SecsPreSuspend=0 LastSchedEval=2021-08-18T22:36:39<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Partition=night AllocNode:Sid=cluster-1:1341505<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> ReqNodeList=(null) ExcNodeList=(null)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> NodeList=cluster-r1n[12-13],cluster-r2n[04-06]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> BatchHost=cluster-r1n12<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> NumNodes=5 NumCPUs=80 NumTasks=5 CPUs/Task=1 ReqB:S:C:T=0:0:*:*<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> TRES=cpu=80,node=5,billing=80,gres/gpu=20<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Features=(null) DelayBoot=00:00:00<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
The job that is being blocked:<o:p></o:p></p>
</div>
<blockquote style="margin-left:30.0pt;margin-right:0in">
<div>
<p class="MsoNormal">JobId=105876 JobName=bash<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Priority=2103 Nice=0 Account=xxx QOS=normal<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> JobState=PENDING Reason=Nodes_required_for_job_are_DOWN,_DRAINED_or_reserved_for_jobs_in_higher_priority_partitions Dependency=(null)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> SubmitTime=2021-08-19T16:19:23 EligibleTime=2021-08-19T16:19:23<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> AccrueTime=2021-08-19T16:19:23<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> StartTime=Unknown EndTime=Unknown Deadline=N/A<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> SuspendTime=None SecsPreSuspend=0 LastSchedEval=2021-08-19T16:26:43<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Partition=day AllocNode:Sid=cluster-1:2776451<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> ReqNodeList=(null) ExcNodeList=(null)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> NodeList=(null)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> NumNodes=3 NumCPUs=40 NumTasks=40 CPUs/Task=1 ReqB:S:C:T=0:0:*:*<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> TRES=cpu=40,node=1,billing=40<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Features=(null) DelayBoot=00:00:00<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</blockquote>
<p class="MsoNormal"><br>
<br>
Why is the day job not preempting the night job? <o:p></o:p></p>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>