<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-AU" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Following on from what Michael said, the default Slurm configuration is to allocate only one job per node. If GRES a100_1g.10gb is on the same node ensure to enable “SelectType=select/cons_res” (info
at <a href="https://slurm.schedmd.com/cons_res.html">https://slurm.schedmd.com/cons_res.html</a>) to permit multiple jobs to use the same node.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Also using “TaskPlugin=task/cgroup” is useful to ensure that users cannot inadvertently access resources not allocated to other jobs on the same node (refer to the slurm.conf man page).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"> -Greg<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
<b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Michael Gutteridge <michael.gutteridge@gmail.com><br>
<b>Date: </b>Wednesday, 2 August 2023 at 5:22 pm<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>[EXTERNAL] Re: [slurm-users] Job in "priority" status - resources available<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">I'm not sure there's enough information in your message- Slurm version and configs are often necessary to make a more confident diagnosis. However, the behaviour you are
looking for (lower priority jobs skipping the line) is called "backfill". There's docs here: <a href="https://urldefense.com/v3/__https:/slurm.schedmd.com/sched_config.html*backfill__;Iw!!Nmw4Hv0!yN2vkwBRGKx9XuVO3o7g6Ca8yN0A5bXazd8I1g0g1FaKEOi2P0xAg6Z_1eyqdwOFsvIv0D64pDlw4G36wojaY0uNkOhsh72u8g$">https://slurm.schedmd.com/sched_config.html#backfill</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">It should be loaded and active by default which is why I'm not super confident here. There may also be something else going on with the node configuration as it looks like
1596 would maybe need the same node? Maybe there's not enough CPU or memory to accommodate both jobs (1596 and 1739)?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">HTH<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""> - Michael<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">On Wed, Aug 2, 2023 at 5:13 AM Cumer Cristiano <<a href="mailto:CristianoMaria.Cumer@unibz.it">CristianoMaria.Cumer@unibz.it</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div style="mso-element:para-border-div;border:solid #D9D9E3 1.0pt;padding:0cm 0cm 0cm 0cm;margin-left:36.0pt;margin-right:0cm">
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:15.0pt;margin-left:0cm;border:none;padding:0cm;box-sizing:border-box">
<span style="font-family:"Segoe UI",sans-serif;color:#374151">Hello,<o:p></o:p></span></p>
<p style="mso-margin-top-alt:15.0pt;margin-right:0cm;margin-bottom:15.0pt;margin-left:0cm;border:none;padding:0cm;box-sizing:border-box">
<span style="font-family:"Segoe UI",sans-serif;color:#374151">I'm quite a newbie regarding Slurm. I recently created a small Slurm instance to manage our GPU resources. I have this situation:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> JOBID STATE TIME ACCOUNT PARTITION PRIORITY REASON CPU MIN_MEM TRES_PER_NODE</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1739 PENDING 0:00 standard gpu-low 5 Priority 1 80G gres:gpu:a100_1g.10gb:1</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1738 PENDING 0:00 standard gpu-low 5 Priority 1 80G gres:gpu:a100-sxm4-80gb:1</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1737 PENDING 0:00 standard gpu-low 5 Priority 1 80G gres:gpu:a100-sxm4-80gb:1</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1736 PENDING 0:00 standard gpu-low 5 Resources 1 80G gres:gpu:a100-sxm4-80gb:1</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1740 PENDING 0:00 standard gpu-low 1 Priority 1 8G gres:gpu:a100_3g.39gb</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1735 PENDING 0:00 standard gpu-low 1 Priority 8 64G gres:gpu:a100-sxm4-80gb:1</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1596 RUNNING 1-13:26:45 standard gpu-low 3 None 2 64G gres:gpu:a100_1g.10gb:1</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1653 RUNNING 21:09:52 standard gpu-low 2 None 1 16G gres:gpu:1</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1734 RUNNING 59:52 standard gpu-low 1 None 8 64G gres:gpu:a100-sxm4-80gb:1</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1733 RUNNING 1:01:54 standard gpu-low 1 None 8 64G gres:gpu:a100-sxm4-80gb:1</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1732 RUNNING 1:02:39 standard gpu-low 1 None 8 40G gres:gpu:a100-sxm4-80gb:1</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1731 RUNNING 1:08:28 standard gpu-low 1 None 8 40G gres:gpu:a100-sxm4-80gb:1</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1718 RUNNING 10:16:40 standard gpu-low 1 None 2 8G gres:gpu:v100</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1630 RUNNING 1-00:21:21 standard gpu-low 1 None 1 30G gres:gpu:a100_3g.39gb</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:Consolas;color:black"> 1610 RUNNING 1-09:53:23 standard gpu-low 1 None 2 8G gres:gpu:v100</span><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div style="mso-element:para-border-div;border:solid #D9D9E3 1.0pt;padding:0cm 0cm 0cm 0cm;margin-left:36.0pt;margin-right:0cm">
<p style="mso-margin-top-alt:15.0pt;margin-right:0cm;margin-bottom:15.0pt;margin-left:0cm;border:none;padding:0cm;box-sizing:border-box">
<span style="font-family:"Segoe UI",sans-serif;color:#374151">Job 1736 is in the PENDING state since there are no more available a100-sxm4-80gb GPUs. The job priority starts to rise with time (priority 5) as expected. Now another user submits job 1739 on a
gres:gpu:a100_1g.10gb:1 that is available, but the job is not starting since its priority is 1. This is obviously not the desired outcome, and I believe I must change the scheduling strategy. Could someone with more experience than me give me some hints?<o:p></o:p></span></p>
<p style="mso-margin-top-alt:15.0pt;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;border:none;padding:0cm;box-sizing:border-box">
<span style="font-family:"Segoe UI",sans-serif;color:#374151">Thanks, Cristiano<o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>