<div dir="ltr"><div class="gmail_default" style="font-family:monospace">I'm not sure there's enough information in your message- Slurm version and configs are often necessary to make a more confident diagnosis. However, the behaviour you are looking for (lower priority jobs skipping the line) is called "backfill". There's docs here: <a href="https://slurm.schedmd.com/sched_config.html#backfill">https://slurm.schedmd.com/sched_config.html#backfill</a></div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">It should be loaded and active by default which is why I'm not super confident here. There may also be something else going on with the node configuration as it looks like 1596 would maybe need the same node? Maybe there's not enough CPU or memory to accommodate both jobs (1596 and 1739)?</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">HTH</div><div class="gmail_default" style="font-family:monospace"> - Michael</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 2, 2023 at 5:13 AM Cumer Cristiano <<a href="mailto:CristianoMaria.Cumer@unibz.it">CristianoMaria.Cumer@unibz.it</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-8822239974845864476">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<p style="text-decoration:none;border:0px solid rgb(217,217,227);box-sizing:border-box;margin:0px 0px 1.25em;font-family:Söhne,ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,Ubuntu,Cantarell,"Noto Sans",sans-serif,"Helvetica Neue",Arial,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";color:rgb(55,65,81)">
Hello,</p>
<p style="text-decoration:none;border:0px solid rgb(217,217,227);box-sizing:border-box;margin:1.25em 0px;font-family:Söhne,ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,Ubuntu,Cantarell,"Noto Sans",sans-serif,"Helvetica Neue",Arial,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";color:rgb(55,65,81)">
I'm quite a newbie regarding Slurm. I recently created a small Slurm instance to manage our GPU resources. I have this situation:</p>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Consolas,Courier,monospace"> JOBID STATE TIME ACCOUNT PARTITION PRIORITY REASON CPU MIN_MEM TRES_PER_NODE</span>
<div><span style="font-family:Consolas,Courier,monospace"> 1739 PENDING 0:00 standard gpu-low 5 Priority 1 80G gres:gpu:a100_1g.10gb:1</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1738 PENDING 0:00 standard gpu-low 5 Priority 1 80G gres:gpu:a100-sxm4-80gb:1</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1737 PENDING 0:00 standard gpu-low 5 Priority 1 80G gres:gpu:a100-sxm4-80gb:1</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1736 PENDING 0:00 standard gpu-low 5 Resources 1 80G gres:gpu:a100-sxm4-80gb:1</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1740 PENDING 0:00 standard gpu-low 1 Priority 1 8G gres:gpu:a100_3g.39gb</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1735 PENDING 0:00 standard gpu-low 1 Priority 8 64G gres:gpu:a100-sxm4-80gb:1</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1596 RUNNING 1-13:26:45 standard gpu-low 3 None 2 64G gres:gpu:a100_1g.10gb:1</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1653 RUNNING 21:09:52 standard gpu-low 2 None 1 16G gres:gpu:1</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1734 RUNNING 59:52 standard gpu-low 1 None 8 64G gres:gpu:a100-sxm4-80gb:1</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1733 RUNNING 1:01:54 standard gpu-low 1 None 8 64G gres:gpu:a100-sxm4-80gb:1</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1732 RUNNING 1:02:39 standard gpu-low 1 None 8 40G gres:gpu:a100-sxm4-80gb:1</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1731 RUNNING 1:08:28 standard gpu-low 1 None 8 40G gres:gpu:a100-sxm4-80gb:1</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1718 RUNNING 10:16:40 standard gpu-low 1 None 2 8G gres:gpu:v100</span></div>
<div><span style="font-family:Consolas,Courier,monospace"> 1630 RUNNING 1-00:21:21 standard gpu-low 1 None 1 30G gres:gpu:a100_3g.39gb</span></div>
<span style="font-family:Consolas,Courier,monospace"> 1610 RUNNING 1-09:53:23 standard gpu-low 1 None 2 8G gres:gpu:v100</span></div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<p style="text-decoration:none;border:0px solid rgb(217,217,227);box-sizing:border-box;margin:1.25em 0px;font-family:Söhne,ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,Ubuntu,Cantarell,"Noto Sans",sans-serif,"Helvetica Neue",Arial,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";color:rgb(55,65,81)">
Job 1736 is in the PENDING state since there are no more available a100-sxm4-80gb GPUs. The job priority starts to rise with time (priority 5) as expected. Now another user submits job 1739 on a gres:gpu:a100_1g.10gb:1 that is available, but the job is not
starting since its priority is 1. This is obviously not the desired outcome, and I believe I must change the scheduling strategy. Could someone with more experience than me give me some hints?</p>
<p style="text-decoration:none;border:0px solid rgb(217,217,227);box-sizing:border-box;margin:1.25em 0px 0px;font-family:Söhne,ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,Ubuntu,Cantarell,"Noto Sans",sans-serif,"Helvetica Neue",Arial,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";color:rgb(55,65,81)">
Thanks, Cristiano</p>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
</div>
</div></blockquote></div>