<div dir="ltr"><div class="gmail_default" style="font-family:monospace">Hi</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">I believe this is how the preemption algorithm works- it selects the entire node's resources:</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">> For performance reasons, the backfill scheduler reserves whole nodes for jobs, not partial nodes.</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">   - <a href="https://slurm.schedmd.com/preempt.html#limitations">https://slurm.schedmd.com/preempt.html#limitations</a><br></div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">However, that does specifically call out the backfill scheduler.  Is that the scheduler type you're using?</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace"> - Michael</div><div class="gmail_default" style="font-family:monospace"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 17, 2023 at 4:06 AM Michał Kadlof <<a href="mailto:michal.kadlof@pw.edu.pl">michal.kadlof@pw.edu.pl</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  

    
  
  <div>
    <p>Hi,</p>
    <p>I struggle with configuring job preempting. I have nodes with 8
      Nvidia A100 GPUs. I have two partitions: short (lower priority)
      and sfglab (higher priority). I want to allow higher priority jobs
      to preempt (REQUEUE mode) lower priority job. It looks like it
      works, however it works too good.</p>
    <p>Job from higher priority partition preempts entire host instead
      of only single job which would be enough to release resources for
      higher priority partition. Whats more it lock the rest of
      resources until high-prio job will end. What am I doing wrong?</p>
    <p>Here is example:</p>
    <p><font face="monospace">$ srun --test-only -G1 -c1 --mem 1M -p
        sfglab<br>
        srun: Job 501151 to start at 2023-01-17T12:46:01 using 1
        processors on nodes dgx-1 in partition sfglab<br>
        srun:   Preempts:
        363278,501001,501029,501075,501076,501077,501120,501121</font></p>
    <p>To release these resources it would be enough to preempt one job
      instead of all.<br>
    </p>
    <p><br>
      Here is my config:<br>
    </p>
    <p>slurm.conf</p>
    <p>(...)<br>
    </p>
    <p><font face="monospace">DefMemPerCPU            = 100<br>
        JobAcctGatherFrequency  = 30<br>
        JobAcctGatherType       = jobacct_gather/linux<br>
        PreemptMode             = REQUEUE<br>
        PreemptType             = preempt/partition_prio<br>
        PreemptExemptTime       = 00:00:00<br>
        SelectType              = select/cons_tres<br>
        SelectTypeParameters    = CR_CORE_MEMORY</font><br>
    </p>
    <p>(...)<br>
    </p>
    <p><font face="monospace">PartitionName=short
        Nodes=dgx-[1-4],sr-[1-3] MaxTime=1-0 State=UP PriorityTier=10000
        Default=YES DefaultTime=0-01:00:00 OverSubscribe=NO
        PreemptMode=requeue<br>
      </font></p>
    <p><font face="monospace">PartitionName=sfglab Nodes=dgx-1
        MaxTime=10-0 State=UP PriorityTier=20000 PreemptMode=off
        OverSubscribe=NO AllowAccounts=sfglab</font><br>
    </p>
    <div>-- <br>
      best regards | pozdrawiam serdecznie<br>
      <b>Michał Kadlof</b><br>
      <table style="font-size:9pt;border:1px solid transparent;padding:0px 10px;border-collapse:collapse">
        <tbody>
          <tr>
            <td style="font-style:italic;border:1px solid transparent;padding:0px 10px">Head of the high performance
              computing center</td>
          </tr>
          <tr>
            <td style="font-style:italic;border:1px solid transparent;padding:0px 10px">Eden^N cluster
              administrator</td>
          </tr>
          <tr>
            <td style="border:1px solid transparent;padding:0px 10px;opacity:0.5">Faculty of Mathematics and Computer
              Science</td>
          </tr>
          <tr>
            <td style="border:1px solid transparent;padding:0px 10px;opacity:0.5">Warsaw University of Technology</td>
          </tr>
        </tbody>
      </table>
    </div>
  </div>

</blockquote></div>