<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
We see the following issue with smaller jobs pushing back large jobs. We are using slurm 19.05.8 so not sure if this is patched in newer releases. <span style="color: rgb(32, 31, 30); font-family: Calibri, sans-serif; font-size: 11pt;">With a 4 node test partition
I submit 3 jobs as 2 users</span></p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
ssh hpcdev1@navy51 'sbatch --nodes=3 --ntasks-per-node=40 --partition=backfilltest --time=120 --wrap="sleep 7200"'</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
ssh hpcdev2@navy51 'sbatch --nodes=4 --ntasks-per-node=40 --partition=backfilltest --time=60 --wrap="sleep 3600"'</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
ssh hpcdev2@navy51 'sbatch --nodes=4 --ntasks-per-node=40 --partition=backfilltest --time=60 --wrap="sleep 3600"'</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
Then I increase the priority of the pending jobs significantly. Reading the manual, my understanding is that nodes job should be held for these jobs.</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
for job in $(squeue -h -p backfilltest -t pd -o %i); do scontrol update job ${job} priority=1000000000;done</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
squeue -p backfilltest -o "%i | %u | %C | %Q | %l | %S | %T"</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
JOBID | USER | CPUS | PRIORITY | TIME_LIMIT | START_TIME | STATE</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
28482 | hpcdev2 | 160 | 1000000000 | 1:00:00 | N/A | PENDING</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
28483 | hpcdev2 | 160 | 1000000000 | 1:00:00 | N/A | PENDING</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
28481 | hpcdev1 | 120 | 50083 | 2:00:00 | 2020-12-08T09:44:15 | RUNNING</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
So, there is one node free in our 4 node partition. Naturally, a small job with a walltime of less than 1 hour could run in that but we are also seeing backfill start longer jobs.</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
backfilltest up 2-12:00:00 3 alloc reddev[001-003]</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
backfilltest up 2-12:00:00 1 idle reddev004</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
ssh hpcdev3@navy51 'sbatch --nodes=1 --ntasks-per-node=40 --partition=backfilltest --time=720 --wrap="sleep 432000"'</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
squeue -p backfilltest -o "%i | %u | %C | %Q | %l | %S | %T"</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
JOBID | USER | CPUS | PRIORITY | TIME_LIMIT | START_TIME | STATE</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
28482 | hpcdev2 | 160 | 1000000000 | 1:00:00 | N/A | PENDING</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
28483 | hpcdev2 | 160 | 1000000000 | 1:00:00 | N/A | PENDING</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
28481 | hpcdev1 | 120 | 50083 | 2:00:00 | 2020-12-08T09:44:15 | RUNNING</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
28484 | hpcdev3 | 40 | 37541 | 12:00:00 | 2020-12-08T09:54:48 | RUNNING</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
Is this expect behaviour? It is also weird that the pending jobs don't have a start time. I have increased the backfill parameters significantly, but <span style="color: rgb(32, 31, 30); font-family: Calibri, sans-serif; font-size: 11pt;">it doesn't seem to
affect this at all.</span></p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
SchedulerParameters=bf_window=14400,bf_resolution=2400,bf_max_job_user=80,bf_continue,default_queue_depth=1000,bf_interval=60</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
<br>
</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
Best regards,</p>
<p class="x_MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri, sans-serif;color:rgb(32, 31, 30);background-color:rgb(255, 255, 255)">
David</p>
<br>
</div>
</body>
</html>