<div dir="ltr">Hi Ken,<div><br></div><div>Here is my slurm.conf:</div><div><br></div><div>





<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">ControlMachine=s19r2b08</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(64,11,217)"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">AuthType=auth/none</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">CryptoType=crypto/openssl</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">JobCredentialPrivateKey=/home/bsc33/bsc33882/slurm_over_slurm/etc/slurm.key</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">JobCredentialPublicCertificate=/home/bsc33/bsc33882/slurm_over_slurm/etc/slurm.cert</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(64,11,217)"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MpiDefault=none</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(64,11,217)"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">ProctrackType=proctrack/linuxproc</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">ReturnToService=1</span></p>
<p class="gmail-p3" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmctldPidFile=/home/bsc33/bsc33882/slurm_over_slurm/var/run/slurmctld.pid</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmctldPort=7001</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmdPidFile=/home/bsc33/bsc33882/slurm_over_slurm/var/run/slurmd.%n.pid</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmdPort=8009</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmdSpoolDir=/home/bsc33/bsc33882/slurm_over_slurm/var/spool/slurmd.%n</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmUser=bsc33882</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmdUser=bsc33882</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">StateSaveLocation=/home/bsc33/bsc33882/slurm_over_slurm/var/state</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SwitchType=switch/none</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(64,11,217)"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">TaskPlugin=task/none</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">TaskPluginParam=autobind=cores</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(64,11,217)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"># TIMERS</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">InactiveLimit=1800</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">KillWait=60</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">MinJobAge=300</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">OverTimeLimit=1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmctldTimeout=300</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmdTimeout=300</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(64,11,217)"><br></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(64,11,217)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"># SCHEDULING</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">FastSchedule=1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SchedulerType=sched/backfill</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SelectType=select/linear</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SchedulerParameters=bf_interval=30,default_queue_depth=50</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(64,11,217)"><br></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(64,11,217)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"># LOGGING AND ACCOUNTING</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(64,11,217)"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">ClusterName=cluster</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">JobCompType=jobcomp/script</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">JobCompLoc=/home/bsc33/bsc33882/slurm_over_slurm/script/trace.sh</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">JobAcctGatherFrequency=30</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">JobAcctGatherType=jobacct_gather/none</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmctldDebug=7</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmctldLogFile=/home/bsc33/bsc33882/slurm_over_slurm/var/slurmctld.log</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmdDebug=7</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SlurmdLogFile=/home/bsc33/bsc33882/slurm_over_slurm/var/slurmd.%n.log</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">DebugFlags=Backfill,SelectType</span></p>
<p class="gmail-p3" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(64,11,217)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"># COMPUTE NODES</span></p></div><div>





<p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">NodeName=s19r2b[09-10,12,14,16] CPUs=48 Sockets=2 CoresPerSocket=24 ThreadsPerCore=1 State=IDLE Port=7009</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">







</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">PartitionName=debug Nodes=s19r2b[09-10,12,14,16] Default=YES MaxTime=INFINITE State=UP</span></p></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, 4 Dec 2018 at 04:59, Kenneth Roberts <<a href="mailto:kroberts@materialsdesign.com">kroberts@materialsdesign.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div class="m_6224428641779981056WordSection1"><p class="MsoNormal">Hi –<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">The time stamps show that your 1^st sbatch job components start at the same time and then run for 1 minute.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">30 seconds after the simultaneous end of all three components of the 1^st sbatch, the two components of the 3^rd sbatch and the three components of the 2^nd all start. The two components of the 3^rd batch each run for 20 seconds. The three components of the 2^nd sbatch all run for 1 minute.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">The 3^rd sbatch start was delayed 5 seconds by the sleep, so they didn’t start with the 1^st batch.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Are you able to give the other parameters of your setup?  The SelectType? The node specs? These will affect scheduling.  Note, I’m wading into deep waters for me ... still learning slurm. (slurming? ;-)<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> <b>On Behalf Of </b>Ana Jokanovic<br><b>Sent:</b> Monday, December 3, 2018 12:40 AM<br><b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br><b>Subject:</b> Re: [slurm-users] backfill scheduler does not work for heterogeneous jobs (version 17.11)<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><div><p class="MsoNormal">Hi Ken,<u></u><u></u></p><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">I have read this page and I understood that in case of my example the third job should be backfilled. The second job can start after 15 minutes, but the third job requires only two nodes and 2 minutes, thus it can start immediately, but this does not happen. <u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">In the page that you referred to, they give an example:<u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p style="margin-right:0in;margin-bottom:.25in;margin-left:0in;line-height:18.0pt;vertical-align:baseline;box-sizing:border-box;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit"><span style="font-size:15.0pt;font-family:"Source Sans Pro",sans-serif;color:#46545c">For example, consider a heterogeneous job with three components. When considered as independent jobs, the components could be initiated at times now (component 0), now plus 2 hour (component 1), and now plus 1 hours (component 2). When the backfill scheduler runs in the first mode:<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.25in;line-height:18.0pt;vertical-align:baseline"><u></u><span style="font-size:15.0pt;font-family:"Source Sans Pro",sans-serif;color:#46545c"><span>1.<span style="font:7.0pt "Times New Roman"">     </span></span></span><u></u><span style="font-size:15.0pt;font-family:"Source Sans Pro",sans-serif;color:#46545c">Component 0 will be noted to possible to start now, but not initiated due to the additional components to be initiated<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.25in;line-height:18.0pt;vertical-align:baseline"><u></u><span style="font-size:15.0pt;font-family:"Source Sans Pro",sans-serif;color:#46545c"><span>2.<span style="font:7.0pt "Times New Roman"">     </span></span></span><u></u><span style="font-size:15.0pt;font-family:"Source Sans Pro",sans-serif;color:#46545c">Component 1 will be noted to be possible to start in 2 hours<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.25in;line-height:18.0pt;vertical-align:baseline"><u></u><span style="font-size:15.0pt;font-family:"inherit",serif;color:#46545c"><span>3.<span style="font:7.0pt "Times New Roman"">     </span></span></span><u></u><span style="font-size:15.0pt;font-family:"inherit",serif;color:#46545c">Component 2 will not be considered for scheduling until 2 hours in the future,<b> which leave some additional resources available for scheduling to other jobs</b><u></u><u></u></span></p><p style="margin-right:0in;margin-bottom:.25in;margin-left:0in;line-height:18.0pt;vertical-align:baseline;box-sizing:border-box;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit"><span style="font-size:15.0pt;font-family:"Source Sans Pro",sans-serif;color:#46545c">When the backfill scheduler executes next, it will use the second mode and (assuming no other state changes) all three job components will be considered available for scheduling no earlier than 2 hours in the future, <b>which may allow other jobs to be allocated resources before heterogeneous job component 0 could be initiated.</b><u></u><u></u></span></p></div><p class="MsoNormal">From this example, I understand that in my experiment the third job should be backfilled. The second job can start after 15 minutes, but the third job requires only two nodes and 2 minutes, thus it can start immediately, but this does not happen. <u></u><u></u></p><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">It seems there is a bug here. I also tried with the version 18.03, but it does not work either.<u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">Ana <u></u><u></u></p><div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">On Fri, 30 Nov 2018 at 17:46, Kenneth Roberts <<a href="mailto:kroberts@materialsdesign.com" target="_blank">kroberts@materialsdesign.com</a>> wrote:<u></u><u></u></p></div><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"><div><div><p class="MsoNormal">There are some Limitations that mention backfill on the heterogeneous job support page.<u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal"><a href="https://slurm.schedmd.com/heterogeneous_jobs.html#limitations" target="_blank">https://slurm.schedmd.com/heterogeneous_jobs.html#limitations</a><u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal">Maybe there’s some information there to help?<u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal">Ken<u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> <b>On Behalf Of </b>Ana Jokanovic<br><b>Sent:</b> Thursday, November 29, 2018 4:28 AM<br><b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br><b>Subject:</b> [slurm-users] backfill scheduler does not work for heterogeneous jobs (version 17.11)<u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p><div><div><p class="MsoNormal"> <u></u><u></u></p><div><div><p class="MsoNormal"> <u></u><u></u></p><div><p class="MsoNormal">Hello,<u></u><u></u></p><div><p class="MsoNormal"> <u></u><u></u></p></div><div><p class="MsoNormal">I did a simple test submitting the workload of three jobs (see below) on a cluster of 5 nodes:<u></u><u></u></p></div><div><p class="MsoNormal"> <u></u><u></u></p></div><div><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s3"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:#b42419">sbatch --cpus-per-task=2 --ntasks=6 --time=15 : --cpus-per-task=2 --ntasks=6 --time=15 : --cpus-per-task=2 --ntasks=6 --time=15</span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s3"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:#b42419">sbatch --cpus-per-task=2 --ntasks=6 --time=15 : --cpus-per-task=2 --ntasks=6 --time=15 : --cpus-per-task=2 --ntasks=6 --time=15</span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p2" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s3"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:#c1651c">sleep</span></span><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s2"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black"> </span></span><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s4"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:#b42419">5</span></span><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s2"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">;</span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s3"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:#b42419">sbatch --ntasks=1 --time=2 : --ntasks=1 --time=1</span></span><u></u><u></u></p><div><p class="MsoNormal"> <u></u><u></u></p></div><div><p class="MsoNormal">I would expect that the third submitted job is backfilled but it does not happen.<u></u><u></u></p></div><div><p class="MsoNormal">Here is the job completion log:<u></u><u></u></p></div><div><p class="MsoNormal"> <u></u><u></u></p></div><div><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">JobId=2 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694 StartTime=1543317714 EndTime=1543317774 NodeList=s19r2b09 NodeCnt=1 ProcCnt=48 </span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">JobId=3 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694 StartTime=1543317714 EndTime=1543317774 NodeList=s19r2b10 NodeCnt=1 ProcCnt=48 </span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">JobId=4 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694 StartTime=1543317714 EndTime=1543317774 NodeList=s19r2b12 NodeCnt=1 ProcCnt=48 </span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">JobId=8 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED Partition=debug TimeLimit=00:02:00 SubmitTime=1543317699 StartTime=1543317804 EndTime=1543317824 NodeList=s19r2b14 NodeCnt=1 ProcCnt=48 </span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">JobId=9 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED Partition=debug TimeLimit=00:01:00 SubmitTime=1543317699 StartTime=1543317804 EndTime=1543317824 NodeList=s19r2b16 NodeCnt=1 ProcCnt=48 </span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">JobId=5 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694 StartTime=1543317804 EndTime=1543317864 NodeList=s19r2b09 NodeCnt=1 ProcCnt=48 </span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">JobId=6 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694 StartTime=1543317804 EndTime=1543317864 NodeList=s19r2b10 NodeCnt=1 ProcCnt=48 </span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">JobId=7 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694 StartTime=1543317804 EndTime=1543317864 NodeList=s19r2b12 NodeCnt=1 ProcCnt=48 </span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black"> </span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">Would you expect this behavior?</span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black"> </span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">Thanks.</span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black"> </span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">Best regards,</span></span><u></u><u></u></p><p class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-p1" style="margin:0in;margin-bottom:.0001pt;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span class="m_6224428641779981056m650910500980912746m6220139011529046574m3208256795183356166gmail-s1"><span style="font-size:8.5pt;font-family:"Menlo",serif;color:black">Ana</span></span><u></u><u></u></p></div><p class="MsoNormal">-- <u></u><u></u></p><div><div><div><p class="MsoNormal"><span style="color:#888888">Ana Jokanovic, PhD<br>Barcelona Supercomputing Center<br>c/ Jordi Girona 1-3, K2M Building, 1st floor<br>08034 Barcelona - SPAIN<br>e-mail: </span><a href="mailto:anaj82@gmail.com" target="_blank">anaj82@gmail.com</a><span style="color:#888888"> or </span><a href="mailto:ana.jokanovic@bsc.es" target="_blank">ana.jokanovic@bsc.es</a><span style="color:#888888"><br>tel: +34 93 4137246</span><u></u><u></u></p></div></div></div></div></div></div><p class="MsoNormal"><br clear="all"><u></u><u></u></p><div><p class="MsoNormal"> <u></u><u></u></p></div><p class="MsoNormal">-- <u></u><u></u></p><div><div><div><p class="MsoNormal"><span style="color:#888888">Ana Jokanovic, PhD<br>Barcelona Supercomputing Center<br>c/ Jordi Girona 1-3, K2M Building, 1st floor<br>08034 Barcelona - SPAIN<br>e-mail: </span><a href="mailto:anaj82@gmail.com" target="_blank">anaj82@gmail.com</a><span style="color:#888888"> or </span><a href="mailto:ana.jokanovic@bsc.es" target="_blank">ana.jokanovic@bsc.es</a><span style="color:#888888"><br>tel: +34 93 4137246</span><u></u><u></u></p></div></div></div></div></div><p class="MsoNormal"><br clear="all"><u></u><u></u></p><div><p class="MsoNormal"> <u></u><u></u></p></div><p class="MsoNormal">-- <u></u><u></u></p><div><div><div><p class="MsoNormal"><span style="color:#888888">Ana Jokanovic, PhD<br>Barcelona Supercomputing Center<br>c/ Jordi Girona 1-3, K2M Building, 1st floor<br>08034 Barcelona - SPAIN<br>e-mail: </span><a href="mailto:anaj82@gmail.com" target="_blank">anaj82@gmail.com</a><span style="color:#888888"> or </span><a href="mailto:ana.jokanovic@bsc.es" target="_blank">ana.jokanovic@bsc.es</a><span style="color:#888888"><br>tel: +34 93 4137246</span><u></u><u></u></p></div></div></div></div></div></div></blockquote></div><p class="MsoNormal"><br clear="all"><u></u><u></u></p><div><p class="MsoNormal"><u></u> <u></u></p></div><p class="MsoNormal">-- <u></u><u></u></p><div><div><div><p class="MsoNormal"><span style="color:#888888">Ana Jokanovic, PhD<br>Barcelona Supercomputing Center<br>c/ Jordi Girona 1-3, K2M Building, 1st floor<br>08034 Barcelona - SPAIN<br>e-mail: </span><a href="mailto:anaj82@gmail.com" target="_blank">anaj82@gmail.com</a><span style="color:#888888"> or </span><a href="mailto:ana.jokanovic@bsc.es" target="_blank">ana.jokanovic@bsc.es</a><span style="color:#888888"><br>tel: +34 93 4137246</span><u></u><u></u></p></div></div></div></div></div></div></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><span><font color="#888888">Ana Jokanovic, PhD<br>Barcelona Supercomputing Center<br>c/ Jordi Girona 1-3, K2M Building, 1st floor<br>08034 Barcelona - SPAIN<br>e-mail: <span><a href="mailto:anaj82@gmail.com" target="_blank">anaj82@gmail.com</a> or <a href="mailto:ana.jokanovic@bsc.es" target="_blank">ana.jokanovic@bsc.es</a></span><br>
tel: <a value="+34600741501">+34 93 4137246</a></font></span><a value="+34600741501"><br></a></div></div></div>