<div dir="ltr"><div>Greetings all:</div><div><br></div>I'm attempting to configure the scheduler to schedule our GPU boxes but have run into a bit of a snag.<div><br></div><div>I have a box with two Tesla K80s. With my current configuration, the scheduler will schedule one job on the box, but if I submit a second job, it queues up until the first one finishes:</div><div><br></div><div>My submit script:</div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#SBATCH --partition=NodeSet1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#SBATCH --nodes=1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#SBATCH --ntasks=1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#SBATCH --gres=gpu:k80:1</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">My slurm.conf (the things I think are relevant)</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">GresTypes=gpu</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SelectType=select/cons_tres</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">PartitionName=NodeSet1 Nodes=cht-c[1-4],cph-gpu1 Default=YES MaxTime=INFINITE OverSubscribe=FORCE State=UP</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">
</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">NodeName=cph-gpu1 CPUs=16 Sockets=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=257541 Gres=gpu:k80:2 Feature=gpu State=UNKNOWN</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">My gres.conf:</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">NodeName=cph-gpu1 Name=gpu Type=k80 File=/dev/nvidia[0-1]</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">and finally, the results of squeue:</span></p><p class="gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">$ squeue</span></p><p class="gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>JOBID PARTITION <span class="gmail-Apple-converted-space"> </span>NAME <span class="gmail-Apple-converted-space"> </span>USER ST <span class="gmail-Apple-converted-space"> </span>TIME<span class="gmail-Apple-converted-space"> </span>NODES NODELIST(REASON)</span></p><p class="gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>208<span class="gmail-Apple-converted-space"> </span>NodeSet1 <span class="gmail-Apple-converted-space"> </span>job.sh jmmosley PD <span class="gmail-Apple-converted-space"> </span>0:00<span class="gmail-Apple-converted-space"> </span>1 (Resources)</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>207<span class="gmail-Apple-converted-space"> </span>NodeSet1 <span class="gmail-Apple-converted-space"> </span>job.sh jmmosley<span class="gmail-Apple-converted-space"> </span>R <span class="gmail-Apple-converted-space"> </span>4:12<span class="gmail-Apple-converted-space"> </span>1 cph-gpu1</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">Any idea what I am missing or have misconfigured?</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">Thanks in advance.</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">Mike</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">-- </span><br></p><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="font-size:12.8000001907349px"><div dir="ltr"><div><span style="font-family:'times new roman',serif"><b><i>J. Michael Mosley</i></b><br>University Research Computing<br>The University of North Carolina at Charlotte<br>9201 University City Blvd<br>Charlotte, NC 28223<br><u>704.687.7065 </u> <u> j<i><a href="mailto:mmosley@uncc.edu" target="_blank">mmosley@uncc.edu</a></i></u></span></div></div></div></div></div></div></div>