<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Remove this line: <span class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p><span class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">#SBATCH
--nodes=1</span></p>
<p><span class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">Slurm assumes
you're requesting the whole node. --ntasks=1 should be adequate.
<br>
</span></p>
<pre class="moz-signature" cols="72">
</pre>
<div class="moz-cite-prefix">On 11/7/19 4:19 PM, Mike Mosley wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJZdgqoYrtFgC1sfZLmi=ocvwKgWQm=Amk99zOMm3POPc-79Fg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Greetings all:</div>
<div><br>
</div>
I'm attempting to configure the scheduler to schedule our GPU
boxes but have run into a bit of a snag.
<div><br>
</div>
<div>I have a box with two Tesla K80s. With my current
configuration, the scheduler will schedule one job on the box,
but if I submit a second job, it queues up until the first one
finishes:</div>
<div><br>
</div>
<div>My submit script:</div>
<div>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">#SBATCH
--partition=NodeSet1</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">#SBATCH
--nodes=1</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">#SBATCH
--ntasks=1</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">#SBATCH
--gres=gpu:k80:1</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">My
slurm.conf (the things I think are relevant)</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">GresTypes=gpu</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">SelectType=select/cons_tres</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">PartitionName=NodeSet1
Nodes=cht-c[1-4],cph-gpu1 Default=YES MaxTime=INFINITE
OverSubscribe=FORCE State=UP</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">NodeName=cph-gpu1
CPUs=16 Sockets=2 CoresPerSocket=8 ThreadsPerCore=1
RealMemory=257541 Gres=gpu:k80:2 Feature=gpu State=UNKNOWN</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">My
gres.conf:</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">NodeName=cph-gpu1
Name=gpu Type=k80 File=/dev/nvidia[0-1]</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">and
finally, the results of squeue:</span></p>
<p class="gmail-p1" style="margin:0px;font:14px
Menlo;color:rgb(16,8,182)"><span class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures">$
squeue</span></p>
<p class="gmail-p1" style="margin:0px;font:14px
Menlo;color:rgb(16,8,182)"><span class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-Apple-converted-space"> </span>JOBID
PARTITION <span class="gmail-Apple-converted-space"> </span>NAME
<span class="gmail-Apple-converted-space"> </span>USER
ST <span class="gmail-Apple-converted-space"> </span>TIME<span
class="gmail-Apple-converted-space"> </span>NODES
NODELIST(REASON)</span></p>
<p class="gmail-p1" style="margin:0px;font:14px
Menlo;color:rgb(16,8,182)"><span class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-Apple-converted-space"> </span>208<span
class="gmail-Apple-converted-space"> </span>NodeSet1 <span
class="gmail-Apple-converted-space"> </span>job.sh
jmmosley PD <span class="gmail-Apple-converted-space">
</span>0:00<span class="gmail-Apple-converted-space">
</span>1 (Resources)</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
class="gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-Apple-converted-space"> </span>207<span
class="gmail-Apple-converted-space"> </span>NodeSet1 <span
class="gmail-Apple-converted-space"> </span>job.sh
jmmosley<span class="gmail-Apple-converted-space"> </span>R
<span class="gmail-Apple-converted-space"> </span>4:12<span
class="gmail-Apple-converted-space"> </span>1
cph-gpu1</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">Any
idea what I am missing or have misconfigured?</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">Thanks
in advance.</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">Mike</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)"><br>
</span></p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span
style="font-family:Arial,Helvetica,sans-serif;font-size:small;color:rgb(34,34,34)">-- </span><br>
</p>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div style="font-size:12.8000001907349px">
<div dir="ltr">
<div><span style="font-family:'times new roman',serif"><b><i>J.
Michael Mosley</i></b><br>
University Research Computing<br>
The University of North Carolina at Charlotte<br>
9201 University City Blvd<br>
Charlotte, NC 28223<br>
<u>704.687.7065 </u> <u> j<i><a
href="mailto:mmosley@uncc.edu"
target="_blank" moz-do-not-send="true">mmosley@uncc.edu</a></i></u></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</body>
</html>