<div dir="ltr">Made the change in the gres.conf on local server file and restarted slurmd and slurmctld on master.... Unfortunately same error... <div><br></div><div>Distributed corrected gres.conf to all k20 servers, restarted slurmd and slurmdctl... Still has same error...</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Dec 3, 2018 at 4:04 PM Brian W. Johanson <<a href="mailto:bjohanso@psc.edu">bjohanso@psc.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Is that a lowercase k in k20 specified in the batch script and
nodename and a uppercase K specified in gres.conf?<br>
<br>
<div class="m_3582960547685116962moz-cite-prefix">On 12/03/2018 09:13 AM, Lou Nicotra
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi All, I have recently set up a slurm
cluster with my servers and I'm running into an
issue while submitting GPU jobs. It has something to
to with gres configurations, but I just can't seem
to figure out what is wrong. Non GPU jobs run fine.
<div><br>
</div>
<div>The error is as follows:</div>
<div>sbatch: error: Batch job submission failed:
Invalid Trackable RESource (TRES) specification
after submitting a batch job.</div>
<div><br>
</div>
<div>My batch job is as follows:</div>
<div>
<div>#!/bin/bash</div>
<div>#SBATCH --partition=tiger_1 # partition
name<br>
</div>
<div>#SBATCH --gres=gpu:k20:1<br>
</div>
<div>#SBATCH --gres-flags=enforce-binding</div>
<div>#SBATCH --time=0:20:00 # wall clock limit<br>
</div>
<div>#SBATCH --output=gpu-%J.txt</div>
<div>#SBATCH --account=lnicotra<br>
</div>
<div>module load cuda<br>
</div>
<div>python gpu1<br>
</div>
<div><br>
</div>
<div>Where gpu1 is a GPU test script that runs
correctly while invoked via python. Tiger_1
partition has servers with GPUs, with a mix of
1080GTX and K20 as specified in slurm.conf</div>
<div><br>
</div>
<div>I have defined GRES resources in the
slurm.conf file:</div>
<div>
<div># GPU GRES</div>
<div>GresTypes=gpu</div>
<div>NodeName=tiger[01,05,10,15,20]
Gres=gpu:1080gtx:2</div>
<div>NodeName=tiger[02-04,06-09,11-14,16-19,21-22]
Gres=gpu:k20:2</div>
</div>
<div><br>
</div>
<div>And have a local gres.conf on the servers
containing GPUs...</div>
<div>
<div>lnicotra@tiger11 ~# cat
/etc/slurm/gres.conf</div>
<div># GPU Definitions</div>
<div>#
NodeName=tiger[02-04,06-09,11-14,16-19,21-22]
Name=gpu Type=K20 File=/dev/nvidia[0-1]</div>
<div>Name=gpu Type=K20 File=/dev/nvidia[0-1]
Cores=0,1</div>
</div>
<div><br>
</div>
<div>and a similar one for the 1080GTX</div>
<div>
<div># GPU Definitions</div>
<div># NodeName=tiger[01,05,10,15,20] Name=gpu
Type=1080GTX File=/dev/nvidia[0-1]</div>
<div>Name=gpu Type=1080GTX File=/dev/nvidia[0-1]
Cores=0,1</div>
</div>
<div><br>
</div>
<div>The account manager seems to know about the
GPUs...</div>
<div>
<div>lnicotra@tiger11 ~# sacctmgr show tres</div>
<div> Type Name ID</div>
<div>-------- --------------- ------</div>
<div> cpu 1</div>
<div> mem 2</div>
<div> energy 3</div>
<div> node 4</div>
<div> billing 5</div>
<div> fs disk 6</div>
<div> vmem 7</div>
<div> pages 8</div>
<div> gres gpu 1001</div>
<div> gres gpu:k20 1002</div>
<div> gres gpu:1080gtx 1003</div>
</div>
<div><br>
</div>
<div>Can anyone point out what am I missing?</div>
<div><br>
</div>
<div>Thanks!</div>
<div>Lou</div>
<div><br>
</div>
<div><br>
</div>
-- <br>
<div dir="ltr" class="m_3582960547685116962gmail_signature">
<div dir="ltr">
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><b><span style="font-size:9.5pt;font-family:Arial,sans-serif;color:rgb(111,168,220)">Lou
Nicotra</span></b><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:Arial,sans-serif;color:rgb(102,102,102)">IT
Systems Engineer -
SLT</span><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:Arial,sans-serif;color:rgb(102,102,102)">Interactions
LLC</span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:Arial,sans-serif">o: </span><span style="font-size:9.5pt;font-family:Arial,sans-serif;color:rgb(102,102,102)"><a href="tel:781-405-5114" target="_blank"><span style="color:rgb(17,85,204)">908-673-1833</span></a></span><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:Arial,sans-serif;color:rgb(102,102,102)">m: <a href="tel:781-405-5114" target="_blank"><span style="color:rgb(17,85,204)">908-451-6983</span></a></span><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><u><span style="font-size:9.5pt;font-family:Arial,sans-serif;color:rgb(17,85,204)"><a href="mailto:lnicotra@interactions.com" target="_blank">lnicotra@interactions.com</a></span></u><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<span style="font-size:9.5pt;line-height:107%;font-family:Arial,sans-serif;color:rgb(102,102,102)"><a href="http://www.interactions.com/" target="_blank"><span style="color:rgb(17,85,204)">www.interactions.com</span></a></span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<font size="3" face="Times New Roman">
</font>
<p style="margin:0in 0in 8pt"><font size="3" face="Calibri">*******************************************************************************</font></p>
<font size="3" face="Times New Roman">
</font>
<p style="margin:0in 0in 8pt"><font size="3" face="Calibri">This
e-mail and any of its attachments may contain
Interactions LLC proprietary information, which is privileged,
confidential, or subject to copyright belonging to the
Interactions
LLC. This e-mail is intended solely for the use of the
individual or
entity to which it is addressed. If you are not the intended
recipient of this
e-mail, you are hereby notified that any dissemination,
distribution, copying,
or action taken in relation to the contents of and attachments
to this e-mail
is strictly prohibited and may be unlawful. If you have
received this e-mail in
error, please notify the sender immediately and permanently
delete the original
and any copy of this e-mail and any printout. Thank You. </font></p>
<font size="3" face="Times New Roman">
</font>
<p style="margin:0in 0in 8pt"><font face="Calibri"><font size="3">*******************************************************************************<span>
</span></font></font></p>
<font size="3" face="Times New Roman">
</font>
</blockquote>
<br>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><b><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#6fa8dc">Lou Nicotra</span></b><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">IT Systems Engineer -
SLT</span><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">Interactions LLC</span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:Arial,sans-serif">o: </span><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666"><a href="tel:781-405-5114" target="_blank"><span style="color:#1155cc">908-673-1833</span></a></span><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">m: <a href="tel:781-405-5114" target="_blank"><span style="color:#1155cc">908-451-6983</span></a></span><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><u><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#1155cc"><a href="mailto:lnicotra@interactions.com" target="_blank">lnicotra@interactions.com</a></span></u><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<span style="font-size:9.5pt;line-height:107%;font-family:"Arial",sans-serif;color:#666666"><a href="http://www.interactions.com/" target="_blank"><span style="color:#1155cc">www.interactions.com</span></a></span><br></div></div>
<br>
<font face="Times New Roman" size="3">
</font><p style="margin:0in 0in 8pt"><font face="Calibri" size="3">******************************<wbr>******************************<wbr>*******************</font></p><font face="Times New Roman" size="3">
</font><p style="margin:0in 0in 8pt"><font face="Calibri" size="3">This e-mail and any of its attachments may contain
Interactions LLC proprietary information, which is privileged,
confidential, or subject to copyright belonging to the Interactions
LLC. This e-mail is intended solely for the use of the individual or
entity to which it is addressed. If you are not the intended recipient of this
e-mail, you are hereby notified that any dissemination, distribution, copying,
or action taken in relation to the contents of and attachments to this e-mail
is strictly prohibited and may be unlawful. If you have received this e-mail in
error, please notify the sender immediately and permanently delete the original
and any copy of this e-mail and any printout. Thank You. </font></p><font face="Times New Roman" size="3">
</font><p style="margin:0in 0in 8pt"><font face="Calibri"><font size="3">******************************<wbr>******************************<wbr>*******************<span> </span></font></font></p><font face="Times New Roman" size="3">
</font>