<div dir="ltr">Thanks Michael. I will try 17.x as I also could not see anything wrong with my settings... Will report back afterwards...<div><br></div><div>Lou</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Dec 4, 2018 at 9:11 AM Michael Di Domenico <<a href="mailto:mdidomenico4@gmail.com">mdidomenico4@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">unfortunately, someone smarter then me will have to help further. I'm<br>
not sure i see anything specifically wrong. The one thing i might try<br>
is backing the software down to a 17.x release series. I recently<br>
tried 18.x and had some issues. I can't say whether it'll be any<br>
different, but you might be exposing an undiagnosed bug in the 18.x<br>
branch<br>
On Mon, Dec 3, 2018 at 4:17 PM Lou Nicotra <<a href="mailto:lnicotra@interactions.com" target="_blank">lnicotra@interactions.com</a>> wrote:<br>
><br>
> Made the change in the gres.conf on local server file and restarted slurmd and slurmctld on master.... Unfortunately same error...<br>
><br>
> Distributed corrected gres.conf to all k20 servers, restarted slurmd and slurmdctl... Still has same error...<br>
><br>
> On Mon, Dec 3, 2018 at 4:04 PM Brian W. Johanson <<a href="mailto:bjohanso@psc.edu" target="_blank">bjohanso@psc.edu</a>> wrote:<br>
>><br>
>> Is that a lowercase k in k20 specified in the batch script and nodename and a uppercase K specified in gres.conf?<br>
>><br>
>> On 12/03/2018 09:13 AM, Lou Nicotra wrote:<br>
>><br>
>> Hi All, I have recently set up a slurm cluster with my servers and I'm running into an issue while submitting GPU jobs. It has something to to with gres configurations, but I just can't seem to figure out what is wrong. Non GPU jobs run fine.<br>
>><br>
>> The error is as follows:<br>
>> sbatch: error: Batch job submission failed: Invalid Trackable RESource (TRES) specification after submitting a batch job.<br>
>><br>
>> My batch job is as follows:<br>
>> #!/bin/bash<br>
>> #SBATCH --partition=tiger_1 # partition name<br>
>> #SBATCH --gres=gpu:k20:1<br>
>> #SBATCH --gres-flags=enforce-binding<br>
>> #SBATCH --time=0:20:00 # wall clock limit<br>
>> #SBATCH --output=gpu-%J.txt<br>
>> #SBATCH --account=lnicotra<br>
>> module load cuda<br>
>> python gpu1<br>
>><br>
>> Where gpu1 is a GPU test script that runs correctly while invoked via python. Tiger_1 partition has servers with GPUs, with a mix of 1080GTX and K20 as specified in slurm.conf<br>
>><br>
>> I have defined GRES resources in the slurm.conf file:<br>
>> # GPU GRES<br>
>> GresTypes=gpu<br>
>> NodeName=tiger[01,05,10,15,20] Gres=gpu:1080gtx:2<br>
>> NodeName=tiger[02-04,06-09,11-14,16-19,21-22] Gres=gpu:k20:2<br>
>><br>
>> And have a local gres.conf on the servers containing GPUs...<br>
>> lnicotra@tiger11 ~# cat /etc/slurm/gres.conf<br>
>> # GPU Definitions<br>
>> # NodeName=tiger[02-04,06-09,11-14,16-19,21-22] Name=gpu Type=K20 File=/dev/nvidia[0-1]<br>
>> Name=gpu Type=K20 File=/dev/nvidia[0-1] Cores=0,1<br>
>><br>
>> and a similar one for the 1080GTX<br>
>> # GPU Definitions<br>
>> # NodeName=tiger[01,05,10,15,20] Name=gpu Type=1080GTX File=/dev/nvidia[0-1]<br>
>> Name=gpu Type=1080GTX File=/dev/nvidia[0-1] Cores=0,1<br>
>><br>
>> The account manager seems to know about the GPUs...<br>
>> lnicotra@tiger11 ~# sacctmgr show tres<br>
>> Type Name ID<br>
>> -------- --------------- ------<br>
>> cpu 1<br>
>> mem 2<br>
>> energy 3<br>
>> node 4<br>
>> billing 5<br>
>> fs disk 6<br>
>> vmem 7<br>
>> pages 8<br>
>> gres gpu 1001<br>
>> gres gpu:k20 1002<br>
>> gres gpu:1080gtx 1003<br>
>><br>
>> Can anyone point out what am I missing?<br>
>><br>
>> Thanks!<br>
>> Lou<br>
>><br>
>><br>
>> --<br>
>><br>
>> Lou Nicotra<br>
>><br>
>> IT Systems Engineer - SLT<br>
>><br>
>> Interactions LLC<br>
>><br>
>> o: 908-673-1833<br>
>><br>
>> m: 908-451-6983<br>
>><br>
>> <a href="mailto:lnicotra@interactions.com" target="_blank">lnicotra@interactions.com</a><br>
>><br>
>> <a href="http://www.interactions.com" rel="noreferrer" target="_blank">www.interactions.com</a><br>
>><br>
>> *******************************************************************************<br>
>><br>
>> This e-mail and any of its attachments may contain Interactions LLC proprietary information, which is privileged, confidential, or subject to copyright belonging to the Interactions LLC. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this e-mail is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original and any copy of this e-mail and any printout. Thank You.<br>
>><br>
>> *******************************************************************************<br>
>><br>
>><br>
><br>
><br>
> --<br>
><br>
> Lou Nicotra<br>
><br>
> IT Systems Engineer - SLT<br>
><br>
> Interactions LLC<br>
><br>
> o: 908-673-1833<br>
><br>
> m: 908-451-6983<br>
><br>
> <a href="mailto:lnicotra@interactions.com" target="_blank">lnicotra@interactions.com</a><br>
><br>
> <a href="http://www.interactions.com" rel="noreferrer" target="_blank">www.interactions.com</a><br>
><br>
> *******************************************************************************<br>
><br>
> This e-mail and any of its attachments may contain Interactions LLC proprietary information, which is privileged, confidential, or subject to copyright belonging to the Interactions LLC. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this e-mail is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original and any copy of this e-mail and any printout. Thank You.<br>
><br>
> *******************************************************************************<br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><b><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#6fa8dc">Lou Nicotra</span></b><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">IT Systems Engineer -
SLT</span><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">Interactions LLC</span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:Arial,sans-serif">o: </span><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666"><a href="tel:781-405-5114" target="_blank"><span style="color:#1155cc">908-673-1833</span></a></span><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">m: <a href="tel:781-405-5114" target="_blank"><span style="color:#1155cc">908-451-6983</span></a></span><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<p style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><u><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#1155cc"><a href="mailto:lnicotra@interactions.com" target="_blank">lnicotra@interactions.com</a></span></u><span style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
<span style="font-size:9.5pt;line-height:107%;font-family:"Arial",sans-serif;color:#666666"><a href="http://www.interactions.com/" target="_blank"><span style="color:#1155cc">www.interactions.com</span></a></span><br></div></div>
<br>
<font face="Times New Roman" size="3">
</font><p style="margin:0in 0in 8pt"><font face="Calibri" size="3">******************************<wbr>******************************<wbr>*******************</font></p><font face="Times New Roman" size="3">
</font><p style="margin:0in 0in 8pt"><font face="Calibri" size="3">This e-mail and any of its attachments may contain
Interactions LLC proprietary information, which is privileged,
confidential, or subject to copyright belonging to the Interactions
LLC. This e-mail is intended solely for the use of the individual or
entity to which it is addressed. If you are not the intended recipient of this
e-mail, you are hereby notified that any dissemination, distribution, copying,
or action taken in relation to the contents of and attachments to this e-mail
is strictly prohibited and may be unlawful. If you have received this e-mail in
error, please notify the sender immediately and permanently delete the original
and any copy of this e-mail and any printout. Thank You. </font></p><font face="Times New Roman" size="3">
</font><p style="margin:0in 0in 8pt"><font face="Calibri"><font size="3">******************************<wbr>******************************<wbr>*******************<span> </span></font></font></p><font face="Times New Roman" size="3">
</font>