<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
Andrew,<br>
<br>
you could try change it to the following:<br>
<span><br>
/etc/slurm/slurm.conf:<br>
NodeName=node[1-3] CPUs=40 RealMemory=48000 Sockets=2
CoresPerSocket=10 ThreadsPerCore=2 Feature="p4000"
Gres=gpu:pascal:8 State=UNKNOWN<br>
</span>
<div>NodeName=node[4-5,7-10] CPUs=8 RealMemory=48000 Sockets=2
CoresPerSocket=4 ThreadsPerCore=1 Feature="p1000"
Gres=gpu:pascal:4
State=UNKNOWN<br>
</div>
<div>NodeName=node[6] CPUs=24 RealMemory=30000 Sockets=2
CoresPerSocket=6 ThreadsPerCore=2 Feature="p1000"
Gres=gpu:pascal:4 State=UNKNOWN</div>
<br>
<span>/etc/slurm/gres.conf</span><br>
<div>NodeName=node[1-3] Name=gpu Type=pascal File=/dev/nvidia[0-7]<br>
</div>
<div>NodeName=node[4-10] Name=gpu Type=pascal File=/dev/nvidia[0-4]</div>
<br>
Best<br>
Marcus<br>
<br>
<div class="moz-cite-prefix">On 5/15/20 11:48 PM, Speer, Andrew
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DM6PR07MB5673587E7A4E1AA59CF935E2BABD0@DM6PR07MB5673.namprd07.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
I've run into a bit of an issue when trying to define GPU's in
our slurm conf. Any insight is appreciated.<br>
Hopefully relevant lines from the configs below.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
Error:<br>
<span>[2020-05-15T16:35:14.862] error:
gres_plugin_node_config_unpack: No plugin configured to
process GRES data from node node3 (Name:gpu Type:(null)
PluginID:7696487 Count:2)<br>
</span>
<div>[2020-05-15T16:35:15.321] error:
gres_plugin_node_config_unpack: No plugin configured to
process GRES data from node node4 (Name:gpu Type:(null)
PluginID:7696487 Count:1)<br>
</div>
<div>[2020-05-15T16:35:15.738] error:
gres_plugin_node_config_unpack: No plugin configured to
process GRES data from node node5 (Name:gpu Type:(null)
PluginID:7696487 Count:1)<br>
</div>
<div>[2020-05-15T16:35:16.229] error:
gres_plugin_node_config_unpack: No plugin configured to
process GRES data from node node6 (Name:gpu Type:(null)
PluginID:7696487 Count:1)</div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<br>
/etc/slurm/slurm.conf:<br>
GresTypes=gpu</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<span>NodeName=node[1-3] CPUs=40 RealMemory=48000 Sockets=2
CoresPerSocket=10 ThreadsPerCore=2 Feature="pascal,p4000"
Gres=gpu:8 State=UNKNOWN<br>
</span>
<div>NodeName=node[4-5,7-10] CPUs=8 RealMemory=48000 Sockets=2
CoresPerSocket=4 ThreadsPerCore=1 Feature="pascal,p1000"
Gres=gpu:8 State=UNKNOWN<br>
</div>
<div>NodeName=node[6] CPUs=24 RealMemory=30000 Sockets=2
CoresPerSocket=6 ThreadsPerCore=2 Feature="pascal,p1000"
Gres=gpu:8 State=UNKNOWN<br>
</div>
<span></span><br>
<span>/etc/slurm/gres.conf</span><br>
<div>NodeName=node[1-3] Name=gpu File=/dev/nvidia[0-7]<br>
</div>
<div>NodeName=node[4-10] Name=gpu File=/dev/nvidia[0-4]</div>
<div><br>
</div>
<div><span>scontrol show node node1<br>
</span>
<div>NodeName=node1 Arch=x86_64 CoresPerSocket=10<br>
</div>
<div> CPUAlloc=0 CPUTot=40 CPULoad=1.75<br>
</div>
<div> AvailableFeatures=pascal,p4000<br>
</div>
<div> ActiveFeatures=pascal,p4000<br>
</div>
<div> Gres=(null)
<------------------------<br>
</div>
<div> NodeAddr=node1 NodeHostName=node1<br>
</div>
<div> OS=Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6
15:49:49 UTC 2019<br>
</div>
<div> RealMemory=48000 AllocMem=0 FreeMem=57465 Sockets=2
Boards=1<br>
</div>
<div> State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1
Owner=N/A MCS_label=N/A<br>
</div>
<div> Partitions=pharmacy<br>
</div>
<div> BootTime=2020-05-15T09:26:45
SlurmdStartTime=2020-05-15T16:35:13<br>
</div>
<div> CfgTRES=cpu=40,mem=48000M,billing=40<br>
</div>
<div> AllocTRES=<br>
</div>
<div> CapWatts=n/a<br>
</div>
<div> CurrentWatts=0 AveWatts=0<br>
</div>
<div> ExtSensorsJoules=n/s ExtSensorsWatts=0
ExtSensorsTemp=n/s<br>
</div>
<span></span></div>
<div><br>
</div>
<div><br>
</div>
<span></span><br>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de">www.itc.rwth-aachen.de</a>
</pre>
</body>
</html>