<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="color: rgb(0, 0, 0); font-family: "Courier New",monospace; font-size: 12pt;">
<span>How do you have fabricnode2 defined in your gres.conf file and the slurm.conf file? Since the type of gpu changed, maybe the definition for it needs to be updated also.</span></div>
<div style="color: rgb(0, 0, 0); font-family: "Courier New",monospace; font-size: 12pt;">
<span><br>
</span></div>
<div style="color: rgb(0, 0, 0); font-family: "Courier New",monospace; font-size: 12pt;">
<span>Jeff</span></div>
<div>
<div id="appendonsend"></div>
<div style="color:rgb(0,0,0); font-family:"Courier New",monospace; font-size:12pt">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font color="#000000" face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Dean Schulze <dean.w.schulze@gmail.com><br>
<b>Sent:</b> Monday, April 27, 2020 11:47 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [slurm-users] Slurm not detecting gpu after swapping out gpu</font>
<div> </div>
</div>
<div>
<div dir="ltr">I replaced a Nvidia v100 with a t4. Now slurm thinks there is no gpu present:
<div><br>
</div>
<div>$ sudo scontrol show node fabricnode2<br>
NodeName=fabricnode2 Arch=x86_64 CoresPerSocket=6 <br>
CPUAlloc=0 CPUTot=12 CPULoad=0.02<br>
AvailableFeatures=(null)<br>
ActiveFeatures=(null)<br>
Gres=gpu:nvidia:1<br>
NodeAddr=fabricnode2 NodeHostName=fabricnode2 Version=19.05.4<br>
OS=Linux 5.3.0-46-generic #38~18.04.1-Ubuntu SMP Tue Mar 31 04:17:56 UTC 2020 <br>
RealMemory=7802 AllocMem=0 FreeMem=6828 Sockets=1 Boards=1<br>
State=IDLE+DRAIN ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<br>
Partitions=debug <br>
BootTime=2020-04-27T10:24:18 SlurmdStartTime=2020-04-27T10:39:53<br>
CfgTRES=cpu=12,mem=7802M,billing=12<br>
AllocTRES=<br>
CapWatts=n/a<br>
CurrentWatts=0 AveWatts=0<br>
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br>
Reason=gres/gpu count reported lower than configured (0 < 1) [root@2020-04-27T10:34:25]<br>
<br>
</div>
<div>The gpu is there and I can execute cuda binaries on it. nvidia-smi also shows it present. I've also rebooted the node, restarted slurmctld and run reconfigure.</div>
<div><br>
</div>
<div>How does slurm determine if a gpu is present, because it's getting it wrong?</div>
</div>
</div>
</div>
</body>
</html>