<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>Are you starting the slurmd via 'slurmd -Z' on the dyanmic node?</div>
<div><br>
</div>
The next steps would be to check the slurmctld log from the master and slurmd log for the invalid node. Those should provide more insight into why the node is seen as invalid. If you can attach those we might be able to see the issue.
<div><br>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Regards,<br>
<br>
--<br>
Willy Markuske<br>
<br>
HPC Systems Engineer</div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
MS Data Science and Engineering<br>
SDSC - Research Data Services<br>
(619) 519-4435<br>
<div>wmarkuske@sdsc.edu</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Sep 1, 2023, at 03:12, Jan Andersen <jan@comind.io> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div>I am building a cluster exclusively with dynamic nodes, which all boot up over the network from the same system image (Debian 12); so far there is just one physical node, as well as a vm that I have used for the initial tests:<br>
<br>
# sinfo<br>
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br>
all* up infinite 1 inval gpu18c04d858b05<br>
all* up infinite 1 down* node080027aea419<br>
<br>
When I compare what the master node thinks of gpu18c04d858b05 with what the node itself reports, they seem to agree:<br>
<br>
On gpu18c04d858b05:<br>
<br>
root@gpu18c04d858b05:~# slurmd -C<br>
NodeName=gpu18c04d858b05 CPUs=16 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=64240<br>
UpTime=0-18:04:06<br>
<br>
And on the master:<br>
<br>
# scontrol show node gpu18c04d858b05<br>
NodeName=gpu18c04d858b05 Arch=x86_64 CoresPerSocket=8<br>
CPUAlloc=0 CPUEfctv=16 CPUTot=16 CPULoad=0.16<br>
AvailableFeatures=(null)<br>
ActiveFeatures=(null)<br>
Gres=gpu:geforce:1<br>
NodeAddr=192.168.50.68 NodeHostName=gpu18c04d858b05 Version=23.02.3<br>
OS=Linux 6.1.0-9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08)<br>
RealMemory=64240 AllocMem=0 FreeMem=63739 Sockets=1 Boards=1<br>
State=DOWN+DRAIN+DYNAMIC_NORM+INVALID_REG ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<br>
Partitions=all<br>
BootTime=2023-08-31T15:25:55 SlurmdStartTime=2023-08-31T15:26:20<br>
LastBusyTime=2023-08-31T10:24:01 ResumeAfterTime=None<br>
CfgTRES=cpu=16,mem=64240M,billing=16<br>
AllocTRES=<br>
CapWatts=n/a<br>
CurrentWatts=0 AveWatts=0<br>
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br>
Reason=hang [root@2023-08-31T16:38:27]<br>
<br>
I tried to fix it with:<br>
<br>
# scontrol update nodename=gpu18c04d858b05 state=down reason=hang<br>
# scontrol update nodename=gpu18c04d858b05 state=resume<br>
<br>
However, that made no difference; what is the next step in troubleshooting this issue?<br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>