<div dir="ltr">Thanks so much! Indeed it was a mismatch between the actual and slurmd.conf SocketsPerBoard value. <div><div><div>Sushil</div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 11, 2022 at 11:25 AM Paul H. Hargrove <<a href="mailto:phhargrove@lbl.gov">phhargrove@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">I think Rob is "on the right track" here. Specifically, I don't think the error message means that "RESUME" is unrecognized as the name of a state. Rather the message means that a state transition from "INVAL" to "RESUME" is invalid. I can reproduce that message by trying to "RESUME" an "IDLE" node, but "RESUME" works fine for node which has been revently rebooted.<div><br></div></div>-Paul<div><br></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 11, 2022 at 8:14 AM Groner, Rob <<a href="mailto:rug262@psu.edu" target="_blank">rug262@psu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Have you checked the logs for slurmd and slurmctld? I seem to recall that the "invalid" state for a node meant that there was some discrepancy between what the node says or thinks it has (slurmd -C) and what the slurm.conf says it has. While there is that
discrepancy and the node is invalid, you can't just tell it to resume.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div id="m_-4219434529615126256m_6366104437507240842appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-4219434529615126256m_6366104437507240842divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Sushil Mishra <<a href="mailto:sushilbioinfo@gmail.com" target="_blank">sushilbioinfo@gmail.com</a>><br>
<b>Sent:</b> Tuesday, October 11, 2022 10:08 AM<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> [slurm-users] slurm_update error: Invalid node state specified</font>
<div> </div>
</div>
<div>
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="border:0px;display:table;width:100%;table-layout:fixed;float:none">
<tbody>
<tr>
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="padding:7px 2px;background-color:rgb(166,166,166)">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" color="#212121" style="width:100%;background-color:rgb(234,234,234);padding:7px 5px 7px 15px;font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;font-size:12px;font-weight:normal;color:rgb(33,33,33);text-align:left">
<div>You don't often get email from <a href="mailto:sushilbioinfo@gmail.com" target="_blank">sushilbioinfo@gmail.com</a>. <a href="https://aka.ms/LearnAboutSenderIdentification" target="_blank">
Learn why this is important</a></div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" color="#212121" style="width:75px;background-color:rgb(234,234,234);padding:7px 5px;font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;font-size:12px;font-weight:normal;color:rgb(33,33,33);text-align:left">
</td>
</tr>
</tbody>
</table>
<div>
<div dir="ltr">
<div>Dear all,</div>
<div><br>
</div>
<div>I am stuck with scontrol not recognizing the state keywords. I wonder if someone can point me to the possible cause of the error. I restarted slurmd a few times, and it didn't help. </div>
<div><br>
</div>
<div>[sushil@fucose ~]$ sinfo <br>
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br>
</div>
<div>LocalQ* up infinite 1 inval fucose<br>
</div>
<div><br>
</div>
[sushil@fucose ~]$ sinfo -R<br>
REASON USER TIMESTAMP NODELIST<br>
cg sushil 2022-10-10T18:11:27 fucose<br>
<br>
<div>[sushil@fucose ~]$ sudo scontrol update NodeName=fucose state=RESUME<br>
[sudo] password for sushil: <br>
slurm_update error: Invalid node state specified<br>
<br>
<div>[sushil@fucose ~]$ squeue <br>
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br>
<br>
<div>Best,<br>
</div>
<div>Sushil</div>
<div> </div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-family:"courier new",monospace">Paul H. Hargrove <<a href="mailto:PHHargrove@lbl.gov" target="_blank">PHHargrove@lbl.gov</a>><br>Pronouns: he, him, his<br></div><div style="font-family:"courier new",monospace">Computer Languages & Systems Software (CLaSS) Group</div><div style="font-family:"courier new",monospace">Computer Science Department</div><div style="font-family:"courier new",monospace">Lawrence Berkeley National Laboratory</div></div></div></div></div></div>
</blockquote></div>