<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">This is what's in /var/log/slurmctld<br>
Invalid node state transition requested for node c01 from=DRAINING to=CANCEL_REBOOT<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">So it looks like, for version 18.08 at least, you have to first undrain, then cancel reboot:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">scontrol update NodeName="c01" State=undrain Reason="cancelling reboot"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">scontrol cancel_reboot c01<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Hanby, Mike" <mhanby@uab.edu><br>
<b>Date: </b>Friday, August 7, 2020 at 11:43 AM<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Cancel "reboot ASAP" for a node<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Howdy, (Slurm 18.08)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We have a bunch of node that we've updated to "scontrol reboot ASAP".</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We'd like to cancel a few of those. From the man page, it's suggested that either of the following should work, however both report the same error " slurm_update error: Invalid node state specified":<br>
<br>
<br>
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">scontrol cancel_reboot c01</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">or</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">scontrol Update NodeName=c01 State=CANCEL_REBOOT</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Here's the 'scontrol show node c01' info for reference:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">NodeName=c01 Arch=x86_64 CoresPerSocket=12</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> CPUAlloc=7 CPUTot=24 CPULoad=7.04</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> AvailableFeatures=(null)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ActiveFeatures=(null)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Gres=(null)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> NodeAddr=c0115 NodeHostName=c01 Version=18.08</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> OS=Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Mon Dec 2 08:31:54 EST 2019</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> RealMemory=191877 AllocMem=6536 FreeMem=176717 Sockets=2 Boards=1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> State=MIXED+DRAIN ThreadsPerCore=1 TmpDisk=887366 Weight=1 Owner=N/A MCS_label=N/A</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Partitions=interactive,short,long,medium,express</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> BootTime=2020-07-08T23:16:27 SlurmdStartTime=2020-07-08T23:32:05</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> CfgTRES=cpu=24,mem=191877M,billing=24</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> AllocTRES=cpu=7,mem=6536M</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> CapWatts=n/a</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Reason=Reboot ASAP [root@2020-08-06T10:29:22]</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Any thoughts as to how to cancel the reboot?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">---------------- </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Mike Hanby</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">mhanby @ uab.edu</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">Systems Analyst III - Enterprise</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">IT Research Computing Services</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">The University of Alabama at Birmingham</span><o:p></o:p></p>
</div>
</body>
</html>