<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Run 'sinfo -R' to see if any of your nodes are out of the mix.</p>
<p>If so, resume them and see if things work.</p>
<p>Brian Andrus<br>
</p>
<div class="moz-cite-prefix">On 4/1/2021 1:53 AM, Steve Brasier
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CA+zeHV6PUuQGXO9kvdnGUADsydy8jvUecjuB_ocv8DdviSgiDw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Hi all, anyone have suggestions for debugging cloud
nodes not resuming? I've had this working before but I'm now
using "configless" mode so wondering if that's an issue.
<div><br>
</div>
<div>If I login as SlurmUser and run the ResumeProgram manually,
the specified node(s) boot, and if I log into them `sinfo`
works although it only shows the "static" nodes, not the newly
booted "cloud" nodes. So that at least shows the program
works, the image works, and new nodes can contact the
slurmctld.<br>
</div>
<div><br>
</div>
<div>However if I run a job which requires cloud nodes it
immediately goes Pending showing "Nodes required for job are
DOWN, DRAINED or reserved for jobs in higher priority
partitions". Looking at SlurmctldLogFile
with SlurmdDebug=debug5 I don't see any attempt to boot the
nodes at all :-(.</div>
<div><br>
</div>
<div>I can post slurm.conf if anyone wants to look but I think
the important parameters are probably that I've got:</div>
<div><br>
</div>
<div>SlurmctldParameters=enable_configless,idle_on_node_suspend,cloud_dns,power_save_interval=10,power_save_min_interval=0<br>
</div>
<div><br>
</div>
<div>That look right?</div>
<div><br>
</div>
<div>thanks for any suggestions!</div>
<div><br>
</div>
<div>Steve</div>
<div><br clear="all">
<div>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr"><a href="http://stackhpc.com/"
target="_blank" moz-do-not-send="true">http://stackhpc.com/</a></div>
<div>Please note I work Tuesday to Friday.</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</body>
</html>