<div dir="ltr"><div><span style="font-family:monospace">I'm running clusters entirely in Google Cloud. I'm not sure I'm understanding the issue--do the nodes disappear from view entirely only when they fail to power up by
ResumeTimeout? Failures of this kind are happening in GCE when resources are momentarily unavailable, but the nodes are still there, only shown as DOWN. FWIW, I'm currently using 19.05.4-1.</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">I have a trigger on the controller to catch and return these nodes back to POWER_SAVE. The offset of 20s lets all moving parts to settle; in any case, Slurm batches trigger runs internally, on a 15s schedule IIRC, so it's not precise.
--flags=PERM makes the trigger permanent, so you need to install it once and for all:<br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">strigger --set --down --flags=PERM --offset=20 --program=$script</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">and the $script points to the full path (on the controller) of the following script. I'm copying the Log function and the _logname gymnastics from a file which is dot-sourced by the main program in my setup, as it's part of a larger set of scripts; it's more complex than it has to be for your case, but I did not want to introduce a bug by hastily paring it down. You'll do that if you want.<br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">
----8<--------8<--------8<---- </span></div><div><span style="font-family:monospace">#!/bin/bash<br><br>set -u<br><br># Tidy up name for logging: '.../slurm_resume.sh' => 'slurm-resume'<br>_logname=$(basename "$0")<br>_logname=${_logname%%.*}<br>_logname=${_logname//_/-}<br><br>Log() {<br> local level=$1; shift;<br> [[ $level == *.* ]] || level=daemon.$level # So we can use e.g. auth.notice.<br> logger -p $level -t $_logname -- "$@"<br>}<br><br>reason=recovery<br><br>for n; do<br> Log notice "Recovering failed node(s) '$n'"<br> scontrol update nodename="$n" reason="$reason" state=DRAIN &&<br> scontrol update nodename="$n" reason="$reason" state=POWER_DOWN ||<br> Log alert "The command 'scontrol update nodename=$n' failed." \<br> "Is scontrol on PATH?"<br>done<br><br>exit 0<br></span></div><div>
<div><span style="font-family:monospace">
----8<--------8<--------8<---- <br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">The sequence of DRAIN first then POWER_DOWN is a magic left over from v18; see if POWER_DOWN alone does the trick. Or don't, as long as it works :)<br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">Also make sure you have (some of) the following in slurm.conf, assuming EC2 provides DNS name resolution--GCE does.<br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"># Important for cloud: do not assume the nodes will retain their IP<br># addresses, and do not cache name-to-IP mapping.<br>CommunicationParameters=NoAddrCache<br>SlurmctldParameters=cloud_dns,idle_on_node_suspend<br>PrivateData=cloud # Always show cloud nodes.</span></div><div><span style="font-family:monospace">ReturnToService=2 # When a DOWN node boots, it becomes available.<br></span></div><div><span style="font-family:monospace"><br></span></div>
</div><div><span style="font-family:monospace">Hope this might help.</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"> -kkm</span><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 27, 2020 at 4:11 PM Carter, Allan <<a href="mailto:cartalla@amazon.com">cartalla@amazon.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_833335283928939998WordSection1">
<p class="MsoNormal">I’m setting up an EC2 SLURM cluster and when an instance doesn’t resume fast enough I get an error like:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">node c7-c5-24xl-464 not resumed by ResumeTimeout(600) - marking down and power_save<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I keep running into issues where my cloud nodes do not show up in sinfo and I can’t display their information with scontrol. This makes it difficult to know which of my CLOUD nodes are available for scheduling and which are down for some
reason and can’t be used. I haven’t figured out when slurm will show a cloud node and when it won’t and this make it pretty hard to manage the cluster.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Would I be better off just removing the CLOUD attribute on my EC2 nodes? What is the advantage of making them CLOUD nodes if it just make it more difficult to manage the cluster?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</blockquote></div>