<div dir="ltr"><div>Hi Ole,</div><div><br></div><div>Thanks for getting back to me.</div><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> the great presentation <br>
> from our own <br>
I presented that talk at SLUG'23 :-)<br></blockquote><div><br></div><div>Yes! That's why I wrote "from our own", but perhaps these are local slangs where I live (and English is my second language)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> 1) I'm not sure I fully understand ReconfigFlags=KeepPowerSaveSettings </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">As I understand it, the ReconfigFlags means that if you updated some <br>
settings using scontrol, they will be lost when slurmctld is reconfigured, <br>
and the settings from slurm.conf will be used in stead.<br></blockquote><div><br></div><div>I see, so that applies to the case in which I change the (power) state of the nodes by scontrol.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> 2) the PDF above says that the problem with nodes in down and drained <br>
> state is solved in 23.02 but that does not appear to be the case. Before <br>
> running my experiment, I had<br>
> <br>
> $ sinfo -R<br>
> REASON USER TIMESTAMP NODELIST<br>
> Not responding root 2023-09-13T13:14:50 node31<br>
> ECC memory errors root 2023-08-26T07:21:04 node27<br>
> <br>
> and after it became<br>
> <br>
> $ sinfo -R<br>
> REASON USER TIMESTAMP NODELIST<br>
> Not responding root 2023-09-13T13:14:50 node31<br>
> none Unknown Unknown node27<br>
<br>
Please use "sinfo -lR" so that we can see the node STATE.<br></blockquote><div><br></div><div><font face="monospace">$ sinfo -lR<br>Thu Oct 05 07:08:18 2023<br>REASON USER TIMESTAMP STATE NODELIST<br>Not responding root(0) 2023-09-13T13:14:50 down~ node31<br>none root(0) Unknown drain node27<br></font></div><div><br></div><div>Somewhat it has now remembered that the user was root (it now shows that even with plain sinfo -R)</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> so probably that's not solved? Anyway, that's a nuisance, not a deal breaker<br>
<br>
With my 23.02.5 the SuspendExcStates is working as documented :-)<br></blockquote><div><br></div><div>Okay so perhaps something happened between 23.02.3 and 23.02.5. I might need to sleuth in the ticketing system.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> 3) The whole thing does not appear to be working as I intended. My <br>
> understanding of the "exclude node" above should have meant that slurm <br>
> should never attempt to shut off more than all idle nodes in that <br>
> partition minus 2. Instead it shut them off all of them, and then tried to <br>
> turn them back on:<br>
> <br>
> $ sinfo | grep 512<br>
> compute512 up infinite 1 alloc# node15<br>
> compute512 up infinite 2 idle# node[14,32]<br>
> compute512 up infinite 3 down~ node[16-17,31]<br>
> compute512 up infinite 1 drain~ node27<br>
> compute512 up infinite 12 idle~ node[18-26,28-30]<br>
> compute512 up infinite 1 alloc node13<br>
<br>
I agree that 2 nodes from node[13-32] shouldn't be suspended, according to <br>
SuspendExcNodes in the slurm.conf manual. I haven't tested this feature.<br></blockquote><div><br></div><div>Good to know that an independent read of the manual is understood the same way as mine. If you don't use this feature, what do you do? Shutting off all idle nodes and leaving newly submitted jobs waiting for a boot? Or something else?</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> But again this is a minor nuisance which I can live with </blockquote><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> 4) Most importantly from the output above you may have noticed two nodes <br>
> (actually three by the time I ran the command below) that slurm deemed down<br>
> <br>> So I can confirm slurm invoked the script, but then waited for something <br>
> (what? starting slurmd?) which failed to occur and marked the node down. <br>
> When I removed the suspend time from the partition to end the experiment, <br>
> the other nodes went "magically" in production , without slurm calling my <br>
> poweron script. Of course the nodes were never powered off, but slurm <br>
> thought they were, so why it did not have the problem it id with the node <br>
> which instead intentionally tried to power on?<br>
<br>
IMHO, "pretending" to power down nodes defies the logic of the Slurm <br>
power_save plugin. </blockquote><div><br></div><div>And it is sure useless ;)</div><div>But I was using the suggestion from <a href="https://slurm.schedmd.com/power_save.html">https://slurm.schedmd.com/power_save.html</a> which says</div><div><br></div><div><span style="color:rgb(70,84,92);font-family:"Source Sans Pro",Helvetica,Arial,sans-serif;font-size:20px">You can also configure Slurm with programs that perform no action as </span><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-variant-alternates:inherit;font-stretch:inherit;font-size:20px;line-height:inherit;font-family:"Source Sans Pro",Helvetica,Arial,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline;color:rgb(70,84,92)">SuspendProgram</b><span style="color:rgb(70,84,92);font-family:"Source Sans Pro",Helvetica,Arial,sans-serif;font-size:20px"> and </span><b style="box-sizing:border-box;margin:0px;padding:0px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-variant-alternates:inherit;font-stretch:inherit;font-size:20px;line-height:inherit;font-family:"Source Sans Pro",Helvetica,Arial,sans-serif;font-kerning:inherit;font-feature-settings:inherit;vertical-align:baseline;color:rgb(70,84,92)">ResumeProgram</b><span style="color:rgb(70,84,92);font-family:"Source Sans Pro",Helvetica,Arial,sans-serif;font-size:20px"> to assess the potential impact of power saving mode before enabling it.</span><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Slurmctld expects suspended nodes to *really* power <br>
down (slurmd is stopped). When slurmctld resumes a suspended node, it <br>
expects slurmd to start up when the node is powered on. There is a <br>
ResumeTimeout parameter which I've set to about 15-30 minutes in case of <br>
delays due to BIOS updates and the like - the default of 60 seconds is WAY <br>
too small!<br></blockquote><div><br></div><div>Sure in fact I upped that to 4 minutes. Typically our nodes reboot in 3 minutes and will not update BIOS or OS automatically. Sometimes they become "hosed" and slower (firmware bug throttling CPU speed for no reason) but in that case better Slurm recognizes it and deems the node down. But in any case this is a moot point since the node is not going down....</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Have you tried to experiment with the IPMI based power down/up method <br>
explained in the above presentation? I'd appreciate independent testing <br>
of my scripts in <br>
<a href="https://github.com/OleHolmNielsen/Slurm_tools/tree/master/power_save" rel="noreferrer" target="_blank">https://github.com/OleHolmNielsen/Slurm_tools/tree/master/power_save</a> :-)<br></blockquote><div><br></div><div>I have not, because I already have my time-tested script from the vendor of the cluster. Those accept slurm's syntax as the node(s) to shut off/on or query so it's extremely convenient, and I use them all the time (interactively) </div><div><br></div><div>Thanks!</div></div></div>