<html dir="ltr"><head></head><body style="text-align:left; direction:ltr;"><div><div>Dear Herbert,</div><div><br></div><div>In our installation we also had this problem.</div><div>Unfortunately we didn't find more elegant solution than change in Slurm code (and recompiling slurmctld).</div><div>Here is the patch we use to prevent DOWN nodes to be suspended:</div><div><br></div><div>diff --git a/src/slurmctld/power_save.c b/src/slurmctld/power_save.c</div><div>index 1f8d77c..752b404 100644</div><div>--- a/src/slurmctld/power_save.c</div><div>+++ b/src/slurmctld/power_save.c</div><div>@@ -368,7 +368,7 @@ static void _do_power_work(time_t now)</div><div> /* Suspend nodes as appropriate */</div><div> if ((susp_state == 0) &&</div><div> ((suspend_rate == 0) || (suspend_cnt < suspend_rate)) &&</div><div>- (IS_NODE_IDLE(node_ptr) || IS_NODE_DOWN(node_ptr)) &&</div><div>+ (IS_NODE_IDLE(node_ptr)) &&</div><div> (node_ptr->sus_job_cnt == 0) &&</div><div> (!IS_NODE_COMPLETING(node_ptr)) &&</div><div> (!IS_NODE_POWER_UP(node_ptr)) &&</div><div><br></div><div><br></div><div>Best regards,</div><div>Jacek Budzowski</div></div><div><br></div><div>W dniu pon, 24.08.2020 o godzinie 08∶52 +0000, użytkownik Steininger, Herbert napisał:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>Hi,</div><div><br></div><div>how can I prevent slurm, to suspend nodes, which I have set to down state for maintenance?</div><div>I know about "SuspendExcNodes", but this doesn't seem the right way, to roll out the slurm.conf every time this changes.</div><div>Is there a state that I can set so that the nodes doesn't get suspended?</div><div><br></div><div>It happened a few times that I was doing some stuff on a server and after our idle time (1h) slurm decided to suspend the node.</div><div><br></div><div>TIA,</div><div>Herbert</div><div><br></div></blockquote><div><pre>-- <br></pre><div>Jacek Budzowski</div><div>System administrator</div><div>ACC Cyfronet AGH</div><span class="-x-evo-signature" id="6367323f173cd27a095ba61c4d696041779381ab"></span></div></body></html>