<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hello,</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">We have just finished an upgrade to slurm 18.08. My last task was to reset the slurmctld/slurmd timeouts to sensible values -- as they were set prior to the update. That is..</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"></p>
<div>SlurmctldTimeout = 60 sec</div>
<div>SlurmdTimeout = 300 sec</div>
<br>
<p></p>
<p style="margin-top:0;margin-bottom:0">With slurm <18.08 I've reconfigure the cluster many times before without an issues. Yesterday I found that this commands "pushed" most of the compute nodes into a "NODE_FAIL" state resulting in the loss of most running
jobs. </p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">I'm wondering if anyone has seen anything like this on their cluster, and if so what the solution was. I would be interested in hearing your experiences, please. Maybe I need to revise/increase the timeout values -- this
sort of issue is tricky to test on an active cluster</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Best regards,</p>
<p style="margin-top:0;margin-bottom:0">David</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
</div>
</body>
</html>