<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">If it's Ethernet problem there should be kernel message (dmesg) showing either link/carrier change or driver reset?<div><br></div><div>OP's problem could have been caused by excessive paging, check the -M flag of slurmd? <a href="https://slurm.schedmd.com/slurmd.html">https://slurm.schedmd.com/slurmd.html</a><br><div><br>Regards,</div><div>Angelos<br><div dir="ltr">(Sent from mobile, please pardon me for typos and cursoriness.)</div><div dir="ltr"><br><blockquote type="cite">2020/07/22 19:48、Riebs, Andy <andy.riebs@hpe.com>のメール:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Check for Ethernet problems. This happens often enough that I have the following definition in my .bashrc file to help track these
down:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">alias flaky_eth='su -c "ssh slurmctld-node grep responding /var/log/slurm/slurmctld.log"'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Andy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> slurm-users [mailto:slurm-users-bounces@lists.schedmd.com]
<b>On Behalf Of </b>???<br>
<b>Sent:</b> Tuesday, July 21, 2020 8:41 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com<br>
<b>Subject:</b> [slurm-users] lots of job failed due to node failure<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi,all<o:p></o:p></p>
<div>
<p class="MsoNormal">We run slurm 19.05 on a cluster about 1k nodes<span style="font-family:"Calibri",sans-serif">，</span>recently, we found lots of job failed due to node failure; check slumctld.log we found nodes are set to down stat then resumed quikly.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">some log info:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2020-07-20T00:21:23.306] error: Nodes j[1608,1802] not responding<br>
[2020-07-20T00:22:27.486] error: Nodes j1608 not responding, setting DOWN<br>
[2020-07-20T00:26:23.725] error: Nodes j1802 not responding<br>
[2020-07-20T00:26:27.323] error: Nodes j1802 not responding, setting DOWN<br>
[2020-07-20T00:26:46.602] Node j1608 now responding<br>
[2020-07-20T00:26:49.449] Node j1802 now responding<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Anyone hit this issue beforce ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Any suggestions will help.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards.<o:p></o:p></p>
</div>
</div>
</div>
</div></blockquote></div></div></body></html>