<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>I believe that fragmentation only happens on routers when passing traffic from one subnet to another. Since this traffic was all on a single subnet there was no router involved to fragment the packets.
<br>
</p>
<p>Mike<br>
</p>
<div class="moz-cite-prefix">On 11/26/18 1:49 PM, Kenneth Roberts wrote:<br>
</div>
<blockquote type="cite" cite="mid:CY4PR11MB13995692DB51F0E2CA94EAE5DBD70@CY4PR11MB1399.namprd11.prod.outlook.com">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">D’oh!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The compute nodes had different MTU on the network interfaces than the master. Once all set to 1500, it works!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So ... any ideas why that was a problem? Maybe the interfaces had no fragmentation set and there were dropped packets?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for listening.<o:p></o:p></p>
<p class="MsoNormal">Ken<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com">
<slurm-users-bounces@lists.schedmd.com></a> <b>On Behalf Of </b>Kenneth Roberts<br>
<b>Sent:</b> Monday, November 26, 2018 9:38 AM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">
slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> Re: [slurm-users] Slurm / OpenHPC socket timeout errors<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I wasn’t looking close enough at the times in the log file.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">c2: [2018-11-26T10:<b><u>09:40</u></b>.963] debug3: in the service_connection<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T10:<b><u>10:00</u></b>.983] debug: slurm_recv_timeout at 0 of 9589, timeout<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T10:10:00.983] error: slurm_receive_msg_and_forward: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T10:10:00.994] error: service_connection: slurm_receive_msg: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T10:10:01.106] debug3: in the service_connection<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It looks like <span style="font-family:"Courier New"">slurm_recv_timeout</span> is attempting for 20 seconds and the call is just hitting continue without reading any data –<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> if ((rc = poll(&ufds, 1, timeleft)) <= 0) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">
<b>if ((errno == EINTR) || (errno == EAGAIN) || (rc == 0))<o:p></o:p></b></span></p>
<p class="MsoNormal"><b><span style="font-family:"Courier
New""> continue;<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> else {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> debug("%s at %d of %zu, poll error: %m",<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> __func__, recvlen, size);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> slurm_seterrno(<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> SLURM_COMMUNICATIONS_RECEIVE_ERROR);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> recvlen = SLURM_ERROR;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> goto done;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> }<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So poll is timing out after 20 seconds. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Back to finding out why ...<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" moz-do-not-send="true">slurm-users-bounces@lists.schedmd.com</a>>
<b>On Behalf Of </b>Kenneth Roberts<br>
<b>Sent:</b> Monday, November 26, 2018 8:35 AM<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" moz-do-not-send="true">
slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> Re: [slurm-users] Slurm / OpenHPC socket timeout errors<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is the debug log on a node (c2) when the job fails ....<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">c2: [2018-11-26T07:35:56.261] debug3: in the service_connection<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T07:36:16.281] <b>debug: slurm_recv_timeout at 0 of 9680, timeout</b><o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T07:36:16.282] error: slurm_receive_msg_and_forward: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T07:36:16.292] error: service_connection: slurm_receive_msg: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T07:36:16.334] debug3: in the service_connection<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">the line, debug: slurm_recv_timeout at 0 of 9680, timeout – looks like it times out before reading even the first byte of the message.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is the code snippet that generates that debug message:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">extern int slurm_recv_timeout(int fd, char *buffer, size_t size, uint32_t flags, int timeout )<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">while (recvlen < size) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> timeleft = timeout - _tot_wait(&tstart);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> if (timeleft <= 0) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">
<b>debug("%s at %d of %zu, timeout", __func__, recvlen, size);<o:p></o:p></b></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> slurm_seterrno(SLURM_PROTOCOL_SOCKET_IMPL_TIMEOUT);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> recvlen = SLURM_ERROR;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> goto done;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> }<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">recvlen</span> is 0 based on the log message, which might indicate it error’d on the first time through
<span style="font-family:"Courier New"">(timeleft <= 0)</span>. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">MessageTimeout=20 in our slurm.conf<o:p></o:p></p>
<p class="MsoNormal">But this code acts like it was passed timeout = 0??<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Up the call stack, <span style="font-family:"Courier New"">slurm_receive_msg_and_forward</span>, sets the timeout to the default:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">if (timeout <= 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> /* convert secs to msec */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> timeout = slurm_get_msg_timeout() * 1000;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""><o:p> </o:p></span></p>
<p class="MsoNormal">Unless<span style="font-family:"Courier New""> slurm_get_msg_timeout()
</span>is not working?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It may be that the slurm.conf values aren’t getting set correctly or used correctly, though I don’t see anything like permission errors reading slurm.conf ...<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Continuing the search ...<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" moz-do-not-send="true">slurm-users-bounces@lists.schedmd.com</a>>
<b>On Behalf Of </b>Kenneth Roberts<br>
<b>Sent:</b> Friday, November 23, 2018 4:15 PM<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" moz-do-not-send="true">
slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> [slurm-users] Slurm / OpenHPC socket timeout errors<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi –<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have the following on a new cluster with OpenHPC & Slurm built off the latest recipe and packages from OpenHPC (built this week).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">One master node and 4 compute nodes.<o:p></o:p></p>
<p class="MsoNormal">NodeName=c[1-4] Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 State=UNKNOWN<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">With simple test scripts, sbatch produces the following error when running across more than one node –<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The batch script is –<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#!/bin/bash<o:p></o:p></p>
<p class="MsoNormal">srun hostname<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">$ sbatch -N4 -n4 hostname.sh<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Out file --<o:p></o:p></p>
<p class="MsoNormal">c1<o:p></o:p></p>
<p class="MsoNormal">srun: error: Task launch for 151.0 failed on node c4: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: error: Task launch for 151.0 failed on node c3: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: error: Task launch for 151.0 failed on node c2: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: error: Application launch failed: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: Job step aborted: Waiting up to 32 seconds for job step to finish.<o:p></o:p></p>
<p class="MsoNormal">srun: error: Timed out waiting for job step to complete<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Searching on this discovers a lot of info about large jobs and starting a lot of tasks really quickly with some timeout and large cluster setting recommendations. BUT I’m running four tasks that are just ‘hostname’!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">AND If I just execute command line srun it works across the nodes<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">$ srun -N4 -n4 hostname<o:p></o:p></p>
<p class="MsoNormal">c1<o:p></o:p></p>
<p class="MsoNormal">c2<o:p></o:p></p>
<p class="MsoNormal">c3<o:p></o:p></p>
<p class="MsoNormal">c4<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Also, if I sbatch 20 tasks on one node max, it launches them fine. But 21 tasks (which tries to launch on two nodes) works on the c1 node (with 20 lines of output) and fails on the 21<sup>st</sup> task on c2 –<o:p></o:p></p>
<p class="MsoNormal">c1<o:p></o:p></p>
<p class="MsoNormal">c1<o:p></o:p></p>
<p class="MsoNormal">c1<o:p></o:p></p>
<p class="MsoNormal">... (17 more)<o:p></o:p></p>
<p class="MsoNormal">srun: error: Task launch for 156.0 failed on node c2: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: error: Application launch failed: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: Job step aborted: Waiting up to 32 seconds for job step to finish.<o:p></o:p></p>
<p class="MsoNormal">srun: error: Timed out waiting for job step to complete<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Maybe I completely don’t get sbatch options/params (I’m using defaults). BUT I’m attempting the simplest thing I could think of just to test this out.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Trying another approach to test, a script that uses a job array and runs 32 copies of a simple python script (so there’s no srun in the batch script) appears to work properly and utilizes all the nodes. But sbatch a script with srun in
the script gives the errors.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Really hoping this is something obvious that as a noob to OpenHPC and Slurm I’m getting wrong.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks in advance for any pointers or answers!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ken<o:p></o:p></p>
</div>
</blockquote>
</body>
</html>