<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Ken,<br>
<br>
if the switches between the hosts are correctly configured, this
SHOULD pose no problem, but we saw ourselves problems with Path MTU
Discovery. If the switches are wrongly configured, it will be a very
one-sided communication between the two hosts with the different
MTUs J<br>
<br>
It gets even more problematic if you have some UDP communication and
routers between several subnets. Our old cluster used IPoIB with a
MTU of 65535 (I think) in connected mode. We had several compute
nodes and one backup master in the Ethernet world, connected via
ib-ethernet gateways.<br>
Whenever our brandnew masters were online, it could take hours up to
one day, until all hosts were discovered as being online in LSF,
whereas it took minutes, when we used our old master from the
ethernet world. UDP does not do something like path discovery, it
just sends packets as big as it is allowed to, leading to crippled
packets on the remote host.<br>
<br>
Nonetheless, as far as me knows, slurm just communicates by TCP, so
there might have been a misconfigured switch between those hosts?<br>
<br>
<br>
Best<br>
Marcus<br>
<br>
<div class="moz-cite-prefix">On 11/26/2018 09:49 PM, Kenneth Roberts
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CY4PR11MB13995692DB51F0E2CA94EAE5DBD70@CY4PR11MB1399.namprd11.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">D’oh!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The compute nodes had different MTU on the
network interfaces than the master. Once all set to 1500, it
works!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So ... any ideas why that was a problem?
Maybe the interfaces had no fragmentation set and there were
dropped packets?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for listening.<o:p></o:p></p>
<p class="MsoNormal">Ken<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> <b>On
Behalf Of </b>Kenneth Roberts<br>
<b>Sent:</b> Monday, November 26, 2018 9:38 AM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> Re: [slurm-users] Slurm / OpenHPC socket
timeout errors<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I wasn’t looking close enough at the times
in the log file.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">c2: [2018-11-26T10:<b><u>09:40</u></b>.963]
debug3: in the service_connection<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T10:<b><u>10:00</u></b>.983]
debug: slurm_recv_timeout at 0 of 9589, timeout<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T10:10:00.983] error:
slurm_receive_msg_and_forward: Socket timed out on send/recv
operation<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T10:10:00.994] error:
service_connection: slurm_receive_msg: Socket timed out on
send/recv operation<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T10:10:01.106] debug3: in
the service_connection<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It looks like <span
style="font-family:"Courier New"">slurm_recv_timeout</span>
is attempting for 20 seconds and the call is just hitting
continue without reading any data –<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> if ((rc = poll(&ufds, 1,
timeleft)) <= 0) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> <b>if ((errno == EINTR) ||
(errno == EAGAIN) || (rc == 0))<o:p></o:p></b></span></p>
<p class="MsoNormal"><b><span style="font-family:"Courier
New""> continue;<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> else {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> debug("%s at %d of %zu, poll
error: %m",<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> __func__, recvlen,
size);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> slurm_seterrno(<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">
SLURM_COMMUNICATIONS_RECEIVE_ERROR);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> recvlen = SLURM_ERROR;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> goto done;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> }<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So poll is timing out after 20 seconds. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Back to finding out why ...<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <<a
href="mailto:slurm-users-bounces@lists.schedmd.com"
moz-do-not-send="true">slurm-users-bounces@lists.schedmd.com</a>>
<b>On Behalf Of </b>Kenneth Roberts<br>
<b>Sent:</b> Monday, November 26, 2018 8:35 AM<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true">slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> Re: [slurm-users] Slurm / OpenHPC socket
timeout errors<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is the debug log on a node (c2) when
the job fails ....<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">c2: [2018-11-26T07:35:56.261] debug3: in
the service_connection<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T07:36:16.281] <b>debug:
slurm_recv_timeout at 0 of 9680, timeout</b><o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T07:36:16.282] error:
slurm_receive_msg_and_forward: Socket timed out on send/recv
operation<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T07:36:16.292] error:
service_connection: slurm_receive_msg: Socket timed out on
send/recv operation<o:p></o:p></p>
<p class="MsoNormal">c2: [2018-11-26T07:36:16.334] debug3: in
the service_connection<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">the line, debug: slurm_recv_timeout at 0
of 9680, timeout – looks like it times out before reading even
the first byte of the message. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is the code snippet that generates
that debug message:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">extern int slurm_recv_timeout(int fd, char
*buffer, size_t size, uint32_t flags, int timeout )<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">while (recvlen < size) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> timeleft = timeout -
_tot_wait(&tstart);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> if (timeleft <= 0) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> <b>debug("%s at %d of %zu,
timeout", __func__, recvlen, size);<o:p></o:p></b></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">
slurm_seterrno(SLURM_PROTOCOL_SOCKET_IMPL_TIMEOUT);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> recvlen = SLURM_ERROR;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> goto done;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> }<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">recvlen</span> is 0 based on the log message,
which might indicate it error’d on the first time through <span
style="font-family:"Courier New"">(timeleft <=
0)</span>. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">MessageTimeout=20 in our slurm.conf<o:p></o:p></p>
<p class="MsoNormal">But this code acts like it was passed
timeout = 0??<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Up the call stack, <span
style="font-family:"Courier New"">slurm_receive_msg_and_forward</span>,
sets the timeout to the default:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New"">if (timeout <= 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> /* convert secs to msec */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""> timeout = slurm_get_msg_timeout() *
1000;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""><o:p> </o:p></span></p>
<p class="MsoNormal">Unless<span
style="font-family:"Courier New"">
slurm_get_msg_timeout() </span>is not working?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It may be that the slurm.conf values aren’t
getting set correctly or used correctly, though I don’t see
anything like permission errors reading slurm.conf ...<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Continuing the search ...<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier
New""><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <<a
href="mailto:slurm-users-bounces@lists.schedmd.com"
moz-do-not-send="true">slurm-users-bounces@lists.schedmd.com</a>>
<b>On Behalf Of </b>Kenneth Roberts<br>
<b>Sent:</b> Friday, November 23, 2018 4:15 PM<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true">slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> [slurm-users] Slurm / OpenHPC socket
timeout errors<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi –<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have the following on a new cluster with
OpenHPC & Slurm built off the latest recipe and packages
from OpenHPC (built this week).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">One master node and 4 compute nodes.<o:p></o:p></p>
<p class="MsoNormal">NodeName=c[1-4] Sockets=2 CoresPerSocket=10
ThreadsPerCore=1 State=UNKNOWN<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">With simple test scripts, sbatch produces
the following error when running across more than one node –<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The batch script is –<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#!/bin/bash<o:p></o:p></p>
<p class="MsoNormal">srun hostname<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">$ sbatch -N4 -n4 hostname.sh<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Out file --<o:p></o:p></p>
<p class="MsoNormal">c1<o:p></o:p></p>
<p class="MsoNormal">srun: error: Task launch for 151.0 failed
on node c4: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: error: Task launch for 151.0 failed
on node c3: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: error: Task launch for 151.0 failed
on node c2: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: error: Application launch failed:
Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: Job step aborted: Waiting up to 32
seconds for job step to finish.<o:p></o:p></p>
<p class="MsoNormal">srun: error: Timed out waiting for job step
to complete<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Searching on this discovers a lot of info
about large jobs and starting a lot of tasks really quickly
with some timeout and large cluster setting recommendations.
BUT I’m running four tasks that are just ‘hostname’!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">AND If I just execute command line srun
it works across the nodes<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">$ srun -N4 -n4 hostname<o:p></o:p></p>
<p class="MsoNormal">c1<o:p></o:p></p>
<p class="MsoNormal">c2<o:p></o:p></p>
<p class="MsoNormal">c3<o:p></o:p></p>
<p class="MsoNormal">c4<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Also, if I sbatch 20 tasks on one node max,
it launches them fine. But 21 tasks (which tries to launch on
two nodes) works on the c1 node (with 20 lines of output) and
fails on the 21<sup>st</sup> task on c2 –<o:p></o:p></p>
<p class="MsoNormal">c1<o:p></o:p></p>
<p class="MsoNormal">c1<o:p></o:p></p>
<p class="MsoNormal">c1<o:p></o:p></p>
<p class="MsoNormal">... (17 more)<o:p></o:p></p>
<p class="MsoNormal">srun: error: Task launch for 156.0 failed
on node c2: Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: error: Application launch failed:
Socket timed out on send/recv operation<o:p></o:p></p>
<p class="MsoNormal">srun: Job step aborted: Waiting up to 32
seconds for job step to finish.<o:p></o:p></p>
<p class="MsoNormal">srun: error: Timed out waiting for job step
to complete<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Maybe I completely don’t get sbatch
options/params (I’m using defaults). BUT I’m attempting the
simplest thing I could think of just to test this out.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Trying another approach to test, a script
that uses a job array and runs 32 copies of a simple python
script (so there’s no srun in the batch script) appears to
work properly and utilizes all the nodes. But sbatch a script
with srun in the script gives the errors.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Really hoping this is something obvious
that as a noob to OpenHPC and Slurm I’m getting wrong.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks in advance for any pointers or
answers!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ken<o:p></o:p></p>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de">www.itc.rwth-aachen.de</a>
</pre>
</body>
</html>