<html>
<head>
<!-- Template generated by Exclaimer Mail Disclaimers on 08:54:08 Wednesday, 11 December 2019 -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p></p>
<p>OK so OpenMPI works fine. That means SLURM, OFED and hardware are fine.<br>
</p>
<p>Which <span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">mvapich2</span> package are you using, a home built one or one provided by Bright ?<br>
</p>
<p><span style="font-size: 12pt;"><br>
</span></p>
<p><span style="font-size: 12pt;">Regards,</span><br>
</p>
<p>--<br>
</p>
<p>Jan-Albert<br>
</p>
<p><br>
</p>
<div style="color: rgb(33, 33, 33);"></div>
<br>
<font style="font-family: Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">Jan-Albert</font><font style="font-family: Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">
</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">van Ree</font><font size="3" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;"> |
</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">Linux System Administrator</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;"> |
</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">Digital Services</font><br>
<font style="font-family:Verdana;font-size:9pt;color:#7D7D7D;font-weight:bold;">MARIN</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;"> | T
</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;">+31 317 49 35 48</font><font color="#7d7d7d" size="2" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt;"> |
</font><span style="font-family:Verdana;font-size:8pt;"><a href="mailto:J.A.v.Ree@marin.nl" title="" target="" style="font-family:Verdana;font-size:8pt;"><span style="font-family:Verdana; font-size:8pt;">J.A.v.Ree@marin.nl</span></a></span><font color="#7d7d7d" size="2" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt;"> |
</font><span style="font-family:Verdana;font-size:8pt;"><a href="http://www.marin.nl" title="" target="" style="font-family:Verdana;font-size:8pt;"><span style="font-family:Verdana; font-size:8pt;">www.marin.nl</span></a></span><br>
<br>
<a href="https://www.linkedin.com/company/marin" target=""><img width="16" height="16" style="border: 0px Solid ; " src="cid:image6cb08d.PNG@87dd545b.408bccb9" alt="LinkedIn"></a> <a href="http://www.youtube.com/marinmultimedia" target=""><img width="16" height="16" style="border: 0px Solid ; " src="cid:image460e3d.PNG@27ecd57f.4e85c50f" alt="YouTube"></a> <a href="https://twitter.com/MARIN_nieuws" target=""><img width="16" height="16" style="border: 0px Solid ; " src="cid:imagedb0bf8.PNG@16e9a314.41a787a3" alt="Twitter"></a> <a href="https://www.facebook.com/marin.wageningen" target=""><img width="16" height="16" style="border: 0px Solid ; " src="cid:image55891e.PNG@5fe85aa2.42b682f9" alt="Facebook"></a><br>
<font style="FONT-FAMILY: Verdana; FONT-SIZE: 8pt" color="#7d7d7d" size="2" face="Verdana">MARIN news:
<a href="https://www.marin.nl/flare-holds-first-general-assembly-meeting-in-bremen-germany" title="" target="" style="">
FLARE holds first General Assembly Meeting in Bremen, Germany</a></font>
<p></p>
<p id="undefined"></p>
<p></p>
<div style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Chris Woelkers - NOAA Federal <chris.woelkers@noaa.gov><br>
<b>Sent:</b> Wednesday, December 11, 2019 01:11<br>
<b>To:</b> Slurm User Community List<br>
<b>Subject:</b> Re: [slurm-users] Multi-node job failure</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Thanks for the reply and the things to try. Here are the answers to your questions/tests in order:</div>
<div><br>
</div>
<div>- I tried mpiexec and the same issue occurred.</div>
<div>- While the job is listed as running I checked all the nodes. None of them have processes spawned. I have no idea on the hydra process.</div>
<div>- I have version 4.7 of the OFED stack installed on all nodes.</div>
<div>- Using openmpi with the hello world example you listed to gives output that seems to match what should normally be given. I upped the number of threads to 16, because 4 doesn't help much, and ran it again with four nodes of 4 threads each, and got the
following which looks like good output.</div>
<div>Hello world from processor bearnode14, rank 4 out of 16 processors<br>
Hello world from processor bearnode14, rank 5 out of 16 processors<br>
Hello world from processor bearnode14, rank 6 out of 16 processors<br>
Hello world from processor bearnode15, rank 10 out of 16 processors<br>
Hello world from processor bearnode15, rank 8 out of 16 processors<br>
Hello world from processor bearnode16, rank 13 out of 16 processors<br>
Hello world from processor bearnode15, rank 11 out of 16 processors<br>
Hello world from processor bearnode13, rank 3 out of 16 processors<br>
Hello world from processor bearnode14, rank 7 out of 16 processors<br>
Hello world from processor bearnode15, rank 9 out of 16 processors<br>
Hello world from processor bearnode16, rank 12 out of 16 processors<br>
Hello world from processor bearnode16, rank 14 out of 16 processors<br>
Hello world from processor bearnode16, rank 15 out of 16 processors<br>
Hello world from processor bearnode13, rank 1 out of 16 processors<br>
Hello world from processor bearnode13, rank 0 out of 16 processors<br>
Hello world from processor bearnode13, rank 2 out of 16 processors<br>
</div>
<div>- I have not tested our test model with openmpi as it was compiled with Intel compilers and expects Intel MPI. It might work but for now I will hold that for later. I did test the hello world again using the Intel modules instead of the openmpi modules
and it still worked.</div>
<div><br clear="all">
<div>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>Thanks,<br>
<br>
Chris Woelkers<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255); font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(33,33,33)">
<div><font style="background-color:rgb(255,255,255)"></font></div>
</div>
<u></u>
<p></p>
<u></u><u></u><u></u></div>
</blockquote>
</div>
</div>
</div>
</EXCLAIMER:ORIGINAL_BODY_AFTER>
<p></p>
</EXCLAIMER:TEXT_ONLY></EXCLAIMER:AD></EXCLAIMER:AD>
</body>
</html>