<div dir="ltr"><div>Hi Chris,</div><div><br></div><div>Your issue sounds similar to a case I ran into once, where I could run jobs on a few nodes, but once it spanned more than a handful it would fail. In that particular case, we figured out that it was due to broadcast storm protection being enabled on the cluster switch. When the first node's slurmd started the job, it would send out a ton of ARP requests for each of the other nodes so it could contact them. That triggered the broadcast storm protection several ARPs in, so a subset of the nodes couldn't be reached and the job wouldn't start. Disabling broadcast storm protection in the switch solved the problems.</div><div><br></div><div>Hope it turns out to be this simple - if not, good luck!</div><div><br></div><div>Regards,</div><div><br></div><div>-Paul</div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div><div>--<br>Paul Kenyon<br>Advanced Clustering Technologies, Inc.<br></div>Main: 866-802-8222<br></div>Direct: 913-643-0306<br></div><a href="mailto:pkenyon@advancedclustering.com" target="_blank">pkenyon@advancedclustering.com</a><br></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 10, 2019 at 6:13 PM Chris Woelkers - NOAA Federal <<a href="mailto:chris.woelkers@noaa.gov">chris.woelkers@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks for the reply and the things to try. Here are the answers to your questions/tests in order:</div><div><br></div><div>- I tried mpiexec and the same issue occurred.</div><div>- While the job is listed as running I checked all the nodes. None of them have processes spawned. I have no idea on the hydra process.</div><div>- I have version 4.7 of the OFED stack installed on all nodes.</div><div>- Using openmpi with the hello world example you listed to gives output that seems to match what should normally be given. I upped the number of threads to 16, because 4 doesn't help much, and ran it again with four nodes of 4 threads each, and got the following which looks like good output.</div><div>Hello world from processor bearnode14, rank 4 out of 16 processors<br>Hello world from processor bearnode14, rank 5 out of 16 processors<br>Hello world from processor bearnode14, rank 6 out of 16 processors<br>Hello world from processor bearnode15, rank 10 out of 16 processors<br>Hello world from processor bearnode15, rank 8 out of 16 processors<br>Hello world from processor bearnode16, rank 13 out of 16 processors<br>Hello world from processor bearnode15, rank 11 out of 16 processors<br>Hello world from processor bearnode13, rank 3 out of 16 processors<br>Hello world from processor bearnode14, rank 7 out of 16 processors<br>Hello world from processor bearnode15, rank 9 out of 16 processors<br>Hello world from processor bearnode16, rank 12 out of 16 processors<br>Hello world from processor bearnode16, rank 14 out of 16 processors<br>Hello world from processor bearnode16, rank 15 out of 16 processors<br>Hello world from processor bearnode13, rank 1 out of 16 processors<br>Hello world from processor bearnode13, rank 0 out of 16 processors<br>Hello world from processor bearnode13, rank 2 out of 16 processors<br></div><div>- I have not tested our test model with openmpi as it was compiled with Intel compilers and expects Intel MPI. It might work but for now I will hold that for later. I did test the hello world again using the Intel modules instead of the openmpi modules and it still worked.</div><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div>Thanks,<br><br>Chris Woelkers</div><div>IT Specialist<br>National Oceanic and Atmospheric Agency</div><div>Great Lakes Environmental Research Laboratory<br>4840 S State Rd | Ann Arbor, MI 48108</div><div>734-741-2446</div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 10, 2019 at 4:36 PM Ree, Jan-Albert van <<a href="mailto:J.A.v.Ree@marin.nl" target="_blank">J.A.v.Ree@marin.nl</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:Calibri,Arial,Helvetica,sans-serif">
<p></p>
<p>We're running multiple clusters using Bright 8.x with Scientific Linux 7 (and have run Scientific Linux releases 5 and 6 with Bright 5.0 and higher in the past without issues on many different pieces of hardware) and never experienced this. But some things
to test :<br>
</p>
<p><br>
</p>
<p>- some implementations prefer mpiexec over mpirun , have you tried that instead ?<br>
</p>
<p>- if you log in to a node while a job is 'hanging' , do you see that on each node the right amount of processes are spawned ? Is the node list of all nodes involved in the job parsed to the hydra process on all nodes ?<br>
</p>
<p>- which version of the Mellanox OFED stack are you using ? One of our vendors recommended against OFED 4.6 due to issues, mostly related to IP over IB but still ; you might want to try 4.5 just to rule things out.<br>
</p>
<p>- what happens if you use openmpi (as supplied by Bright) together with a simple hello world example ? There's a good one at <a href="https://mpitutorial.com/tutorials/mpi-hello-world/" title="https://mpitutorial.com/tutorials/mpi-hello-world/
Ctrl+Click or tap to follow the link" target="_blank">https://mpitutorial.com/tutorials/mpi-hello-world/</a>
which I know to work fine with Bright supplied openmpi<br>
</p>
<p>- what happens if you test with openmpi and force it to use ethernet instead of infiniband ? See <a href="https://www.open-mpi.org/faq/?category=tcp" target="_blank">https://www.open-mpi.org/faq/?category=tcp</a> for info to force a specific interface with openmpi<br>
</p>
<p><br>
</p>
<p>I've just successfully tested the above hello-world example with the Bright supplied mvapich2/gcc/64/2.3b to compile the code , with the jobfile below to run it over 2 nodes, each 20 cores.<br>
</p>
<p><br>
</p>
<div>#!/bin/bash</div>
<div>#SBATCH -n 40<br>
</div>
<div>#SBATCH --exclusive</div>
<div>#SBATCH --partition=normal</div>
<div>#SBATCH --job-name=P80000.000_test</div>
<div>#SBATCH --time=2:00:00</div>
<div>#SBATCH --ntasks-per-node=20</div>
<div>#SBATCH --begin=now</div>
<div>#SBATCH --error=errors</div>
<div>#SBATCH --output=output</div>
<div>source /etc/profile.d/modules.sh</div>
<div>module load mvapich2/gcc/64/2.3b</div>
<div>mpiexec -n 40 ./hello<br>
</div>
<p><br>
</p>
<p><br>
</p>
<p><span style="font-size:12pt">Good luck!</span><br>
</p>
<p>--<br>
</p>
<p>Jan-Albert van Ree<br>
</p>
<p><br>
</p>
<div style="color:rgb(33,33,33)"></div>
<br>
<font style="font-family:Verdana;font-size:8pt;color:rgb(125,125,125);font-weight:bold">Jan-Albert</font><font style="font-family:Verdana;font-size:8pt;color:rgb(125,125,125);font-weight:bold">
</font><font style="font-family:Verdana;font-size:8pt;color:rgb(125,125,125);font-weight:bold">van Ree</font><font size="3" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt;color:rgb(125,125,125);font-weight:bold"> |
</font><font style="font-family:Verdana;font-size:8pt;color:rgb(125,125,125);font-weight:bold">Linux System Administrator</font><font style="font-family:Verdana;font-size:8pt;color:rgb(125,125,125);font-weight:bold"> |
</font><font style="font-family:Verdana;font-size:8pt;color:rgb(125,125,125);font-weight:bold">Digital Services</font><br>
<font style="font-family:Verdana;font-size:9pt;color:rgb(125,125,125);font-weight:bold">MARIN</font><font style="font-family:Verdana;font-size:8pt;color:rgb(125,125,125)"> | T
</font><font style="font-family:Verdana;font-size:8pt;color:rgb(125,125,125)">+31 317 49 35 48</font><font size="2" face="Segoe UI" color="#7d7d7d"></font><font style="font-family:Verdana;font-size:8pt"> |
</font><span style="font-family:Verdana;font-size:8pt"><a href="mailto:J.A.v.Ree@marin.nl" title="" style="font-family:Verdana;font-size:8pt" target="_blank"><span style="font-family:Verdana;font-size:8pt">J.A.v.Ree@marin.nl</span></a></span><font size="2" face="Segoe UI" color="#7d7d7d"></font><font style="font-family:Verdana;font-size:8pt"> |
</font><span style="font-family:Verdana;font-size:8pt"><a href="http://www.marin.nl" title="" style="font-family:Verdana;font-size:8pt" target="_blank"><span style="font-family:Verdana;font-size:8pt">www.marin.nl</span></a></span><br>
<br>
<a href="https://www.linkedin.com/company/marin" target="_blank"><img style="border: 0px solid;" src="cid:16ef24422a42bf835d71" alt="LinkedIn" width="16" height="16"></a> <a href="http://www.youtube.com/marinmultimedia" target="_blank"><img style="border: 0px solid;" src="cid:16ef24422a45a9232652" alt="YouTube" width="16" height="16"></a> <a href="https://twitter.com/MARIN_nieuws" target="_blank"><img style="border: 0px solid;" src="cid:16ef24422a45ff5be4d3" alt="Twitter" width="16" height="16"></a> <a href="https://www.facebook.com/marin.wageningen" target="_blank"><img style="border: 0px solid;" src="cid:16ef24422a4c6c94b874" alt="Facebook" width="16" height="16"></a><br>
<font style="font-family:Verdana;font-size:8pt" size="2" face="Verdana" color="#7d7d7d">MARIN news:
<a href="https://www.marin.nl/flare-holds-first-general-assembly-meeting-in-bremen-germany" title="" target="_blank">
FLARE holds first General Assembly Meeting in Bremen, Germany</a></font>
<p></p>
<p id="gmail-m_5113260076341410542gmail-m_6919286964285410808undefined"></p>
<p></p>
<div style="color:rgb(33,33,33)">
<hr style="display:inline-block;width:98%">
<div id="gmail-m_5113260076341410542gmail-m_6919286964285410808divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Chris Woelkers - NOAA Federal <<a href="mailto:chris.woelkers@noaa.gov" target="_blank">chris.woelkers@noaa.gov</a>><br>
<b>Sent:</b> Tuesday, December 10, 2019 20:49<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> [slurm-users] Multi-node job failure</font>
<div> </div>
</div>
<div>
<div dir="ltr">I have a 16 node HPC that is in the process of being upgraded from CentOS 6 to 7. All nodes are diskless and connected via 1Gbps Ethernet and FDR Infiniband. I am using Bright Cluster Management to manage it and their support has not found a
solution to this problem.
<div>For the most part the cluster is up and running with all nodes booting and able to communicate with each other via all interfaces on a basic level.</div>
<div>Test jobs, submitted via sbatch, are able to run on one node with no problem but will not run on multiple nodes. The jobs are using mpirun and mvapich2 is installed.</div>
<div>Any job trying to run on multiple nodes ends up timing out, as set via -t, with no output data written and no error messages in the slurm.err or slurm.out files. The job shows up in the squeue output and the nodes used show up as allocated in the sinfo
output.</div>
<div>
<div>
<div><br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>Thanks,<br>
<br>
Chris Woelkers</div>
<div>IT Specialist<br>
National Oceanic and Atmospheric Agency</div>
<div>Great Lakes Environmental Research Laboratory<br>
4840 S State Rd | Ann Arbor, MI 48108</div>
<div>734-741-2446</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<br>
<font style="background-color:rgb(255,255,255)">
<p align="center"><font style="background-color:rgb(255,255,255)"><b>Help us improve the spam filter. If this message contains SPAM, click
<a href="https://www.mailcontrol.com/sr/jhTwuwISfT_GX2PQPOmvUgItITKVa7z0k6_JDvhE_EooCnj2ZVOWPNLeAoygBsgADsU9DA6Go4T46EHnGWFGZQ==" target="_blank">
here</a> to report. Thank you, MARIN Support Group</b></font></p>
</font></div>
</div>
<u></u>
<p></p>
<u></u><u></u><u></u>
</div>
</blockquote></div>
</blockquote></div>