<div dir="ltr">Partial progress. The scientist that developed the model took a look at the output and found that instead of one model run being ran in parallel srun had ran multiple instances of the model, one per thread, which for this test was 110 threads.<div>I have a feeling this just verified the same thing that the hello world test did.<br><div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Thanks,<br><br>Chris Woelkers</div><div>IT Specialist<br>National Oceanic and Atmospheric Agency</div><div>Great Lakes Environmental Research Laboratory<br>4840 S State Rd | Ann Arbor, MI 48108</div><div>734-741-2446</div></div></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 11, 2019 at 10:35 AM Chris Woelkers - NOAA Federal <<a href="mailto:chris.woelkers@noaa.gov">chris.woelkers@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I tried a simple thing of swapping out mpirun in the sbatch script for srun. Nothing more, nothing less.<div>The model is now working on at least two nodes, I will have to test again on more but this is progress.</div><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div>Thanks,<br><br>Chris Woelkers</div><div>IT Specialist<br>National Oceanic and Atmospheric Agency</div><div>Great Lakes Environmental Research Laboratory<br>4840 S State Rd | Ann Arbor, MI 48108</div><div>734-741-2446</div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 11, 2019 at 10:17 AM Chris Woelkers - NOAA Federal <<a href="mailto:chris.woelkers@noaa.gov" target="_blank">chris.woelkers@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks all for the ideas and possibilities. I will answer all in turn.</div><div><br></div>Paul: Neither of the switches in use, Ethernet and Infiniband, have any form of broadcast storm protection enabled.<div><br></div><div>Chris: I have passed on your question to the scientist that created the sbatch script. I will also look into other scripts that may make use of srun to find out if the same thing occurs.</div><div><br></div><div>Jan-Albert: The mvapich2 package is provided by Bright and loaded as a module by the script before mpirun is executed.</div><div><br></div><div>Zacarias: The drive that the data and script lives on is used is mounted on all the nodes at boot.<br><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div>Thanks,<br><br>Chris Woelkers</div><div>IT Specialist<br>National Oceanic and Atmospheric Agency</div><div>Great Lakes Environmental Research Laboratory<br>4840 S State Rd | Ann Arbor, MI 48108</div><div>734-741-2446</div></div></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 11, 2019 at 5:15 AM Zacarias Benta <<a href="mailto:zacarias@lip.pt" target="_blank">zacarias@lip.pt</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="text-align:left;direction:ltr"><div>I had a simmilar issue, please check if the home drive, or the place the data should be stored is mounted on the nodes.</div><div><br></div><div>On Tue, 2019-12-10 at 14:49 -0500, Chris Woelkers - NOAA Federal wrote:</div><blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:2px solid rgb(114,159,207);padding-left:1ex"><div dir="ltr">I have a 16 node HPC that is in the process of being upgraded from CentOS 6 to 7. All nodes are diskless and connected via 1Gbps Ethernet and FDR Infiniband. I am using Bright Cluster Management to manage it and their support has not found a solution to this problem.<div>For the most part the cluster is up and running with all nodes booting and able to communicate with each other via all interfaces on a basic level.</div><div>Test jobs, submitted via sbatch, are able to run on one node with no problem but will not run on multiple nodes. The jobs are using mpirun and mvapich2 is installed.</div><div>Any job trying to run on multiple nodes ends up timing out, as set via -t, with no output data written and no error messages in the slurm.err or slurm.out files. The job shows up in the squeue output and the nodes used show up as allocated in the sinfo output.</div><div><div><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div>Thanks,<br><br>Chris Woelkers</div><div>IT Specialist<br>National Oceanic and Atmospheric Agency</div><div>Great Lakes Environmental Research Laboratory<br>4840 S State Rd | Ann Arbor, MI 48108</div><div>734-741-2446</div></div></div></div></div></div></div></div></div></div>
</blockquote><div><span><pre>-- <br></pre><div>Cumprimentos / Best Regards,</div><div>Zacarias Benta</div><div>INCD @ LIP - UMinho</div><div> </div><div><br></div><div><img src="cid:16ef58510b5cb24ec821" width="156"> <br></div></span></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>