<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css">html { -webkit-text-size-adjust: none; -webkit-touch-callout: none;}</style>
</head>
<body style="margin-top: 8px;">
<div dir="ltr" style="font-size:11.0pt;font-family:Calibri,Tahoma,Arial,Helvetica,sans-serif;color:black;">
Hi, </div>
<div dir="ltr" style="font-size:11.0pt;font-family:Calibri,Tahoma,Arial,Helvetica,sans-serif;color:black;">
<br>
</div>
<div dir="ltr" style="font-size:11.0pt;font-family:Calibri,Tahoma,Arial,Helvetica,sans-serif;color:black;">
Thank you so much for your quick responses! </div>
<div dir="ltr" style="font-size:11.0pt;font-family:Calibri,Tahoma,Arial,Helvetica,sans-serif;color:black;">
It is much appreciated. </div>
<div dir="ltr" style="font-size:11.0pt;font-family:Calibri,Tahoma,Arial,Helvetica,sans-serif;color:black;">
I dont have access to the cluster until next week, but I’ll be sure to follow up on all of your suggestions and get back you next week.
</div>
<div dir="ltr" style="font-size:11.0pt;font-family:Calibri,Tahoma,Arial,Helvetica,sans-serif;color:black;">
<br>
</div>
<div dir="ltr" style="font-size:11.0pt;font-family:Calibri,Tahoma,Arial,Helvetica,sans-serif;color:black;">
Have a nice weekend! </div>
<div dir="ltr" style="font-size:11.0pt;font-family:Calibri,Tahoma,Arial,Helvetica,sans-serif;color:black;">
Best regards </div>
<div dir="ltr" style="font-size:11.0pt;font-family:Calibri,Tahoma,Arial,Helvetica,sans-serif;color:black;">
Palle <br>
<br>
<hr>
<b>From:</b> "slurm-users" <slurm-users-bounces@lists.schedmd.com> <br>
<b>Sent:</b> 12 juli 2019 17:37 <br>
<b>To:</b> "Slurm User Community List" <slurm-users@lists.schedmd.com> <br>
<b>Subject:</b> Re: [slurm-users] Running pyMPI on several nodes <br>
<br>
</div>
<div dir="ltr">
<div>Par, by 'poking around' Crhis means to use tools such as netstat and lsof. </div>
<div>Also I would look as ps -eaf --forest to make sure there are no 'orphaned' jusbs sitting on that compute node.
</div>
<div><br>
</div>
<div>Having said that though, I have a dim memory of a classic PBSPro error message which says something about a network connection,
</div>
<div>but really means that you cannot open a remote session on that compute server.
</div>
<div><br>
</div>
<div>As an aside, you have checked that your username exists on that compue server? getent passwd par
</div>
<div>Also that your home directory is mounted - or something substituting for your home directory?
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div class="gmail_attr" dir="ltr">On Fri, 12 Jul 2019 at 15:55, Chris Samuel < <a href="mailto:chris@csamuel.org">
chris@csamuel.org</a>> wrote: <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
On 12/7/19 7:39 am, Pär Lundö wrote: <br>
<br>
> Presumably, the first 8 tasks originates from the first node (in this <br>
> case the lxclient11), and the other node (lxclient10) response as <br>
> predicted. <br>
<br>
That looks right, it seems the other node has two processes fighting <br>
over the same socket and that's breaking Slurm there. <br>
<br>
> Is it neccessary to have passwordless ssh communication alongside the <br>
> munge authentication? <br>
<br>
No, srun doesn't need (or use) that at all. <br>
<br>
> In addition I checked the slurmctld-log from both the server and client <br>
> and found something (noted in bold): <br>
<br>
This is from the slurmd log on the client from the look of it. <br>
<br>
> *[2019-07-12T14:57:53.771][83.0] task_p_pre_launch: Using sched affinity <br>
> for tasks lurm.pmix.83.0: Address already in use[98]* <br>
> [2019-07-12T14:57:53.682][83.0] error: lxclient[0] /pmix.server.c:386 <br>
> [pmix_stepd_init] mpi/pmix: ERROR: pmixp_usock_create_srv <br>
> [2019-07-12T14:57:53.683][83.0] error: (null) [0] /mpi_pmix:156 <br>
> [p_mpi_hook_slurmstepd_prefork] mpi/pmix: ERROR: pmixp_stepd_init() failed <br>
<br>
That indicates that something else has grabbed the socket it wants and <br>
that's why the setup of the MPI ranks on the second node fails. <br>
<br>
You'll want to poke around there to see what's using it. <br>
<br>
Best of luck! <br>
Chris <br>
-- <br>
Chris Samuel : <a href="http://www.csamuel.org/" target="_blank" rel="noreferrer">
http://www.csamuel.org/</a> : Berkeley, CA, USA <br>
<br>
</blockquote>
</div>
</body>
</html>