<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>
<blockquote type="cite">We're running nscd on all nodes, with an
extremely stable list of users/accounts, so I think we should be
good here.</blockquote>
</p>
<p>Don't bet on it. I've had issues in the past with nscd in similar
situations to this. There's a reason that daemon has a "paranoid"
option. <br>
</p>
<p>Hostname should be completely local and not use any nameservices
like DNS or nscd in the first place, though. But just to be sure,
what if you ran a command like 'srun ls -ld /etc' I just chose
/etc because every Unix/Linux system should have that directory
and it should be on the local system disk, and -d so that the
output will be only one line. An ls on any single file or dir
that's on all systems should work. or 'srun date' should be pretty
safe from 'outside forces'. <br>
</p>
<p>Prentice<br>
</p>
<pre class="moz-signature" cols="72">
</pre>
<div class="moz-cite-prefix">On 4/26/19 11:24 AM, Andy Riebs wrote:<br>
</div>
<blockquote type="cite"
cite="mid:f8d9d757-bb6e-c249-83d3-a069015bd68a@hpe.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Hi John,<br>
<br>
> It's a DNS problem, isn't it? Seriously though - how long
does srun hostname take for a single system?<br>
<br>
We're running nscd on all nodes, with an extremely stable list of
users/accounts, so I think we should be good here.<br>
<br>
"time srun hostname" reports on the order of 0.2 seconds, so at
least single node requests are handled expediently!<br>
<br>
Andy<br>
<div id="smartTemplate4-quoteHeader">
<hr> <b>From:</b> John Hearns <a class="moz-txt-link-rfc2396E"
href="mailto:hearnsj@googlemail.com" moz-do-not-send="true"><hearnsj@googlemail.com></a>
<br>
<b>Sent:</b> Friday, April 26, 2019 10:56AM <br>
<b>To:</b> Slurm User Community List <a
class="moz-txt-link-rfc2396E"
href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true"><slurm-users@lists.schedmd.com></a><br>
<b>Cc:</b> <br>
<b>Subject:</b> Re: [slurm-users] job startup timeouts? <br>
</div>
<div class="replaced-blockquote"
cite="mid:CAPqNE2UjDeK3K60_N9CX78GY3Xi7A_rsMxHiDXt8wmm4fokFLQ@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<div dir="ltr">
<div>It's a DNS problem, isn't it? Seriously though - how
long does srun hostname take for a single system?</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div class="gmail_attr" dir="ltr">On Fri, 26 Apr 2019 at
15:49, Douglas Jacobsen <<a
href="mailto:dmjacobsen@lbl.gov" moz-do-not-send="true">dmjacobsen@lbl.gov</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">We
have 12,000 nodes in our system, 9,600 of which are KNL. We
can<br>
start a parallel application within a few seconds in most
cases (when<br>
the machine is dedicated to this task), even at full scale.
So I<br>
don't think there is anything intrinsic to Slurm that would<br>
necessarily be limiting you, though we have seen cases in
the past<br>
where arbitrary task distribution has caused contoller
slow-down<br>
issues as the detailed scheme was parsed.<br>
<br>
Do you know if all the slurmstepd's are starting quickly on
the<br>
compute nodes? How is the OS/Slurm/executable delivered to
the node?<br>
----<br>
Doug Jacobsen, Ph.D.<br>
NERSC Computer Systems Engineer<br>
Acting Group Lead, Computational Systems Group<br>
National Energy Research Scientific Computing Center<br>
<a href="mailto:dmjacobsen@lbl.gov" target="_blank"
moz-do-not-send="true">dmjacobsen@lbl.gov</a><br>
<br>
------------- __o<br>
---------- _ '\<,_<br>
----------(_)/ (_)__________________________<br>
<br>
<br>
On Fri, Apr 26, 2019 at 7:40 AM Riebs, Andy <<a
href="mailto:andy.riebs@hpe.com" target="_blank"
moz-do-not-send="true">andy.riebs@hpe.com</a>> wrote:<br>
><br>
> Thanks for the quick response Doug!<br>
><br>
> Unfortunately, I can't be specific about the cluster
size, other than to say it's got more than a thousand nodes.<br>
><br>
> In a separate test that I had missed, even "srun
hostname" took 5 minutes to run. So there was no remote file
system or MPI involvement.<br>
><br>
> Andy<br>
><br>
> -----Original Message-----<br>
> From: slurm-users [mailto:<a
href="mailto:slurm-users-bounces@lists.schedmd.com"
target="_blank" moz-do-not-send="true">slurm-users-bounces@lists.schedmd.com</a>]
On Behalf Of Douglas Jacobsen<br>
> Sent: Friday, April 26, 2019 9:24 AM<br>
> To: Slurm User Community List <<a
href="mailto:slurm-users@lists.schedmd.com"
target="_blank" moz-do-not-send="true">slurm-users@lists.schedmd.com</a>><br>
> Subject: Re: [slurm-users] job startup timeouts?<br>
><br>
> How large is very large? Where is the executable being
started? In<br>
> the parallel filesystem/NFS? If that is the case you
may be able to<br>
> trim start times by using sbcast to transfer the
executable (and its<br>
> dependencies if dynamically linked) into a node-local
resource, such<br>
> as /tmp or /dev/shm depending on your local
configuration.<br>
> ----<br>
> Doug Jacobsen, Ph.D.<br>
> NERSC Computer Systems Engineer<br>
> Acting Group Lead, Computational Systems Group<br>
> National Energy Research Scientific Computing Center<br>
> <a href="mailto:dmjacobsen@lbl.gov" target="_blank"
moz-do-not-send="true">dmjacobsen@lbl.gov</a><br>
><br>
> ------------- __o<br>
> ---------- _ '\<,_<br>
> ----------(_)/ (_)__________________________<br>
><br>
><br>
> On Fri, Apr 26, 2019 at 5:34 AM Andy Riebs <<a
href="mailto:andy.riebs@hpe.com" target="_blank"
moz-do-not-send="true">andy.riebs@hpe.com</a>> wrote:<br>
> ><br>
> > Hi All,<br>
> ><br>
> > We've got a very large x86_64 cluster with lots of
cores on each node, and hyper-threading enabled. We're
running Slurm 18.08.7 with Open MPI 4.x on CentOS 7.6.<br>
> ><br>
> > We have a job that reports<br>
> ><br>
> > srun: error: timeout waiting for task launch,
started 0 of xxxxxx tasks<br>
> > srun: Job step 291963.0 aborted before step
completely launched.<br>
> ><br>
> > when we try to run it at large scale. We
anticipate that it could take as long as 15 minutes for the
job to launch, based on our experience with smaller numbers
of nodes.<br>
> ><br>
> > Is there a timeout setting that we're missing that
can be changed to accommodate a lengthy startup time like
this?<br>
> ><br>
> > Andy<br>
> ><br>
> > --<br>
> ><br>
> > Andy Riebs<br>
> > <a href="mailto:andy.riebs@hpe.com"
target="_blank" moz-do-not-send="true">andy.riebs@hpe.com</a><br>
> > Hewlett-Packard Enterprise<br>
> > High Performance Computing Software Engineering<br>
> > +1 404 648 9024<br>
> > My opinions are not necessarily those of HPE<br>
> > May the source be with you!<br>
><br>
<br>
</blockquote>
</div>
</div>
<br>
</blockquote>
</body>
</html>