<div dir="ltr"><div>Ronan, as far as I can see this means that you cannot launch a job.</div><div><br></div><div>What state are the compute nodes in when you run sinfo?</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 17 July 2018 at 10:08, Buckley, Ronan <span dir="ltr"><<a href="mailto:Ronan.Buckley@dell.com" target="_blank">Ronan.Buckley@dell.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div class="m_-8760101946774615639WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Yes, srun just hangs. Commands like sinfo and squeue run fine.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I also have no slurm logs in /var/log ??<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> slurm-users [mailto:<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@<wbr>lists.schedmd.com</a>]
<b>On Behalf Of </b>John Hearns<br>
<b>Sent:</b> Tuesday, July 17, 2018 8:57 AM</span></p><div><div class="h5"><br>
<b>To:</b> Slurm User Community List<br>
<b>Subject:</b> Re: [slurm-users] 'srun hostname' hangs on the command line<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Ronan, sorry to ask but this is a bit unclear.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Are you unable to launch ANY sessions with srun?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">In which case you need to look at the logs to see why the job is not being scheduled.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Is it only the hostname command which fails?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I would guess very much you have already run an ssh into a node and run the hostname command manually.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On 17 July 2018 at 09:50, Buckley, Ronan <<a href="mailto:Ronan.Buckley@dell.com" target="_blank">Ronan.Buckley@dell.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="color:#1f497d">Yes I do.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users [mailto:<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@<wbr>lists.schedmd.com</a>]
<b>On Behalf Of </b>Williams, Gareth (IM&T, Clayton)<br>
<b>Sent:</b> Tuesday, July 17, 2018 12:33 AM<br>
<b>To:</b> Slurm User Community List<br>
<b>Subject:</b> Re: [slurm-users] 'srun hostname' hangs on the command line<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d" lang="EN-AU">Do you get the same problem as a non-root user?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d" lang="EN-AU"> </span><u></u><u></u></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users [<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">mailto:slurm-users-bounces@<wbr>lists.schedmd.com</a>]
<b>On Behalf Of </b>Buckley, Ronan<br>
<b>Sent:</b> Tuesday, 17 July 2018 12:53 AM<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> [slurm-users] 'srun hostname' hangs on the command line<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-AU"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">Hi All,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">Verbose mode doesn’t show much.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">I hashed out the hostnames.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">Any ideas/suggestions?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif"># srun hostname</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">^Csrun: interrupt (one more within 1 sec to abort)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: task 0: unknown</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">^Z</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">[1]+ Stopped srun hostname</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">#</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif"> </span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif"># srun -v hostname</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: defined options for program `srun'</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: --------------- ---------------------</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: user : `root'</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: uid : 0</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: gid : 0</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: cwd : /root</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: ntasks : 1 (default)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: nodes : 1 (default)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: jobid : 4294967294 (default)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: partition : default</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: profile : `NotSet'</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: job name : `(null)'</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: reservation : `(null)'</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: burst_buffer : `(null)'</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: wckey : `(null)'</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: cpu_freq_min : 4294967294</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: cpu_freq_max : 4294967294</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: cpu_freq_gov : 4294967294</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: switches : -1</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: wait-for-switches : -1</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: distribution : unknown</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: cpu_bind : default (0)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: mem_bind : default (0)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: verbose : 1</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: slurmd_debug : 0</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: immediate : false</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: label output : false</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: unbuffered IO : false</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: overcommit : false</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: threads : 60</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: checkpoint_dir : /var/slurm/checkpoint</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: wait : 0</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: nice : -2</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: account : (null)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: comment : (null)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: dependency : (null)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: exclusive : false</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: bcast : false</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: qos : (null)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: constraints :</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: geometry : (null)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: reboot : yes</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: rotate : no</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: preserve_env : false</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: network : (null)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: propagate : NONE</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: prolog : (null)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: epilog : (null)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: mail_type : NONE</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: mail_user : (null)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: task_prolog : (null)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: task_epilog : (null)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: multi_prog : no</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: sockets-per-node : -2</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: cores-per-socket : -2</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: threads-per-core : -2</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: ntasks-per-node : -2</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: ntasks-per-socket : -2</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: ntasks-per-core : -2</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: plane_size : 4294967294</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: core-spec : NA</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: power :</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: remote command : `hostname'</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: Waiting for nodes to boot (delay looping 450 times @ 0.100000 secs x index)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: Nodes ####### are ready for job</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: jobid 50871: nodes(1):`#######', cpu counts: 64(x1)</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: launching 50871.0 on host #######, 1 tasks: 0</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: route default plugin loaded</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: error: timeout waiting for task launch, started 0 of 1 tasks</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: Job step 50871.0 aborted before step completely launched.</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: Job step aborted: Waiting up to 32 seconds for job step to finish.</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">srun: error: Timed out waiting for job step to complete</span></i><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<i><span style="font-size:9.0pt;font-family:"Tahoma",sans-serif">#</span></i><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal">Rgds<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>