<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Try with SBATCH script and use “mpirun” executable without </span><span style="font-family:Consolas;color:black">”--mpi=pmi2”.</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>masber masber<br>
<b>Sent:</b> Tuesday, March 1, 2022 12:54 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com<br>
<b>Subject:</b> [EXT] [slurm-users] step creation temporarily disabled, retrying (Requested nodes are busy)<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div id="APLWarningText">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td width="100%" style="width:100.0%;background:#E0E0E0;padding:0in 0in 0in 0in">
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<b><span style="color:red">APL external email warning: </span></b><span style="color:black">Verify sender
<a href="mailto:slurm-users-bounces@lists.schedmd.com">slurm-users-bounces@lists.schedmd.com</a> before clicking links or attachments</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Dear slurm user community,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">I have a slurm cluster on centos7 installed through yum, I also have mpich installed.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">I can ssh into on of the nodes and run an mpi job:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black"># /usr/lib64/mpich/bin/mpirun --hosts nid001001-bae562bc0bd98e50ad5c03200efaf799d6e82469,nid001002-bae562bc0bd98e50ad5c03200efaf799d6e82469,nid001003-bae562bc0bd98e50ad5c03200efaf799d6e82469
/scratch/mpi-helloworld </span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">Warning: Permanently added 'nid001003-bae562bc0bd98e50ad5c03200efaf799d6e82469,10.233.88.25' (ECDSA) to the list of known hosts.</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">Hello world from processor nid001003-bae562bc0bd98e50ad5c03200efaf799d6e82469, rank 2 out of 3 processors</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">Hello world from processor nid001001-bae562bc0bd98e50ad5c03200efaf799d6e82469, rank 0 out of 3 processors</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">Hello world from processor nid001002-bae562bc0bd98e50ad5c03200efaf799d6e82469, rank 1 out of 3 processors</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">However I can't make it work through slurm, these are the logs form running the job:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black"># srun --mpi=pmi2 -N3 -vvv /usr/lib64/mpich/bin/mpirun /scratch/mpi-helloworld
</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: defined options</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: -------------------- --------------------</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: mpi : pmi2</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: nodes : 3</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: verbose : 3</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: -------------------- --------------------</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: end of defined options</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: propagating RLIMIT_CPU=18446744073709551615</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: propagating RLIMIT_FSIZE=18446744073709551615</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: propagating RLIMIT_DATA=18446744073709551615</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: propagating RLIMIT_STACK=8388608</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: propagating RLIMIT_CORE=18446744073709551615</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: propagating RLIMIT_RSS=18446744073709551615</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: propagating RLIMIT_NPROC=18446744073709551615</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: propagating RLIMIT_NOFILE=1048576</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: propagating RLIMIT_AS=18446744073709551615</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: propagating SLURM_PRIO_PROCESS=0</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: propagating UMASK=0022</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: srun PMI messages to port=33065</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: Entering slurm_allocation_msg_thr_create()</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: port from net_stream_listen is 44387</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: Entering _msg_thr_internal</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: auth/munge: init: Munge authentication plugin loaded</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: jobid 8: nodes(3):`nid001001-bae562bc0bd98e50ad5c03200efaf799d6e82469,nid001002-bae562bc0bd98e50ad5c03200efaf799d6e82469,nid001003-bae562bc0bd98e50ad5c03200efaf799d6e82469', cpu counts:
1(x3)</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: creating job with 3 tasks</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: requesting job 8, user 0, nodes 3 including ((null))</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: cpus 3, tasks 3, name mpirun, relative 65534</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: launch/slurm: launch_p_step_launch: CpuBindType=(null type)</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: Entering slurm_step_launch</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: mpi type = (null)</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: mpi/pmi2: p_mpi_hook_client_prelaunch: mpi/pmi2: client_prelaunch</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: mpi/pmi2: _get_proc_mapping: mpi/pmi2: processor mapping: (vector,(0,3,1))</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: mpi/pmi2: _setup_srun_socket: mpi/pmi2: srun pmi port: 37029</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: mpi/pmi2: _tree_listen_readable: mpi/pmi2: _tree_listen_readable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: mpi/pmi2: pmi2_start_agent: mpi/pmi2: started agent thread</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: Entering _msg_thr_create()</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: initialized stdio listening socket, port 41275</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: Started IO server thread (140538792195840)</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: Entering _launch_tasks</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: launching StepId=8.0 on host nid001001-bae562bc0bd98e50ad5c03200efaf799d6e82469, 1 tasks: 0</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_readable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_writable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_writable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: launching StepId=8.0 on host nid001002-bae562bc0bd98e50ad5c03200efaf799d6e82469, 1 tasks: 1</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: launching StepId=8.0 on host nid001003-bae562bc0bd98e50ad5c03200efaf799d6e82469, 1 tasks: 2</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: route/default: init: route default plugin loaded</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Tree head got back 0 looking for 3</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Tree head got back 1</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Tree head got back 2</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Tree head got back 3</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: launch returned msg_rc=0 err=0 type=8001</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: launch returned msg_rc=0 err=0 type=8001</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug: launch returned msg_rc=0 err=0 type=8001</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Activity on IO listening socket 17</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Entering io_init_msg_read_from_fd</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Leaving io_init_msg_read_from_fd</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Entering io_init_msg_validate</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Leaving io_init_msg_validate</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Validated IO connection from 10.233.88.26:33470, node rank 0, sd=18</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: eio_message_socket_accept: got message connection from 10.233.88.26:53410 19</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: received task launch</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: launch/slurm: _task_start: Node nid001001-bae562bc0bd98e50ad5c03200efaf799d6e82469, 1 tasks started</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_readable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_writable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_writable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Activity on IO listening socket 17</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Entering io_init_msg_read_from_fd</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Leaving io_init_msg_read_from_fd</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Entering io_init_msg_validate</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Leaving io_init_msg_validate</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Validated IO connection from 10.233.88.25:52764, node rank 2, sd=19</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Entering io_init_msg_read_from_fd</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Leaving io_init_msg_read_from_fd</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Entering io_init_msg_validate</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Leaving io_init_msg_validate</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Validated IO connection from 10.233.88.27:52768, node rank 1, sd=20</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: eio_message_socket_accept: got message connection from 10.233.88.25:47948 21</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: received task launch</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: launch/slurm: _task_start: Node nid001003-bae562bc0bd98e50ad5c03200efaf799d6e82469, 1 tasks started</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: eio_message_socket_accept: got message connection from 10.233.88.27:41996 21</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: received task launch</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: launch/slurm: _task_start: Node nid001002-bae562bc0bd98e50ad5c03200efaf799d6e82469, 1 tasks started</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_readable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_writable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_writable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_readable</span><span style="font-family:"Calibri",sans-serif;color:black">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_writable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Called _file_writable</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: debug2: Entering _file_write</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">srun: Job 8 step creation temporarily disabled, retrying (Requested nodes are busy)</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">The output clearly says the nodes are busy but they are not, actually I can run other jobs:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black"># squeue</span><span style="font-family:"Calibri",sans-serif;color:black">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black"> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black"># sinfo</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">PARTITION AVAIL TIMELIMIT NODES STATE NODELIST</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">workq* up infinite 3 idle nid001001-bae562bc0bd98e50ad5c03200efaf799d6e82469,nid001002-bae562bc0bd98e50ad5c03200efaf799d6e82469,nid001003-bae562bc0bd98e50ad5c03200efaf799d6e82469</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">[root@nid001001-bae562bc0bd98e50ad5c03200efaf799d6e82469 /]# srun -N3 hostname</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">nid001003-bae562bc0bd98e50ad5c03200efaf799d6e82469</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">nid001001-bae562bc0bd98e50ad5c03200efaf799d6e82469</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:Consolas;color:black">nid001002-bae562bc0bd98e50ad5c03200efaf799d6e82469</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Any idea what is stopping the mpi job from starting?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">thank you very much<o:p></o:p></span></p>
</div>
</div>
</body>
</html>