<div dir="auto"><div dir="auto"><br></div><div>Hi</div><div dir="auto"><br></div><div dir="auto">I've not used mpich for years but I think I see the problem. By asking for 24 CPUs per task and specifying 2 tasks you are asking slurm to allocate 48 CPUs per node.</div><div dir="auto"><br></div><div dir="auto">Your nodes have 24 CPUs in total so you don't have any nodes that can service this request</div><div dir="auto"><br></div><div dir="auto">Try asking for 24 tasks. I've only ever used CPU per task for hybrid MPI/openMP codes with 2 MPI tasks and 12 threads per task. </div><div dir="auto"><br></div><div dir="auto">Antony</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div dir="auto"><br></div><div dir="auto"><br></div><br><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Tue, 28 Dec 2021, 23:02 Mccall, Kurt E. (MSFC-EV41), <<a href="mailto:kurt.e.mccall@nasa.gov" target="_blank" rel="noreferrer">kurt.e.mccall@nasa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div>
<p class="MsoNormal">Hi,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">My MPICH jobs are being launched and the desired number of processes are created, but when one of those processes trys to spawn a new process using MPI_Comm_spawn(), that process just spins in the polling code deep within the MPICH library.
See the Slurm <span style="color:red">error message</span> below. This all works without problems on other clusters that have Torque as the process manager. We are using Slurm 20.02.3 on redhat 4.18.0, and MPICH 4.0b1.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">salloc: defined options<u></u><u></u></p>
<p class="MsoNormal">salloc: -------------------- --------------------<u></u><u></u></p>
<p class="MsoNormal">salloc: cpus-per-task : 24<u></u><u></u></p>
<p class="MsoNormal">salloc: ntasks : 2<u></u><u></u></p>
<p class="MsoNormal">salloc: verbose : 1<u></u><u></u></p>
<p class="MsoNormal">salloc: -------------------- --------------------<u></u><u></u></p>
<p class="MsoNormal">salloc: end of defined options<u></u><u></u></p>
<p class="MsoNormal">salloc: Linear node selection plugin loaded with argument 4<u></u><u></u></p>
<p class="MsoNormal">salloc: select/cons_res loaded with argument 4<u></u><u></u></p>
<p class="MsoNormal">salloc: Cray/Aries node selection plugin loaded<u></u><u></u></p>
<p class="MsoNormal">salloc: select/cons_tres loaded with argument 4<u></u><u></u></p>
<p class="MsoNormal">salloc: Granted job allocation 34330<u></u><u></u></p>
<p class="MsoNormal"><span style="color:red">srun: error: Unable to create step for job 34330: Requested node configuration is not availableta</span><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’m wondering if the salloc command I am using is correct. I intend for it to launch 2 processes, one per node, but reserve 24 cores on each node for the 2 launched processes to spawn new processes using MPI_Comm_spawn. Could the reservation
of all 24 cores make slurm or MPICH think that there are no more cores available?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><b>salloc –ntasks=2 –cpus-per-task=24 –verbose runscript.bash …<u></u><u></u></b></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I think that our cluster’s compute nodes are configured correctly –<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><b>$ scontrol show node=n001<u></u><u></u></b></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">NodeName=n001 Arch=x86_64 CoresPerSocket=6 <u></u><u></u></p>
<p class="MsoNormal"> CPUAlloc=0 CPUTot=24 CPULoad=0.00<u></u><u></u></p>
<p class="MsoNormal"> AvailableFeatures=(null)<u></u><u></u></p>
<p class="MsoNormal"> ActiveFeatures=(null)<u></u><u></u></p>
<p class="MsoNormal"> Gres=(null)<u></u><u></u></p>
<p class="MsoNormal"> NodeAddr=n001 NodeHostName=n001 Version=20.02.3<u></u><u></u></p>
<p class="MsoNormal"> OS=Linux 4.18.0-348.el8.x86_64 #1 SMP Mon Oct 4 12:17:22 EDT 2021
<u></u><u></u></p>
<p class="MsoNormal"> RealMemory=128351 AllocMem=0 FreeMem=126160 Sockets=4 Boards=1<u></u><u></u></p>
<p class="MsoNormal"> State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<u></u><u></u></p>
<p class="MsoNormal"> Partitions=normal,low,high <u></u><u></u></p>
<p class="MsoNormal"> BootTime=2021-12-21T14:25:05 SlurmdStartTime=2021-12-21T14:25:52<u></u><u></u></p>
<p class="MsoNormal"> CfgTRES=cpu=24,mem=128351M,billing=24<u></u><u></u></p>
<p class="MsoNormal"> AllocTRES=<u></u><u></u></p>
<p class="MsoNormal"> CapWatts=n/a<u></u><u></u></p>
<p class="MsoNormal"> CurrentWatts=0 AveWatts=0<u></u><u></u></p>
<p class="MsoNormal"> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks for any help.<u></u><u></u></p>
</div>
</div>
</blockquote></div></div></div>