<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<div style="mso-element:para-border-div;border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><span style="font-size:11.0pt">Nodes are probably misconfigured in slurm.conf, yes. You can use the output of 'slurmd -C' on a compute node to get started on what your NodeName entry in slurm.conf should
be:<br>
<br>
<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
[root@node001 ~]# slurmd -C<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">NodeName=node001 CPUs=28 Boards=1 SocketsPerBoard=2 CoresPerSocket=14 ThreadsPerCore=1 RealMemory=64333<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">UpTime=161-22:35:13<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@node001 ~]# grep -i 'nodename=node\[001' /etc/slurm/slurm.conf<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">NodeName=node[001-022] CoresPerSocket=14 RealMemory=62000 Sockets=2 ThreadsPerCore=1 Weight=10201<o:p></o:p></span></p>
<div style="mso-element:para-border-div;border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Make sure that RealMemory in slurm.conf is no larger than what 'slurmd -C' reports. If I recall correctly, my slurm.conf settings are otherwise equivalent, but not word-for-word identical, with what 'slurmd
-C' reports (I just specified sockets instead of both boards and socketsperboard, for example).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall@nasa.gov><br>
<b>Date: </b>Friday, November 26, 2021 at 1:22 PM<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] Reserving cores without immediately launching tasks on all of them<o:p></o:p></span></p>
</div>
<p align="center" style="margin:0in;text-align:center;background:white"><b><span style="font-size:12.0pt;color:red;background:white">External Email Warning</span></b><o:p></o:p></p>
<p align="center" style="mso-margin-top-alt:0in;margin-right:12.0pt;margin-bottom:0in;margin-left:12.0pt;text-align:center;background:white">
<b><span style="font-size:12.0pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt">
<hr size="0" width="100%" align="center">
</span></div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Mike, <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m working through your suggestions. I tried
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt">$ salloc –ntasks=20 --cpus-per-task=24 --verbose myscript.bash</span></b><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">but salloc says that the resources are not available:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: defined options<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: -------------------- --------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: cpus-per-task : 24<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: ntasks : 20<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: verbose : 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: -------------------- --------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: end of defined options<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: Linear node selection plugin loaded with argument 4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: select/cons_res loaded with argument 4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: Cray/Aries node selection plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: select/cons_tres loaded with argument 4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">salloc: Granted job allocation 34299<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">srun: error: Unable to create step for job 34299: Requested node configuration is not available<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt">$ scontrol show nodes /* oddly says that there is one core per socket. could our nodes be misconfigured? */</span></b><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">NodeName=n020 Arch=x86_64 CoresPerSocket=1
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> CPUAlloc=0 CPUTot=24 CPULoad=0.00<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> AvailableFeatures=(null)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ActiveFeatures=(null)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Gres=(null)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> NodeAddr=n020 NodeHostName=n020 Version=20.02.3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> OS=Linux 4.18.0-305.7.1.el8_4.x86_64 #1 SMP Mon Jun 14 17:25:42 EDT 2021
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> RealMemory=1 AllocMem=0 FreeMem=126431 Sockets=24 Boards=1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Partitions=normal,low,high
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> BootTime=2021-11-18T08:43:44 SlurmdStartTime=2021-11-18T08:44:31<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> CfgTRES=cpu=24,mem=1M,billing=24<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> AllocTRES=<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> CapWatts=n/a<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> CurrentWatts=0 AveWatts=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Renfro, Michael<br>
<b>Sent:</b> Friday, November 26, 2021 8:15 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [EXTERNAL] Re: [slurm-users] Reserving cores without immediately launching tasks on all of them<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The end of the MPICH section at [1] shows an example using salloc [2].
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Worst case, you should be able to use the output of “scontrol show hostnames” [3] and use that data to make mpiexec command parameters to run one rank per node, similar to what’s shown at the end of the synopsis
section of [4].<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">[1] <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fmpi_guide.html%23mpich2&data=04%7C01%7Crenfro%40tntech.edu%7Cc9123a18a2934ad9e8a008d9b111b224%7C66fecaf83dc04d2cb8b8eff0ddea46f0%7C1%7C0%7C637735513496482886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rBK8XedubO1pmIa8dSHCCAnM713gruugH9pSamSvpX4%3D&reserved=0">https://slurm.schedmd.com/mpi_guide.html#mpich2</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">[2] <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fsalloc.html&data=04%7C01%7Crenfro%40tntech.edu%7Cc9123a18a2934ad9e8a008d9b111b224%7C66fecaf83dc04d2cb8b8eff0ddea46f0%7C1%7C0%7C637735513496492881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=oUelRHVc3ZrU0p9WMj9PiTNow9apx0Bc%2Fp0Kkg4aZic%3D&reserved=0">https://slurm.schedmd.com/salloc.html</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">[3] <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fscontrol.html&data=04%7C01%7Crenfro%40tntech.edu%7Cc9123a18a2934ad9e8a008d9b111b224%7C66fecaf83dc04d2cb8b8eff0ddea46f0%7C1%7C0%7C637735513496502874%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=q6VP%2FZGZil%2Fj2uweeCis1Wz5z94gNpdnBB9k2ojbf9U%3D&reserved=0">https://slurm.schedmd.com/scontrol.html</a><br>
[4] <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mpich.org%2Fstatic%2Fdocs%2Fv3.1%2Fwww1%2Fmpiexec.html&data=04%7C01%7Crenfro%40tntech.edu%7Cc9123a18a2934ad9e8a008d9b111b224%7C66fecaf83dc04d2cb8b8eff0ddea46f0%7C1%7C0%7C637735513496512872%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xJmBSMC4I5Fodzbpg%2FrchbD8Ml6v3l0ZjoGHL5Hl3KE%3D&reserved=0">https://www.mpich.org/static/docs/v3.1/www1/mpiexec.html</a><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">--<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Mike Renfro, PhD / HPC Systems Administrator, Information Technology Services<br>
<a href="tel:931%20372-3601">931 372-3601</a> / Tennessee Tech University<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">On Nov 25, 2021, at 12:45 PM, Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov">kurt.e.mccall@nasa.gov</a>> wrote:<o:p></o:p></span></p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p align="center" style="margin:0in;text-align:center;background:white"><b><span style="font-size:12.0pt;color:red;background:white">External Email Warning</span></b><o:p></o:p></p>
<p align="center" style="mso-margin-top-alt:0in;margin-right:12.0pt;margin-bottom:0in;margin-left:12.0pt;text-align:center;background:white">
<b><span style="font-size:12.0pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt">
<hr size="0" width="100%" align="center">
</span></div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">I want to launch an MPICH job with sbatch with one task per node (each a manager), while also reserving a certain number of cores on each node for the managers to fill up with spawned workers (via MPI_Comm_spawn).
I’d like to avoid using –exclusive.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I tried the arguments –ntasks=20 –cpus-per-task=24, but it appears that 20 * 24 tasks will be launched. Is there a way to reserve cores without immediately launching tasks on them? Thanks for any help.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: defined options<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: -------------------- --------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: cpus-per-task : 24<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: ignore-pbs : set<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: ntasks : 20<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: test-only : set<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: verbose : 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: -------------------- --------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: end of defined options<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: Linear node selection plugin loaded with argument 4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: select/cons_res loaded with argument 4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: Cray/Aries node selection plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: select/cons_tres loaded with argument 4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch: Job 34274 to start at 2021-11-25T12:15:05 using
<b><span style="color:red">480 processors</span></b><span style="color:red"> </span>
on nodes n[001-020] in partition normal<o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>