<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Mike, <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m working through your suggestions. I tried <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>$ salloc –ntasks=20 --cpus-per-task=24 --verbose myscript.bash<o:p></o:p></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">but salloc says that the resources are not available:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">salloc: defined options<o:p></o:p></p>
<p class="MsoNormal">salloc: -------------------- --------------------<o:p></o:p></p>
<p class="MsoNormal">salloc: cpus-per-task : 24<o:p></o:p></p>
<p class="MsoNormal">salloc: ntasks : 20<o:p></o:p></p>
<p class="MsoNormal">salloc: verbose : 1<o:p></o:p></p>
<p class="MsoNormal">salloc: -------------------- --------------------<o:p></o:p></p>
<p class="MsoNormal">salloc: end of defined options<o:p></o:p></p>
<p class="MsoNormal">salloc: Linear node selection plugin loaded with argument 4<o:p></o:p></p>
<p class="MsoNormal">salloc: select/cons_res loaded with argument 4<o:p></o:p></p>
<p class="MsoNormal">salloc: Cray/Aries node selection plugin loaded<o:p></o:p></p>
<p class="MsoNormal">salloc: select/cons_tres loaded with argument 4<o:p></o:p></p>
<p class="MsoNormal">salloc: Granted job allocation 34299<o:p></o:p></p>
<p class="MsoNormal">srun: error: Unable to create step for job 34299: Requested node configuration is not available<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>$ scontrol show nodes /* oddly says that there is one core per socket. could our nodes be misconfigured? */<o:p></o:p></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">NodeName=n020 Arch=x86_64 CoresPerSocket=1 <o:p></o:p></p>
<p class="MsoNormal"> CPUAlloc=0 CPUTot=24 CPULoad=0.00<o:p></o:p></p>
<p class="MsoNormal"> AvailableFeatures=(null)<o:p></o:p></p>
<p class="MsoNormal"> ActiveFeatures=(null)<o:p></o:p></p>
<p class="MsoNormal"> Gres=(null)<o:p></o:p></p>
<p class="MsoNormal"> NodeAddr=n020 NodeHostName=n020 Version=20.02.3<o:p></o:p></p>
<p class="MsoNormal"> OS=Linux 4.18.0-305.7.1.el8_4.x86_64 #1 SMP Mon Jun 14 17:25:42 EDT 2021
<o:p></o:p></p>
<p class="MsoNormal"> RealMemory=1 AllocMem=0 FreeMem=126431 Sockets=24 Boards=1<o:p></o:p></p>
<p class="MsoNormal"> State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<o:p></o:p></p>
<p class="MsoNormal"> Partitions=normal,low,high <o:p></o:p></p>
<p class="MsoNormal"> BootTime=2021-11-18T08:43:44 SlurmdStartTime=2021-11-18T08:44:31<o:p></o:p></p>
<p class="MsoNormal"> CfgTRES=cpu=24,mem=1M,billing=24<o:p></o:p></p>
<p class="MsoNormal"> AllocTRES=<o:p></o:p></p>
<p class="MsoNormal"> CapWatts=n/a<o:p></o:p></p>
<p class="MsoNormal"> CurrentWatts=0 AveWatts=0<o:p></o:p></p>
<p class="MsoNormal"> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Renfro, Michael<br>
<b>Sent:</b> Friday, November 26, 2021 8:15 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [EXTERNAL] Re: [slurm-users] Reserving cores without immediately launching tasks on all of them<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The end of the MPICH section at [1] shows an example using salloc [2].
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Worst case, you should be able to use the output of “scontrol show hostnames” [3] and use that data to make mpiexec command parameters to run one rank per node, similar to what’s shown at the end of the synopsis section of [4].<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">[1] <a href="https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fmpi_guide.html%23mpich2&data=04%7C01%7Ckurt.e.mccall%40nasa.gov%7Ce6f6860268d745f9bde108d9b0e992ea%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637735339520105658%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=awAJ7NLxanv3WsH0h9O%2BA5zemiBPbGfQZ9PZfPRux%2Bk%3D&reserved=0">https://slurm.schedmd.com/mpi_guide.html#mpich2</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2] <a href="https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fsalloc.html&data=04%7C01%7Ckurt.e.mccall%40nasa.gov%7Ce6f6860268d745f9bde108d9b0e992ea%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637735339520115614%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DN2HFdOgRQD2PysTTpxwuyAvue%2FsNXR%2F2Is%2BDGiNoZ4%3D&reserved=0">https://slurm.schedmd.com/salloc.html</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">[3] <a href="https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fscontrol.html&data=04%7C01%7Ckurt.e.mccall%40nasa.gov%7Ce6f6860268d745f9bde108d9b0e992ea%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637735339520115614%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=FwouWVlL08O2kgUidxL9MLJQZJ7g5frTYccTlwmX6O4%3D&reserved=0">https://slurm.schedmd.com/scontrol.html</a><br>
[4] <a href="https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mpich.org%2Fstatic%2Fdocs%2Fv3.1%2Fwww1%2Fmpiexec.html&data=04%7C01%7Ckurt.e.mccall%40nasa.gov%7Ce6f6860268d745f9bde108d9b0e992ea%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637735339520125570%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=3sPjiFGuDEGGbgFMwj2jqUTyzMFVpsURrQyH9Z%2B0yWs%3D&reserved=0">https://www.mpich.org/static/docs/v3.1/www1/mpiexec.html</a><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<p class="MsoNormal">Mike Renfro, PhD / HPC Systems Administrator, Information Technology Services<br>
<a href="tel:931%20372-3601">931 372-3601</a> / Tennessee Tech University<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">On Nov 25, 2021, at 12:45 PM, Mccall, Kurt E. (MSFC-EV41) <<a href="mailto:kurt.e.mccall@nasa.gov">kurt.e.mccall@nasa.gov</a>> wrote:<o:p></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"> <o:p></o:p></p>
<p align="center" style="margin:0in;text-align:center;background:white"><b><span style="font-size:12.0pt;color:red;background:white">External Email Warning</span></b><o:p></o:p></p>
<p align="center" style="mso-margin-top-alt:0in;margin-right:12.0pt;margin-bottom:0in;margin-left:12.0pt;text-align:center;background:white">
<b><span style="font-size:12.0pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<div>
<p class="MsoNormal">I want to launch an MPICH job with sbatch with one task per node (each a manager), while also reserving a certain number of cores on each node for the managers to fill up with spawned workers (via MPI_Comm_spawn). I’d like to avoid using
–exclusive.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I tried the arguments –ntasks=20 –cpus-per-task=24, but it appears that 20 * 24 tasks will be launched. Is there a way to reserve cores without immediately launching tasks on them? Thanks for any help.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">sbatch: defined options<o:p></o:p></p>
<p class="MsoNormal">sbatch: -------------------- --------------------<o:p></o:p></p>
<p class="MsoNormal">sbatch: cpus-per-task : 24<o:p></o:p></p>
<p class="MsoNormal">sbatch: ignore-pbs : set<o:p></o:p></p>
<p class="MsoNormal">sbatch: ntasks : 20<o:p></o:p></p>
<p class="MsoNormal">sbatch: test-only : set<o:p></o:p></p>
<p class="MsoNormal">sbatch: verbose : 1<o:p></o:p></p>
<p class="MsoNormal">sbatch: -------------------- --------------------<o:p></o:p></p>
<p class="MsoNormal">sbatch: end of defined options<o:p></o:p></p>
<p class="MsoNormal">sbatch: Linear node selection plugin loaded with argument 4<o:p></o:p></p>
<p class="MsoNormal">sbatch: select/cons_res loaded with argument 4<o:p></o:p></p>
<p class="MsoNormal">sbatch: Cray/Aries node selection plugin loaded<o:p></o:p></p>
<p class="MsoNormal">sbatch: select/cons_tres loaded with argument 4<o:p></o:p></p>
<p class="MsoNormal">sbatch: Job 34274 to start at 2021-11-25T12:15:05 using <b><span style="color:red">480 processors</span></b><span style="color:red">
</span>on nodes n[001-020] in partition normal<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>