<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am running SLURM 17.11 have a user who has a complicated workflow. The user wants 250 cores for 2 weeks to do some work semi-interactively. I'm not going to give the user a reservation to do this work,
because the whole point of having a scheduler is to minimize human intervention in job scheduling.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The code uses MPI (openmpi-1.8 with gcc-4.9.2). The process that I originally envisioned was to allocate an interactive job, a new shell gets spawned and then run `mpirun`, with SLURM dispatching the work
to allocation. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">i.e. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> [headnode01] $ salloc --ntasks=2 --nodes=2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> (SLURM grants allocation on node[01,02] and new shell spawns)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> [headnode01] $ mpirun -np 2 ./executable # SLURM dispatches work to node[01,02]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This doesn't work in the user's situation. Their workflow involves having master job that automatically spawns daughter MPI jobs (using 5 cores per job, for a total of 50 jobs) that get dispatched using `sbatch`.
It would be impractical to have manage 50 interactive shells.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I was imagining doing something like the following :
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">1. Get interactive allocation using `salloc`<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2. Submitting a batch job, that within it uses `srun --jobid=XXXX` to use the resources allocated in step 1.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I created a simple code, `tmp.c`, to test this process.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">`tmp.c`:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> #include <unistd.h><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> #include <stdio.h><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> #include <stdlib.h><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> #include <mpi.h><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> int main(int argc, char * argv[])<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> int taskID = -1;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> int ntasks = -1;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> int nSteps = 100; // Number of times to do the integration<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> int step = 0; // Current step<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> char hostname[250];<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> hostname[249]='\0';<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> /* MPI Initializations */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MPI_Init(&argc, &argv);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MPI_Comm_rank(MPI_COMM_WORLD, &taskID);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MPI_Comm_size(MPI_COMM_WORLD, &ntasks);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> gethostname(hostname, 1023);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> printf("Hello from Task %i on %s\n", taskID,hostname);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> /* Master Loop */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> for(step=0; step<nSteps; step++){<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> printf("%i : task %i hostname %s\n", step, taskID, hostname);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> usleep(1000000);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> fflush(stdout);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MPI_Barrier(MPI_COMM_WORLD); // Ensure every task completes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> MPI_Finalize();<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> return 0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I compile, I allocate resources and then I try to use `srun` to utilize those resources.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">i.e.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> [headnode01] $ mpicc tmp.c -o tmp<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> [headnode01] $ salloc --ntasks=3 --nodes=3
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> (SLURM grants allocation on node[14-16] and new shell spawns)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> [headnode01] $ srun --jobid=XXXX --ntasks=1 --pty ./tmp # Done from a different shell, not the new shell...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Hello World from Task 0 on node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 0 : task 0 hostname node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 1 : task 0 hostname node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 2 : task 0 hostname node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ok, this is expected. 1 MPI task running on 1 node with 1 core. If I do
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> [headnode01] $ srun --jobid=XXXX --ntasks=2 --pty ./tmp # Done from a different shell, not the new shell...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Hello World from Task 0 on node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 0 : task 0 hostname node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 1 : task 0 hostname node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 2 : task 0 hostname node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This is unexpected. I would expect task 0 and task 1 to be on node[14,15] because I have 3 cores/tasks allocated across 3 nodes. Instead, if I look at node[14,15] I see that both nodes have a process `tmp`
running, but I only catch the stdout from node14. Why is that?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If I try instead, excluding --pty<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> srun --jobid=2440814 --ntasks=2 --mpi=openmpi ./tmp<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Hello from Task 0 on node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 0 : task 0 hostname node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Hello from Task 0 on node15.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 0 : task 0 hostname node15.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 1 : task 0 hostname node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 1 : task 0 hostname node15.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This is also not what I want. I don't want two separate instances of `tmp` running on two separate nodes. I want the program `tmp` to utilize two cores on two different nodes. I'd instead expect the output
to be : <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Hello from Task 0 on node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 0 : task 0 hostname node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Hello from Task 1 on node15.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 0 : task 1 hostname node15.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 1 : task 0 hostname node14.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 1 : task 1 hostname node15.cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I can achieve the above expected output if I run
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sbatch --ntasks=2 --nodes=2 --wrap="mpirun -np 2 ./tmp"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">but I'd like to do this interactively.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">QUESTION: <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">How do I create an allocation and then utilize parts and pieces of that single allocation using `srun` with MPI processes? I'd like for an MPI process being run via `srun` to be able to utilize multiple cores
spread across multiple nodes. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">======================================<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">Ali Snedden, Ph.D.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">HPC Scientific Programmer<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">The High Performance Computing Facility<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">Nationwide Children’s Hospital Research Institute<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>