<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.airmailon, li.airmailon, div.airmailon
{mso-style-name:airmail_on;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Spencer,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you for your response!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It does appear that the memory allocation was the issue. When I specify --mem=1, I am able to queue jobs on a single node.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">That being said, I was under the impression that the DefMemPerCPU, DefMemPerNode (what sbatch claims to default to), etc. values defaulted to 0 which was interpreted as unlimited. I understood this to mean that the job/task, when not explicitly
defining a memory request, had unlimited access to the memory resource. I’m assuming that’s incorrect? Is this possibly related to the scheduler configuration I have defined (making cores AND memory consumable resources):<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">SchedulerType=sched/backfill<o:p></o:p></p>
<p class="MsoNormal">SelectType=select/cons_tres<o:p></o:p></p>
<p class="MsoNormal">SelectTypeParameters=CR_Core_Memory,CR_CORE_DEFAULT_DIST_BLOCK<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you again for the help!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">Jason Dana<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">JHUAPL<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">REDD/RA2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Lead Systems Administrator/Software Engineer<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><a href="mailto:jason.dana@jhuapl.edu"><span style="color:#954F72">jason.dana@jhuapl.edu</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">240-564-1045 (w)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Need Support from </span></b><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:red">REDD</span></b><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">? </span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">You
can enter a ticket using the new REDD Help Desk Portal (<a href="https://help.rcs.jhuapl.edu/"><span style="color:#954F72">https://help.rcs.jhuapl.edu</span></a>) if you have an active account or e-mail <a href="mailto:redd-help@outermail.jhuapl.edu"><span style="color:#954F72">redd-help@outermail.jhuapl.edu</span></a>.</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Spencer Bliven <spencer.bliven@psi.ch><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Tuesday, September 1, 2020 at 5:11 PM<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>[EXT] Re: [slurm-users] Question/Clarification: Batch array multiple tasks on nodes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<div id="APLWarningText">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%">
<tbody>
<tr>
<td width="100%" style="width:100.0%;background:#E0E0E0;padding:0in 0in 0in 0in">
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<b><span style="font-size:10.0pt;font-family:Helvetica;color:red">APL external email warning:
</span></b><span style="font-size:10.0pt;font-family:Helvetica;color:black">Verify sender slurm-users-bounces@lists.schedmd.com before clicking links or attachments</span><span style="font-size:10.0pt;font-family:Helvetica"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p style="margin-left:.5in"><span style="color:white"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:Helvetica">Jason,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:Helvetica">The array jobs are designed to behave like independent jobs (but are stored more efficiently internally to avoid straining the controller). So in principle slurm
could schedule them one per node or multiple per node. The --nodes and --ntasks parameters apply to individual jobs in the array; thus setting --nodes=1 would definitely force jobs to run on different nodes.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:Helvetica">The fact that they queue when forced to a single node is suspicious. Maybe you set up the partition as --exclusive? Or maybe jobs are requesting some other limited
resource (e.g. if DefMemPerCPU is set to all the memory) preventing slurm from scheduling them simultaneously. If you're struggling with the array syntax, try just submitting two jobs to the same node and checking that you can get them to run simultaneously.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:Helvetica">Best of luck,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:Helvetica">-Spencer<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="airmailon" style="margin-left:.5in">On 1 September 2020 at 18:50:30, Dana, Jason T. (<a href="mailto:jason.dana@jhuapl.edu">jason.dana@jhuapl.edu</a>) wrote:<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
Hello,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
I am new to Slurm and I am working on setting up a cluster. I am testing out running a batch execution using an array and am seeing only one task executed in the array per node. Even if I specify in the sbatch command that only one node should be used, it executes
a single task on each of the available nodes in the partition. I was under the impression that it would continue to execute tasks until the resources on the node or for the user were at their limit. Am I missing something or have I misinterpreted how sbatch
and/or the job scheduling should work?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
Here is one of the commands I have run:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
sbatch --array=0-15 --partition=htc-amd --wrap 'python3 -c "import time; print(\"working\"); time.sleep(5)"'<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
The htc-amd partition has 8 nodes and the results of this command are a single task being run on each node while the others are queued waiting for them to finish. As I mentioned before, if I specify --nodes=1 it will still execute a single task on every node
in the partition. The only way I have gotten it to use on a single node was to use --nodelist, which worked but only to execute a single task and queued the rest. I have also tried specifying --ntasks and --ntasks-per-node. It appears to reserve resources,
as I can cause it to hit the QOS core/cpu limit, but it does not affect the number of tasks executed on each node.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
Thank you for any help you can offer!<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="color:black">Jason</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>