<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Prentice,<br>
<br>
<br>
<div class="moz-cite-prefix">On 2/19/19 2:58 PM, Prentice Bisbal
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:a6295737-9021-a73a-9cf8-ed1bc535f556@pppl.gov">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>--ntasks-per-node is meant to be used in conjunction with
--nodes option. From <a class="moz-txt-link-freetext"
href="https://slurm.schedmd.com/sbatch.html"
moz-do-not-send="true">https://slurm.schedmd.com/sbatch.html</a>:</p>
<p> </p>
<blockquote type="cite">
<dl compact="compact">
<dt><b>--ntasks-per-node</b>=<<i>ntasks</i>></dt>
<dd> Request that <i>ntasks</i> be invoked on each node. If
used with the <b>--ntasks</b> option, the <b>--ntasks</b>
option will take precedence and the <b>--ntasks-per-node</b>
will be treated as a <i>maximum</i> count of tasks per
node. Meant to be used with the <b>--nodes</b> option...</dd>
</dl>
</blockquote>
</blockquote>
Yes, but used together with --ntasks would mean to use e.g. 48 tasks
at maximum per node. I don't see, where there lies the difference
regarding submission of the job. Even if the semantic (how or how
many cores will be scheduled onto which number of hosts) might be
incorrect, at least the syntax should be correct.<br>
<blockquote type="cite"
cite="mid:a6295737-9021-a73a-9cf8-ed1bc535f556@pppl.gov">
<blockquote type="cite">
<dl compact="compact">
</dl>
</blockquote>
<p>If you don't specify --ntasks, it defaults to --ntasks=1, as
Andreas said. <a class="moz-txt-link-freetext"
href="https://slurm.schedmd.com/sbatch.html"
moz-do-not-send="true">https://slurm.schedmd.com/sbatch.html</a>:
<br>
</p>
<p> </p>
<blockquote type="cite">
<dl compact="compact">
<dt><b>-n</b>, <b>--ntasks</b>=<<i>number</i>></dt>
<dd> sbatch does not launch tasks, it requests an allocation
of resources and submits a batch script. This option advises
the Slurm controller that job steps run within the
allocation will launch a maximum of <i>number</i> tasks and
to provide for sufficient resources. The default is one task
per node, but note that the <b>--cpus-per-task</b> option
will change this default. </dd>
</dl>
</blockquote>
<p>So the correct way to specify your job is either like this</p>
<p>--ntasks=48</p>
<p>or <br>
</p>
<p>--nodes=1 --ntasks-per-node=48</p>
<p>Specifying both --ntasks-per-node and --ntasks at the same time
is not correct. <br>
</p>
</blockquote>
funnily the result is the same:<br>
<br>
$> sbatch -N 1 --ntasks-per-node=48 --wrap hostname<br>
sbatch: error: CPU count per node can not be satisfied<br>
sbatch: error: Batch job submission failed: Requested node
configuration is not available<br>
<br>
whereas just using --ntasks=48 gets submitted and it gets scheduled
onto one host:<br>
<br>
$> sbatch --ntasks=48 --wrap hostname <br>
sbatch: [I] No output file given, set to: output_%j.txt<br>
sbatch: [I] No runtime limit given, set to: 15 minutes<br>
Submitted batch job 199784<br>
$> scontrol show job 199784 | egrep "NumNodes|TRES"<br>
NumNodes=1 NumCPUs=48 NumTasks=48 CPUs/Task=1 ReqB:S:C:T=0:0:*:*<br>
TRES=cpu=48,mem=182400M,node=1,billing=48<br>
<br>
To me, this still looks like a bug, not like the wrong usage of
submission parameters.<br>
<br>
Does no one else use nodes in this shared way? <br>
If nodes are shared, do you schedule by hardware threads or by
cores?<br>
If you schedule by cores, how did you implement this in slurm?<br>
<br>
<br>
Best<br>
Marcus<br>
<blockquote type="cite"
cite="mid:a6295737-9021-a73a-9cf8-ed1bc535f556@pppl.gov">
<p> </p>
<p><br>
</p>
<pre class="moz-signature" cols="72">Prentice </pre>
<div class="moz-cite-prefix">On 2/14/19 1:09 AM, Henkel, Andreas
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:58774C62-1104-4A76-8DC5-BE4BCF2F534D@uni-mainz.de">
<pre class="moz-quote-pre" wrap="">Hi Marcus,
What just came to my mind: if you don’t set —ntasks isn’t the default just 1? All examples I know using ntasks-per-node also set ntasks with ntasks >= ntasks-per-node.
Best,
Andreas
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Am 14.02.2019 um 06:33 schrieb Marcus Wagner <a class="moz-txt-link-rfc2396E" href="mailto:wagner@itc.rwth-aachen.de" moz-do-not-send="true"><wagner@itc.rwth-aachen.de></a>:
Hi all,
I have narrowed this down a little bit.
the really astonishing thing is, that if I use
--ntasks=48
I can submit the job, it will be scheduled onto one host:
NumNodes=1 NumCPUs=48 NumTasks=48 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=48,mem=182400M,node=1,billing=48
but as soon as I change --ntasks to --ntasks-per-node (which should be the same, as --ntasks=48 schedules onto one host), I get the error:
sbatch: error: CPU count per node can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available
Is there no one else, who observes this behaviour?
Any explanations?
Best
Marcus
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">On 2/13/19 1:48 PM, Marcus Wagner wrote:
Hi all,
I have a strange behaviour here.
We are using slurm 18.08.5-2 on CentOS 7.6.
Let me first describe our computenodes:
NodeName=ncm[0001-1032] CPUs=48 Sockets=4 CoresPerSocket=12 ThreadsPerCore=2 RealMemory=185000 Feature=skx8160,hostok,hpcwork Weight=10541 State=UNKNOWN
we have the following config set:
$>scontrol show config | grep -i select
SelectType = select/cons_res
SelectTypeParameters = CR_CORE_MEMORY,CR_ONE_TASK_PER_CORE
So, I have 48 cores on one node. According to the manpage of sbatch, I should be able to do the following:
#SBATCH --ntasks=48
#SBATCH --ntasks-per-node=48
But I get the following error:
sbatch: error: Batch job submission failed: Requested node configuration is not available
Has anyone an explanation for this?
Best
Marcus
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de" moz-do-not-send="true">wagner@itc.rwth-aachen.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de" moz-do-not-send="true">www.itc.rwth-aachen.de</a>
</pre>
</blockquote>
</blockquote>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de">www.itc.rwth-aachen.de</a>
</pre>
</body>
</html>