<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Antonio,<br>
<br>
if you don't care, which nodes are used, but want to ensure that
only uniform nodes are used, you could also use the topology/tree
plugin, where you define on "switch" for every node type and then
use sbatch --switches=1.<br>
<br>
Best<br>
Marcus<br>
<br>
<div class="moz-cite-prefix">On 05/11/2018 05:49 PM, Antonio Lara
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:d01ea526-7caf-82e6-f665-b955e27cafc2@uam.es">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<p>Thank you all for your answers, I will research some more along
these lines!</p>
<p>Any other opinion is welcome<br>
</p>
<p>Regards,</p>
<p>Antonio<br>
</p>
<br>
<div class="moz-cite-prefix">El 11/05/18 a las 16:05, Vicker,
Darby (JSC-EG311) escribió:<br>
</div>
<blockquote type="cite"
cite="mid:41690EE7-7679-4A01-8F1E-D637D45CCED3@nasa.gov">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:2 0 5 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal">I’ll second that – we have a cluster with
4 generations of nodes. We assign a processor type feature
to each node and require the users to ask for at least one
of those features in their jobs via job_submit.lua – see the
code below. For a job that can run on any processor type,
you can use this:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#SBATCH --constraint=[wes|san|has|bro]<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">See the constraint section of “man
sbatch” for more details but this will constrain the job to
any processor type but all nodes of one type. It really
works great from a utilization standpoint – jobs will run on
the first processor type that is free. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:Courier">
local feature_count = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier"> if
job_desc ~= nil and job_desc.features ~= nil then<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">
if string.match(job_desc.features, "wes") then
feature_count=feature_count+1 end<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">
if string.match(job_desc.features, "san") then
feature_count=feature_count+1 end<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">
if string.match(job_desc.features, "has") then
feature_count=feature_count+1 end<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">
if string.match(job_desc.features, "bro") then
feature_count=feature_count+1 end<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier"> end<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier"> if
feature_count > 0 then<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">
slurm.log_info("Found %s valid cpu
features",feature_count)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier"> else<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">
slurm.log_user("Invalid features - aerolab policy requires
specifying one or more of wes,san,has,bro.")<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">
slurm.log_error("Found %s cpu features from
%s",feature_count,submit_uid)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">
-- See slurm/slurm_errno.h and src/common/slurm_errno.c<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">
-- for the list of error codes and messages.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">
return 2002<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier"> end<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Of course, the user can leave off the
square brackets and get any mix of processor types. We have
some codes that run fine across different processor types so
we allow this. Our group is small enough that we can easily
educate and police the users to do the right thing. But you
could add more logic to job_submit.lua to require the
brackets if you wanted to. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Darby<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:12.0pt;color:black">From: </span></b><span
style="font-size:12.0pt;color:black">slurm-users <a
class="moz-txt-link-rfc2396E"
href="mailto:slurm-users-bounces@lists.schedmd.com"
moz-do-not-send="true"><slurm-users-bounces@lists.schedmd.com></a>
on behalf of Hadrian Djohari <a
class="moz-txt-link-rfc2396E"
href="mailto:hxd58@case.edu" moz-do-not-send="true"><hxd58@case.edu></a><br>
<b>Reply-To: </b>Slurm User Community List <a
class="moz-txt-link-rfc2396E"
href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true"><slurm-users@lists.schedmd.com></a><br>
<b>Date: </b>Friday, May 11, 2018 at 5:22 AM<br>
<b>To: </b>Slurm User Community List <a
class="moz-txt-link-rfc2396E"
href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true"><slurm-users@lists.schedmd.com></a><br>
<b>Cc: </b><a class="moz-txt-link-rfc2396E"
href="mailto:slurm-users@schedmd.com"
moz-do-not-send="true">"slurm-users@schedmd.com"</a> <a
class="moz-txt-link-rfc2396E"
href="mailto:slurm-users@schedmd.com"
moz-do-not-send="true"><slurm-users@schedmd.com></a><br>
<b>Subject: </b>Re: [slurm-users] Distribute jobs in
similar nodes in the same partition<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">You can use node feature in defining
the node types in slurm.conf. <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Then when requesting for the job,
use -C <feature name> toy just use those node
type.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, May 11, 2018, 5:38 AM Antonio
Lara <<a href="mailto:antonio.lara@uam.es"
moz-do-not-send="true">antonio.lara@uam.es</a>>
wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello
everyone,<br>
<br>
Hopefully someone can help me with this, I cannot find
in the manual if <br>
this is even possible:<br>
<br>
I'm a system administrator, and the following question
is from the <br>
administrator point of view, not the user's point of
view:<br>
<br>
I work with a cluster which has a partition containing
many nodes. These <br>
nodes belong to "different categories". This is, we
bought at once <br>
several machines that are of the same type, and we did
this several <br>
times. So, for example, we have 10 machines of type A,
20 machines of <br>
type B and 15 machines of type C. Machines of type A are
more powerful <br>
than machines of type B, which are more powerful than
machines of type C.<br>
<br>
What I am trying to achieve is that Slurm "forces"
parallelized jobs to <br>
be allocated in machines of the same type, if possible.
That is, that <br>
there is some type of priority which tries to allocate
only machines of <br>
type A, or only machines of type B, or only of type C,
and only <br>
distribute jobs among machines of different types when
there are not <br>
enough nodes of the same type available.<br>
<br>
Does anyone know if this is possible? The idea behind
this is that <br>
slower machines are not delaying the calculations in
faster machines <br>
when a job is distributed among them, and all machines
work more or less <br>
at the same pace.<br>
<br>
I've been told that It is NOT an option to create
different partitions, <br>
each containing only one type of machine.<br>
<br>
Please, note that I'm not looking for a way to choose as
a user which <br>
nodes to use for a job, what I need is that slurm does
that, and decides <br>
what nodes to use, using similar nodes if available.<br>
<br>
The closest that I could find in the manual was using
consumable <br>
resources, but I think this is not what I need, there
are several <br>
examples, but they don't seem to fit with this.<br>
<br>
Thank you for your help!<br>
<br>
<o:p></o:p></p>
</blockquote>
</div>
</div>
</blockquote>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de">www.itc.rwth-aachen.de</a>
</pre>
</body>
</html>