<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">I’m not sure if this is what you are trying to accomplish but we do something similar using features and job constraints to get jobs to run on any set of processor types that are available. We have 4 generations of processors on one of
our clusters and our MPI jobs need them to run on all of one type. Each node has the appropriate processor feature assigned:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">NodeName=r1i[0-3]n[0-15] Feature=wes<o:p></o:p></p>
<p class="MsoNormal">NodeName=r2i[0-3]n[0-17] Feature=san<o:p></o:p></p>
<p class="MsoNormal">NodeName=r3i[0-3]n[0-17] Feature=has<o:p></o:p></p>
<p class="MsoNormal">NodeName=r4i[0-3]n[0-17] Feature=bro<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">All of those nodes are in the same partition. Jobs that need it then request any processor type, but all require all the same type like this:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#SBATCH -n 96<o:p></o:p></p>
<p class="MsoNormal">#SBATCH --constraint=[wes|san|has|bro]<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Read the sbatch man page on constraints to see the options and what those brackets mean. This works very well – users get whatever processor type is free. You might be able to adapt this to your situation.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Alexander John Mamach <alex.mamach@northwestern.edu><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Thursday, March 22, 2018 at 4:35 PM<br>
<b>To: </b>Slurm Mailing List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>[slurm-users] Automatically migrating jobs to different partitions?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><a name="_MailOriginalBody">Hi all, <o:p></o:p></a></p>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">I’ve been looking into a way to automatically migrate queued jobs from one partition to another. For example, if someone submits in partition A and must wait for resources, move their job request
to partition B and try to run, and if they must still wait, then try partition C, etc?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">Thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">Alex<o:p></o:p></span></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><span style="color:black">Systems Administrator<br>
<br>
Processing & Information Platform Services, Cyberinfrastructure<br>
Northwestern University Information Technology (NUIT)<br>
<br>
2020 Ridge Ave<br>
Evanston, IL 60208-4311<br>
<br>
O: (847) 491-2219<br>
M: (312) 887-1881<br>
</span></span><a href="http://www.it.northwestern.edu"><span style="mso-bookmark:_MailOriginalBody">www.it.northwestern.edu</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><span style="color:black"><o:p></o:p></span></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
</div>
</body>
</html>