<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello,</p>
<p>the solution we are currently using on our site is indeed a
separate partition; according to your example It'd look like this:</p>
<p>Partition Nodes #CPUs Available</p>
<p>cpu cpu-[01-03] 64<br>
</p>
<p>cpu_any gpu-[01-02] 32 (set with MaxCPUsPerNode=32)<br>
</p>
<p>gpu gpu-[01-02] 64</p>
<p>The trick now is to have CPU-only jobs with
<cores_per_node> <= 32 set "--partition=cpu,cpu_any" to
signal to the scheduler that they can run in either. <br>
Together with node weights you can then make sure that CPU-only
jobs will prefer to fill up the cpu-<xy> nodes first before
taking cores form the gpu-<xy> nodes by using the cpu_any
partition.</p>
<p>This also opens up the possibility for automatically changing
--partition=cpu to --partition=cpu,cpu_any if
<cores_per_node> <= 32 via job_submit.lua (a good example
to use as a starting template can be found e.g. here:
<a class="moz-txt-link-freetext" href="https://gist.github.com/mikerenfro/92d70562f9bb3f721ad1b221a1356de5">https://gist.github.com/mikerenfro/92d70562f9bb3f721ad1b221a1356de5</a>
- although I'd be careful and test this first, as I cannot say if
this is still applicable in unmodified form for current-day SLURM
versions)</p>
<p>Regards,<br>
René Sitt<br>
</p>
<div class="moz-cite-prefix">Am 21.06.22 um 16:11 schrieb Willy
Markuske:<br>
</div>
<blockquote type="cite"
cite="mid:ced828b6-3165-d604-69f8-a9420cea5d58@sdsc.edu">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Hello All,</p>
<p>I'm trying to clarify how the MaxCPUsPerNode can be configured.
I'm looking to enable my "cpu" partition to run on our GPU nodes
while ensuring there are always some cpus available for the
"gpu" partition. I know I can set the "cpu" partition to have a
MaxCPUsPerNode less than the number of available cpus on the GPU
nodes to do this. However, I don't also want to limit the number
of cpus available on a CPU node which doesn't seem possible
currently because only a single partition definition can be
included in slurm.conf.</p>
<p>The desired configuration would be something like this</p>
<p>Partition Nodes #CPUs Available</p>
<p>cpu cpu-[01-03] 64<br>
</p>
<p>cpu gpu-[01-02] 32<br>
</p>
<p>gpu gpu-[01-02] 64</p>
<p>It doesn't seem possible to set a partition to limit
MaxCPUsPerNode on a per node basis. Is the real solution a
different partition/QOS to handle this?</p>
<p>Regards,<br>
</p>
<div class="moz-signature">-- <br>
<title></title>
<table cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td width="150" valign="top" height="30" align="left">
<p style="font-size:14px;">Willy Markuske</p>
</td>
</tr>
<tr>
<td style="border-right: 1px solid #000;" align="left">
<p style="font-size:12px;">HPC Systems Engineer</p>
</td>
<td rowspan="3" width="180" valign="center" height="42"
align="center"><tt><img moz-do-not-send="false"
src="cid:part1.Lop1RZ1K.10n265tQ@hrz.uni-marburg.de"
alt="" class="" width="168" height="48"></tt> </td>
</tr>
<tr>
<td style="border-right: 1px solid #000;" align="left">
<p style="font-size:12px;">Research Data Services</p>
</td>
</tr>
<tr>
<td style="border-right: 1px solid #000;" align="left">
<p style="font-size:12px;">P: (619) 519-4435</p>
</td>
</tr>
</tbody>
</table>
<p> </p>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Dipl.-Chem. René Sitt
Hessisches Kompetenzzentrum für Hochleistungsrechnen
Philipps-Universität Marburg
Hans-Meerwein-Straße
35032 Marburg
Tel. +49 6421 28 23523
<a class="moz-txt-link-abbreviated" href="mailto:sittr@hrz.uni-marburg.de">sittr@hrz.uni-marburg.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.hkhlr.de">www.hkhlr.de</a></pre>
</body>
</html>