<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>We set up a partition that underlies all our hardware that is
preemptable by all higher priority partitions. That way it can
grab idle cycles while permitting higher priority jobs to run.
This also allows users to do:</p>
<p>#SBATCH -p primarypartition,requeuepartition</p>
<p>So that the scheduler will select which one their job will run on
more quickly. Then we rely on fairshare to adjudicate priority.</p>
<p>-Paul Edmon-<br>
</p>
<div class="moz-cite-prefix">On 10/6/2020 11:37 AM, Jason Simms
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAP7JYwfD8QsK0Akp-umJDTa5kFOo8eP3ksjxtnE-pt-=1+BMqg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Hello David,
<div><br>
</div>
<div>I'm still relatively new at Slurm, but one way we handle
this is that for users/groups who have "bought in" to the
cluster, we use a QOS to provide them preemptible access to
the set of resources provided by, e.g., a set number of nodes,
but not the nodes themselves. That is, in one example, two
researchers each have priority preemptible access to up to 52
cores in the cluster, but those cores can come from any
physical node. I set the priority of the QOS for each
researcher equal, such that they cannot preempt each other.</div>
<div><br>
</div>
<div>Admittedly, this works best and most simply in a situation
where your nodes are relatively homogeneous, as ours currently
are. I am trying to avoid a situation where a given physical
node is restricted to a specific researcher/group, as I want
all nodes, as much as possible, to be available to all users,
precisely so that idle cycles don't go to waste. It aligns
with the general philosophy that nodes are more like cattle
and less like pets, in my opinion, so I try to treat them like
a giant shared pool rather than multiple independent, gated
systems.</div>
<div><br>
</div>
<div>Anyway, I suspect other users here with more experience
might have a different, or better, approach and I look forward
to hearing their thoughts as well.</div>
<div><br>
</div>
<div>Warmest regards,</div>
<div>Jason</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Oct 6, 2020 at 11:12
AM David Baker <<a href="mailto:D.J.Baker@soton.ac.uk"
moz-do-not-send="true">D.J.Baker@soton.ac.uk</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Hello,</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">I
would appreciate your advice on how to deal with this
situation in Slurm, please. If I have a set of nodes used
by 2 groups, and normally each group would each have
access to half the nodes. So, I could limit each group to
have access to 3 nodes each, for example. I am trying to
devise a scheme that allows each group to make best use of
the node always. In other words, each group could
potentially use all the nodes (assuming they all free and
the other group isn't using the nodes at all). </div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">I
cannot set hard and soft limits in slurm, and so I'm not
sure how to make the situation flexible. Ideally It would
be good for each group to be able to use their allocation
and then take advantage of any idle nodes via a scavenging
mechanism. The other group could then pre-empt the
scavenger jobs and claim their nodes. I'm struggling with
this since this seems like a two-way scavenger situation. </div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Could
anyone please help? I have, by the way, set up
partition-based pre-emption in the cluster. This allows
the general public to scavenge nodes owned by research
groups. </div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Best
regards,</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">David</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div
style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><span
style="color:rgb(130,36,51)"><font
face="Century Gothic"><b>Jason L. Simms,
Ph.D., M.P.H.</b></font></span></div>
<div
style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font
face="Century Gothic"><span>Manager of
Research and High-Performance Computing</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Helvetica;font-size:14px;margin:0px"><font
face="Century Gothic"><span>XSEDE Campus
Champion<br>
</span><span style="color:gray">Lafayette
College<br>
Information Technology Services<br>
710 Sullivan Rd | Easton, PA 18042<br>
Office: 112 Skillman Library<br>
p: (610) 330-5632</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</body>
</html>