<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Thank you for your reply and apologies for not reacting sooner I
have kept busy until now. I have attached our partition
definitions to this mail. <br>
</p>
<p>As for your second question MPI jobs aren't really a issue in our
cluster there are a few in between but not nearly enough to
explain up to 20 nodes at a time remaining idle for up to a day
when we have the backfill scheduler configured and jobs in the
short partition that would fit on the nodes that remain idle.</p>
<p>Kind regards,<br>
Erik Eisold <br>
</p>
<div class="moz-cite-prefix">On 14/08/2020 14:18, Renfro, Michael
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:D5E68FC8-228A-4250-905E-D3440B85D0B4@tntech.edu">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
We’ve run a similar setup since I moved to Slurm 3 years ago, with
no issues. Could you share partition definitions from your
slurm.conf?
<div><br>
</div>
<div>When you see a bunch of jobs pending, which ones have a
reason of “Resources� Those should be the next ones to run, and
ones with a reason of “Priority†are waiting for higher priority
jobs to start (including the ones marked “Resourcesâ€). The only
time I’ve seen nodes sit idle is when there’s an MPI job pending
with “Resourcesâ€, and if any smaller jobs started, it would
delay that job’s start.<br>
<br>
<div dir="ltr">
<div><span style="background-color: rgba(255, 255, 255, 0);">--</span></div>
<span style="background-color: rgba(255, 255, 255, 0);">Mike
Renfro, PhD Â / HPC Systems Administrator, Information
Technology Services<br>
<a href="tel:931%20372-3601" dir="ltr"
x-apple-data-detectors="true"
x-apple-data-detectors-type="telephone"
x-apple-data-detectors-result="0/1"
style="-webkit-text-decoration-color: rgba(0, 0, 0,
0.258824);" moz-do-not-send="true">931 372-3601</a>Â Â Â /
Tennessee Tech University</span></div>
<div dir="ltr"><br>
<blockquote type="cite">On Aug 14, 2020, at 4:20 AM, Erik
Eisold <a class="moz-txt-link-rfc2396E" href="mailto:eisold@pks.mpg.de"><eisold@pks.mpg.de></a> wrote:<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr"><span></span><br>
<span>Our node topology is a bit special where almost all
our nodes are in one</span><br>
<span>common partition a subset of all those nodes are then
in another</span><br>
<span>partition and this repeats once more the only
difference between the</span><br>
<span>partitions except the nodes in it are the maximum run
time. The reason I</span><br>
<span>originally set it up this way was to ensure that users
with shorter jobs</span><br>
<span>had a quicker response time and the whole cluster
wouldn't be clogged up</span><br>
<span>with long running jobs for days on end this and I was
new to the whole</span><br>
<span>cluster setup and Slurm itself. I have attached a
rough visualization of</span><br>
<span>this setup to this mail. There are 2 more totally
separate partitions</span><br>
<span>that are not in this image.</span><br>
<span></span><br>
<span>My idea for a solution would be to move all nodes to
one common</span><br>
<span>partition and using partition QOS to implement time
and resource</span><br>
<span>restrictions because I think the scheduler is not
really meant to handle</span><br>
<span>the type of setup we choose in the beginning.</span><br>
</div>
</blockquote>
</div>
</blockquote>
</body>
</html>