<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you very much for your comments. Oddly enough, I came up with the 3-partition model as well once I'd sent my email. So, your comments helped to confirm that I was thinking on the right lines. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Best regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
David</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Thomas M. Payerle <payerle@umd.edu><br>
<b>Sent:</b> 06 October 2020 18:50<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Controlling access to idle nodes</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>We use a scavenger partition, and although we do not have the policy you describe, it could be used in your case.</div>
<div><br>
</div>
<div>Assume you have 6 nodes (node-[0-5]) and two groups A and B.</div>
<div>Create partitions</div>
<div>partA = node-[0-2]</div>
<div>partB = node-[3-5]</div>
<div>all = node-[0-6]</div>
<div><br>
</div>
<div>Create QoSes normal and scavenger. <br>
</div>
<div>Allow normal QoS to preempt jobs with scavenger QoS<br>
</div>
<div><br>
</div>
<div>In sacctmgr, give members of group A access to use partA with normal QoS and group B access to use partB with normal QoS</div>
<div>Allow both A and B to use part all with scavenger QoS.</div>
<div><br>
</div>
<div>So members of A can launch jobs on partA with normal QoS (probably want to make that their default), and similarly member of B can launch jobs on partB with normal QoS.</div>
<div>But membes of A can also launch jobs on partB with scavenger QoS and vica versa. If the partB nodes used by A are needed by B, they will get preempted.</div>
<div><br>
</div>
<div>This is not automatic (users need to explicitly say they want to run jobs on the other half of the cluster), but that is probably reasonable because there are some jobs one does not wish to get preempted even if they have to wait a while in the queue to
ensure such.<br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Tue, Oct 6, 2020 at 11:12 AM David Baker <<a href="mailto:D.J.Baker@soton.ac.uk">D.J.Baker@soton.ac.uk</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hello,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I would appreciate your advice on how to deal with this situation in Slurm, please. If I have a set of nodes used by 2 groups, and normally each group would each have access to half the nodes. So, I could limit each group to have access to 3 nodes each, for
example. I am trying to devise a scheme that allows each group to make best use of the node always. In other words, each group could potentially use all the nodes (assuming they all free and the other group isn't using the nodes at all). </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I cannot set hard and soft limits in slurm, and so I'm not sure how to make the situation flexible. Ideally It would be good for each group to be able to use their allocation and then take advantage of any idle nodes via a scavenging mechanism. The other group
could then pre-empt the scavenger jobs and claim their nodes. I'm struggling with this since this seems like a two-way scavenger situation. </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Could anyone please help? I have, by the way, set up partition-based pre-emption in the cluster. This allows the general public to scavenge nodes owned by research groups. </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Best regards,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
David</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">Tom Payerle <br>
DIT-ACIGS/Mid-Atlantic Crossroads <a href="mailto:payerle@umd.edu" target="_blank">
payerle@umd.edu</a><br>
</div>
<div>5825 University Research Park (301) 405-6135<br>
</div>
<div dir="ltr">University of Maryland<br>
College Park, MD 20740-3831<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>