<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hello Rich:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">You could create partitions "bulk_a", "bulk_b", "bulk_c" (names are arbitrary) which map onto those three groups of nodes and have the intended resource limits set at partition level. Then make job_submit lua cause all jobs submitted to
"bulk" (or only the subset requesting a specific shared resource, or any subset you desire that job_submit.lua can detect) to also get submitted to the intended one or more of bulk_[abc]. I can imagine this meeting your need but am not certain it does.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Node features requested by jobs (keying off of them in lua filter, or adding them there) might help too.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="color:black">-- <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Paul Brunk, system administrator</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Georgia Advanced Resource Computing Center<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Enterprise IT Svcs, the University of Georgia<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">On 2/1/22, 5:45 AM, "slurm-users" <slurm-users-bounces@lists.schedmd.com> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal">[EXTERNAL SENDER - PROCEED CAUTIOUSLY]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hi,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I am wondering if this possible with slurm, I have an application where I want to create groups of nodes (group size would be between 1 and n servers) which have exclusive access to a shared resources and then on that group of nodes allow
a configurable amount of jobs to run.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">For example I could have:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">partition: bulk, containing:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">group1, max 4 jobs:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> - node1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> - node2<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> - node3<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> - node4<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">group 2, max 2 jobs:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> - node5<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">group 3, max 1 job: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> - node6<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> - node7<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> - node8<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> - node9<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Ideally the user could submit a job to a generic queue and I could set a configurable gres/license in the background for them and the jobs get placed in a free group or pend if it requires the exclusive resource.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I've taken a look at: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">1. Using the job submit lua plugin to look at the groups and if a group has available resources set a gres so the job is correctly placed.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">2. Licenses, but I can't see how to limit a license to a group of hosts without creating clusters. Can you limit licenses to specific nodes?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">3. On the scheduler, script building the node configuration and update the node gres and issue a 'scontrol reconfigure'<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Option 3 works, but isn't great. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So I would really like the be able to use a plugin to look at the current allocation and set the a gres/license/partition for the user in the background, is it possible for the job_submit lua plugin to access an external resources or the
license part of the slurm? As I could use that.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Or am I missing something or doing something very wrong. <o:p>
</o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks in advance for any assistance its much appreciated.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Rich Cardwell<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Snr IT Engineer<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:richc@graphcore.ai">richc@graphcore.ai</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">www.graphcore.ai <<a href="http://www.graphcore.ai%3e">http://www.graphcore.ai></a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">** We have updated our privacy policy, which contains important information about how we collect and process your personal data. To read the policy, please click here <<a href="http://www.graphcore.ai/privacy%3e">http://www.graphcore.ai/privacy></a>
**<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This email and its attachments are intended solely for the addressed recipients and may contain confidential or legally privileged information.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">If you are not the intended recipient you must not copy, distribute or disseminate this email in any way; to do so may be unlawful.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Any personal data/special category personal data herein are processed in accordance with UK data protection legislation.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">All associated feasible security measures are in place. Further details are available from the Privacy Notice on the website and/or from the Company.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Graphcore Limited (registered in England and Wales with registration number 10185006) is registered at 107 Cheapside, London, UK, EC2V 6DN.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">This message was scanned for viruses upon transmission. However Graphcore accepts no liability for any such transmission.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>