<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Source Code Pro";
panose-1:2 11 5 9 3 4 3 2 2 4;}
@font-face
{font-family:"Helvetica Neue";
panose-1:2 0 5 3 0 0 0 2 0 4;}
@font-face
{font-family:Times;
panose-1:0 0 5 0 0 0 0 2 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1540044294;
mso-list-type:hybrid;
mso-list-template-ids:-1075422696 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">If you haven't already seen it there's an example Lua script from SchedMD at [1], and I've got a copy of our local script at [2]. Otherwise, in the order you asked:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">That seems reasonable, but our script just checks if there's a gres at all. I don't *think* any gres other than gres=gpu would let the job run, since our GPU nodes only have Gres=gpu:2
entries. Same thing for asking for more GPUs than are in the node: if someone asked for gres=gpu:3 or higher, the job would get blocked.<br>
<br>
The above might be an annoyance to your users if their job just sits in the queue with no other notice, but it hasn't really been an issue here. The big benefit from your side would be that you could simplify the if statement down to something like 'if (job_desc.gres
~= nil)'.<br>
<br>
<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">yes, uncomment JobSubmitPlugins=lua<br>
<br>
<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Far as I know, if you uncomment the JobSubmitPlugin line and have a job_submit.lua file in the same folder as your slurm.conf, the Lua script should get executed automatically.<br>
<br>
<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Our RPM installations of Slurm contained the job_submit_lua.so, both for Bright 8 and for OpenHPC.<o:p></o:p></li></ol>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[1] <a href="https://github.com/SchedMD/slurm/blob/master/contribs/lua/job_submit.lua">
https://github.com/SchedMD/slurm/blob/master/contribs/lua/job_submit.lua</a><o:p></o:p></p>
<p class="MsoNormal">[2] <a href="https://gist.github.com/mikerenfro/df89fac5052a45cc2c1651b9a30978e0">
https://gist.github.com/mikerenfro/df89fac5052a45cc2c1651b9a30978e0</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Ratnasamy, Fritz <fritz.ratnasamy@chicagobooth.edu><br>
<b>Date: </b>Saturday, September 25, 2021 at 12:23 AM<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>[slurm-users] Block jobs on GPU partition when GPU is not specified<o:p></o:p></span></p>
</div>
<p align="center" style="margin:0in;text-align:center;background:white"><b><span style="font-size:12.0pt;color:red;background:white">External Email Warning</span></b><o:p></o:p></p>
<p align="center" style="mso-margin-top-alt:0in;margin-right:12.0pt;margin-bottom:0in;margin-left:12.0pt;text-align:center;background:white">
<b><span style="font-size:12.0pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div>
<div>
<p class="MsoNormal">Hi, <br>
<br>
I would like to block jobs submitted in our GPU partition when gres=gpu:1 (or any number between 1 and 4) is not specified when submitting a job through sbatch or requesting an interactive session with srun.
<o:p></o:p></p>
<div>
<p class="MsoNormal">Currently, /etc/slurm/slurm.conf has JobSumitPlugins=lua commented. <br>
The liblua.so is now installed. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I would like to use something similar as the example mentioned at the end of the page: <br>
<a href="https://slurm.schedmd.com/resource_limits.html%0b">https://slurm.schedmd.com/resource_limits.html<br>
</a>Can I use the following code : <br>
<br>
<o:p></o:p></p>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE">function slurm_job_submit(job_desc, part_list, submit_uid)<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> if (job_desc.gres ~= nil)<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> then<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> for g in job_desc.gres:gmatch("[^,]+")<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> do<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> bad = string.match(g,'^gpu[:]*[0-9]*$')<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> if (bad ~= nil)<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> then<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> slurm.log_info("User specified gpu GRES without type: %s", bad)<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> slurm.user_msg("You must always specify a type when requesting gpu GRES")<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> return slurm.ERROR<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> end<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> end<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE"> end<o:p></o:p></span></pre>
<pre style="margin-bottom:.25in;line-height:18.0pt;background:#333333;vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:#EEEEEE">end<o:p></o:p></span></pre>
<p class="MsoNormal">I do not need to check if the model is specified though. In that case, <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">1/ Should I change the line bad = string.match(g,'^gpu[:]*[0-9]*$') to string.match(g,'^gpu[:]*[0-9]') <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2/ Do I need to uncomment JobSumitPlugins=lua<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">3/ Where to specify the function call slurm_job_submit so I make sure the check to see if gres=gpu:1 is happening?<br>
4/ I would need job_submit_lua.so, where can I find that library and if it is not there, how can i dowload it? <br>
<br>
Thanks for your help. I am new to regular expressions, lua and Slurm so I apologize if my questions do not make sense. <br>
<br>
<br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#9C1D21">Fritz Ratnasamy</span></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#9C1D21">Data Scientist</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#9C1D21">Information Technology</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:#676E73">The University of Chicago</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:#676E73">Booth School of Business</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:#676E73">5807 S. Woodlawn</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:#676E73">Chicago,</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"> </span><span style="font-size:10.5pt;font-family:Times;color:#676E73">Illinois</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"> </span><span style="font-size:10.5pt;font-family:Times;color:#676E73">60637</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:#676E73">Phone: +(1) 773-834-4556</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>