<div dir="ltr">Hi Michael Renfro, <div><br>Thanks for your reply. Based on your answers, would this work: <br>1/ a function job_submit.lua with the following contents (just need a function that errored when gres:gpu is not specified in srun or in sbatch): <br><br></div><div>function slurm_job_submit(job_desc, part_list, submit_uid)<br><br> if job_desc.partition == 'gpu' then<br> if (job_desc.gres == nil) then<br> slurm.log_info("User did not specified gres=gpu: ")<br> slurm.user_msg("You have to specify gres=gpu:x where x is number of GPUs.")<br> return slurm.ERROR<br> end<br> end<br>end</div><div><br></div><div><br></div><div>4/ I found out a file the file job_submit_lua.so in our controller in /lib64/slurm/ and also the lua lib seems to be installed: <br> sudo rpm -qa | grep lua<br><br></div><div>lua-5.3.4-11.el8.x86_64<br>lua-libs-5.3.4-11.el8.x86_64<br>lua-devel-5.3.4-11.el8.x86_64<br></div><div><br></div><div> so I guess for now I just need to create job_submit.lua, uncomment the job plugin in slurm.conf/ is there any Slurm service to restart after that?<br><br>Thanks again<br clear="all"><div><div dir="ltr" data-smartmail="gmail_signature"><div dir="ltr"><p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#9c1d21">Fritz Ratnasamy</span></b><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"></span></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#9c1d21">Data Scientist</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"></span></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#9c1d21">Information Technology</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"></span></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><span style="color:rgb(103,110,115);font-family:Times,serif;font-size:10.5pt">The University of Chicago</span></p><p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"></span></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10.5pt;font-family:"Times",serif;color:#676e73">Booth School of Business</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"></span></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10.5pt;font-family:"Times",serif;color:#676e73">5807 S. Woodlawn</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"></span></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10.5pt;font-family:"Times",serif;color:#676e73">Chicago,</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"> </span><span style="font-size:10.5pt;font-family:"Times",serif;color:#676e73">Illinois</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"> </span><span style="font-size:10.5pt;font-family:"Times",serif;color:#676e73">60637</span></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10.5pt;font-family:"Times",serif;color:#676e73">Phone: +(1) 773-834-4556</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"></span></p></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 25, 2021 at 11:08 AM Renfro, Michael <<a href="mailto:Renfro@tntech.edu" target="_blank">Renfro@tntech.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div>
<p class="MsoNormal">If you haven't already seen it there's an example Lua script from SchedMD at [1], and I've got a copy of our local script at [2]. Otherwise, in the order you asked:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<ol style="margin-top:0in" start="1" type="1">
<li style="margin-left:0in">That seems reasonable, but our script just checks if there's a gres at all. I don't *think* any gres other than gres=gpu would let the job run, since our GPU nodes only have Gres=gpu:2
entries. Same thing for asking for more GPUs than are in the node: if someone asked for gres=gpu:3 or higher, the job would get blocked.<br>
<br>
The above might be an annoyance to your users if their job just sits in the queue with no other notice, but it hasn't really been an issue here. The big benefit from your side would be that you could simplify the if statement down to something like 'if (job_desc.gres
~= nil)'.<br>
<br>
<u></u><u></u></li><li style="margin-left:0in">yes, uncomment JobSubmitPlugins=lua<br>
<br>
<u></u><u></u></li><li style="margin-left:0in">Far as I know, if you uncomment the JobSubmitPlugin line and have a job_submit.lua file in the same folder as your slurm.conf, the Lua script should get executed automatically.<br>
<br>
<u></u><u></u></li><li style="margin-left:0in">Our RPM installations of Slurm contained the job_submit_lua.so, both for Bright 8 and for OpenHPC.<u></u><u></u></li></ol>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">[1] <a href="https://github.com/SchedMD/slurm/blob/master/contribs/lua/job_submit.lua" target="_blank">
https://github.com/SchedMD/slurm/blob/master/contribs/lua/job_submit.lua</a><u></u><u></u></p>
<p class="MsoNormal">[2] <a href="https://gist.github.com/mikerenfro/df89fac5052a45cc2c1651b9a30978e0" target="_blank">
https://gist.github.com/mikerenfro/df89fac5052a45cc2c1651b9a30978e0</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Ratnasamy, Fritz <<a href="mailto:fritz.ratnasamy@chicagobooth.edu" target="_blank">fritz.ratnasamy@chicagobooth.edu</a>><br>
<b>Date: </b>Saturday, September 25, 2021 at 12:23 AM<br>
<b>To: </b>Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject: </b>[slurm-users] Block jobs on GPU partition when GPU is not specified<u></u><u></u></span></p>
</div>
<p align="center" style="margin:0in;text-align:center;background:white"><b><span style="font-size:12pt;color:red;background:white">External Email Warning</span></b><u></u><u></u></p>
<p align="center" style="margin-right:12pt;margin-bottom:0in;margin-left:12pt;text-align:center;background:white">
<b><span style="font-size:12pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><u></u><u></u></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div>
<div>
<p class="MsoNormal">Hi, <br>
<br>
I would like to block jobs submitted in our GPU partition when gres=gpu:1 (or any number between 1 and 4) is not specified when submitting a job through sbatch or requesting an interactive session with srun.
<u></u><u></u></p>
<div>
<p class="MsoNormal">Currently, /etc/slurm/slurm.conf has JobSumitPlugins=lua commented. <br>
The liblua.so is now installed. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I would like to use something similar as the example mentioned at the end of the page: <br>
<a href="https://slurm.schedmd.com/resource_limits.html%0b" target="_blank">https://slurm.schedmd.com/resource_limits.html<br>
</a>Can I use the following code : <br>
<br>
<u></u><u></u></p>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)">function slurm_job_submit(job_desc, part_list, submit_uid)<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> if (job_desc.gres ~= nil)<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> then<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> for g in job_desc.gres:gmatch("[^,]+")<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> do<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> bad = string.match(g,'^gpu[:]*[0-9]*$')<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> if (bad ~= nil)<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> then<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> slurm.log_info("User specified gpu GRES without type: %s", bad)<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> slurm.user_msg("You must always specify a type when requesting gpu GRES")<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> return slurm.ERROR<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> end<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> end<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)"> end<u></u><u></u></span></pre>
<pre style="margin-bottom:0.25in;line-height:18pt;background:rgb(51,51,51);vertical-align:baseline"><span style="font-size:9.5pt;font-family:"Source Code Pro";color:rgb(238,238,238)">end<u></u><u></u></span></pre>
<p class="MsoNormal">I do not need to check if the model is specified though. In that case, <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1/ Should I change the line bad = string.match(g,'^gpu[:]*[0-9]*$') to string.match(g,'^gpu[:]*[0-9]') <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2/ Do I need to uncomment JobSumitPlugins=lua<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">3/ Where to specify the function call slurm_job_submit so I make sure the check to see if gres=gpu:1 is happening?<br>
4/ I would need job_submit_lua.so, where can I find that library and if it is not there, how can i dowload it? <br>
<br>
Thanks for your help. I am new to regular expressions, lua and Slurm so I apologize if my questions do not make sense. <br>
<br>
<br clear="all">
<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:Arial,sans-serif;color:rgb(156,29,33)">Fritz Ratnasamy</span></b><u></u><u></u></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Arial,sans-serif;color:rgb(156,29,33)">Data Scientist</span><u></u><u></u></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Arial,sans-serif;color:rgb(156,29,33)">Information Technology</span><u></u><u></u></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:rgb(103,110,115)">The University of Chicago</span><u></u><u></u></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:rgb(103,110,115)">Booth School of Business</span><u></u><u></u></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:rgb(103,110,115)">5807 S. Woodlawn</span><u></u><u></u></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:rgb(103,110,115)">Chicago,</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:rgb(51,51,51)"> </span><span style="font-size:10.5pt;font-family:Times;color:rgb(103,110,115)">Illinois</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:rgb(51,51,51)"> </span><span style="font-size:10.5pt;font-family:Times;color:rgb(103,110,115)">60637</span><u></u><u></u></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:rgb(103,110,115)">Phone: +(1) 773-834-4556</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p></p>
<div style="background-color:rgb(255,235,156);width:100%;border-style:solid;border-color:rgb(156,101,0);border-width:1pt;padding:2pt;font-size:10pt;line-height:12pt;font-family:Calibri;color:black;text-align:left">
<span style="color:rgb(156,101,0)">CAUTION:</span> This email has originated outside of University email systems. Please do not click links or open attachments unless you recognize the sender and trust the contents as safe.</div>
<br>
<p></p>
</div>
</blockquote></div>