<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Fritz,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
job_submit_lua.so gets made upon compilation of Slurm if you have the lua-devel package installed at the time of configure/make.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Sean<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Ratnasamy, Fritz <fritz.ratnasamy@chicagobooth.edu><br>
<b>Sent:</b> Tuesday, 31 August 2021 15:05<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [EXT] Re: [slurm-users] EXTERNAL-Re: [External] scancel gpu jobs when gpu is not requested</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div class="x_mc-ip-hide">
<div style="color:#000000; font-size:12px; text-align:left; font-family:Helvetica,Arial,sans-serif">
<strong>
<table border="0" cellspacing="0" cellpadding="0" style="width:100%; float:left">
<tbody>
<tr>
<td style="color:red"><b>External email: </b>Please exercise caution</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
Hi Michael,
<div><br>
Thanks for your message. Does the installation of the library job_submit_lua.so need to have Slurm recompiled as well, ie, do I have to compile slurm with the library job_submit_lua.so to be able to add any plugin?I do not see it in the yum repo.</div>
<div>Thanks,<br clear="all">
<div>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<b><span style="font-size:10.5pt; font-family:"Arial",sans-serif; color:#9c1d21">Fritz Ratnasamy</span></b><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:#333333"></span></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:"Arial",sans-serif; color:#9c1d21">Data Scientist</span><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:#333333"></span></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:"Arial",sans-serif; color:#9c1d21">Information Technology</span><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:#333333"></span></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="color:rgb(103,110,115); font-family:Times,serif; font-size:10.5pt">The University of Chicago</span></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:#333333"></span></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:"Times",serif; color:#676e73">Booth School of Business</span><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:#333333"></span></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:"Times",serif; color:#676e73">5807 S. Woodlawn</span><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:#333333"></span></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:"Times",serif; color:#676e73">Chicago,</span><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:#333333"> </span><span style="font-size:10.5pt; font-family:"Times",serif; color:#676e73">Illinois</span><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:#333333"> </span><span style="font-size:10.5pt; font-family:"Times",serif; color:#676e73">60637</span></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:"Times",serif; color:#676e73">Phone: +(1) 773-834-4556</span><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:#333333"></span></p>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Thu, Aug 26, 2021 at 9:18 AM Michael Robbert <<a href="mailto:mrobbert@mines.edu">mrobbert@mines.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div lang="EN-US" style="">
<div class="x_gmail-m_6097797238232889236WordSection1">
<p class="x_MsoNormal">You need to set the following option in slurm.conf<u></u><u></u></p>
<p class="x_MsoNormal" style="background:white"><b><span style="font-size:15pt; color:rgb(70,84,92); border:1pt none windowtext; padding:0in">JobSubmitPlugins</span></b><span style="font-size:15pt; font-family:"Source Sans Pro",sans-serif; color:rgb(70,84,92)"><u></u><u></u></span></p>
<p class="x_MsoNormal" style="margin-left:0.5in; background:white"><span style="font-size:15pt; font-family:"Source Sans Pro",sans-serif; color:rgb(70,84,92)">A comma delimited list of job submission plugins to be used. The specified plugins will be executed
in the order listed. These are intended to be site-specific plugins which can be used to set default job parameters and/or logging events. Sample plugins available in the distribution include "all_partitions", "defaults", "logging", "lua", and "partition".
For examples of use, see the Slurm code in "src/plugins/job_submit" and "contribs/lua/job_submit*.lua" then modify the code to satisfy your needs. Slurm can be configured to use multiple job_submit plugins if desired, however the lua plugin will only execute
one lua script named "job_submit.lua" located in the default script directory (typically the subdirectory "etc" of the installation directory). No job submission plugins are used by default.<u></u><u></u></span></p>
<p class="x_MsoNormal"><u></u> <u></u></p>
<p class="x_MsoNormal"><u></u> <u></u></p>
<p class="x_MsoNormal">Then as this documentation states, put the job_submit.lua into your script directory. Mine is in /etc/slurm/. You may want to make sure that you have the job_submit_lua.so library installed with your build of Slurm. I agree that finding
complete documentation for this feature is a little difficult.<u></u><u></u></p>
<p class="x_MsoNormal"><u></u> <u></u></p>
<p class="x_MsoNormal">Mike<u></u><u></u></p>
<p class="x_MsoNormal"><u></u> <u></u></p>
<div style="border-right:none; border-bottom:none; border-left:none; border-top:1pt solid rgb(181,196,223); padding:3pt 0in 0in">
<p class="x_MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt; color:black">From:
</span></b><span style="font-size:12pt; color:black">slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Ratnasamy, Fritz <<a href="mailto:fritz.ratnasamy@chicagobooth.edu" target="_blank">fritz.ratnasamy@chicagobooth.edu</a>><br>
<b>Date: </b>Wednesday, August 25, 2021 at 23:13<br>
<b>To: </b>Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject: </b>Re: [slurm-users] EXTERNAL-Re: [External] scancel gpu jobs when gpu is not requested<u></u><u></u></span></p>
</div>
<div>
<p class="x_MsoNormal">Hi Michael,<u></u><u></u></p>
<div>
<p class="x_MsoNormal">Thanks for your message. Yes I was able to get all interactive sessions killed quickly when trying other partitions and deactivating the prolog. I read your example and I understand how it could possibly work (in the ex., maybe instead
of looking if the gpu model is passed, we could look at the number of gpu passed?), but where do i set up that function and where do i call it? <br>
Thanks,<br clear="all">
<u></u><u></u></p>
<div>
<div>
<div>
<p class="x_MsoNormal"><b><span style="font-size:10.5pt; font-family:Arial,sans-serif; color:rgb(156,29,33)">Fritz Ratnasamy</span></b><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Arial,sans-serif; color:rgb(156,29,33)">Data Scientist</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Arial,sans-serif; color:rgb(156,29,33)">Information Technology</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">The University of Chicago</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">Booth School of Business</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">5807 S. Woodlawn</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">Chicago,</span><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:rgb(51,51,51)"> </span><span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">Illinois</span><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:rgb(51,51,51)"> </span><span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">60637</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">Phone: +(1) 773-834-4556</span><u></u><u></u></p>
</div>
</div>
</div>
<p class="x_MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="x_MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="x_MsoNormal">On Wed, Aug 25, 2021 at 9:54 AM Michael Robbert <<a href="mailto:mrobbert@mines.edu" target="_blank">mrobbert@mines.edu</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none; border-right:none; border-bottom:none; border-left:1pt solid rgb(204,204,204); padding:0in 0in 0in 6pt; margin-left:4.8pt; margin-right:0in">
<div>
<div>
<p class="x_MsoNormal">I doubt that it is a problem with your script and suspect that there is some weird interaction with scancel on interactive jobs. If you wanted to get to the bottom of that I’d suggest disabling the prolog and test by manually cancelling
some interactive jobs. <u></u><u></u></p>
<p class="x_MsoNormal">Another suggestion is to try a completely different approach to solve your problem. Why wait until the job starts to do the check? You can use a submit filter and it will alert the user as soon as they try to submit. That will prevent
them from potentially having to wait in the queue if the cluster is busy and gets around having to cancel a running job. There is a description and simple example at the bottom of this page:
<a href="https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fresource_limits.html&data=04%7C01%7Cmrobbert%40mines.edu%7C577fad20cd024e8f8d5a08d96850336c%7C997209e009b346239a4d76afa44a675c%7C0%7C0%7C637655515944014175%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=vnuQFWtAkvixWlJaLCVa%2Bcmt0Zt97RCWhStXO1VLoss%3D&reserved=0" target="_blank">
https://slurm.schedmd.com/resource_limits.html</a><u></u><u></u></p>
<p class="x_MsoNormal"> <u></u><u></u></p>
<p class="x_MsoNormal">Mike<u></u><u></u></p>
<p class="x_MsoNormal"> <u></u><u></u></p>
<div style="border-right:none; border-bottom:none; border-left:none; border-top:1pt solid rgb(181,196,223); padding:3pt 0in 0in">
<p class="x_MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt; color:black">From:
</span></b><span style="font-size:12pt; color:black">slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Ratnasamy, Fritz <<a href="mailto:fritz.ratnasamy@chicagobooth.edu" target="_blank">fritz.ratnasamy@chicagobooth.edu</a>><br>
<b>Date: </b>Tuesday, August 24, 2021 at 21:00<br>
<b>To: </b><a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a> <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject: </b>[External] [slurm-users] scancel gpu jobs when gpu is not requested</span><u></u><u></u></p>
</div>
<div style="border:1pt solid rgb(156,101,0); padding:2pt">
<p class="x_MsoNormal" style="line-height:12pt; background:rgb(255,235,156)"><b><span style="font-size:10pt; color:rgb(156,101,0)">CAUTION:</span></b><span style="font-size:10pt; color:black"> This email originated from outside of the Colorado School of Mines
organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.</span><u></u><u></u></p>
</div>
<p class="x_MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="x_MsoNormal">Hello, <br>
<br>
I have written a script in my prolog.sh that cancels any slurm job if the parameter gres=gpu is not present. This is the script i added to my prolog.sh<br>
<br>
<span style="font-family:"Courier New"">if [ $SLURM_JOB_PARTITION == "gpu" ]; then<br>
if [ ! -z "${GPU_DEVICE_ORDINAL}" ]; then<br>
echo "GPU ID used is ID: $GPU_DEVICE_ORDINAL "<br>
list_gpu=$(echo "$GPU_DEVICE_ORDINAL" | sed -e "s/,//g")<br>
Ngpu=$(expr length $list_gpu)<br>
else<br>
echo "No GPU selected"<br>
Ngpu=0<br>
fi</span> <u></u><u></u></p>
<div>
<p class="x_MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-family:"Courier New""> # if 0 gpus were allocated, cancel the job</span><u></u><u></u></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-bottom:12pt"><span style="font-family:"Courier New""> if [ "$Ngpu" -eq "0" ]; then<br>
scancel ${SLURM_JOB_ID} fi<br>
fi</span><br>
<br>
What the code does is look at the number of gpus allocated, and if it is 0, cancel the job ID. It working fine if a user use sbatch submit.sh (and the submit.sh do not have the value --gres=gpu:1). However, when requesting an interactive session without gpus,
the job is getting killed and the job hangs for 5-6 mins before getting killed. <u></u><u></u></p>
<pre style="margin-bottom:3pt"><span style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">jlo@mfe01:~ $ srun --partition=gpu --pty bash --login</span><u></u><u></u></pre>
<pre style="margin-bottom:3pt"><span style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">srun: job 4631872 queued and waiting for resources</span><u></u><u></u></pre>
<pre style="margin-bottom:3pt"><span style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">srun: job 4631872 has been allocated resources</span><u></u><u></u></pre>
<pre style="margin-bottom:3pt"><span style="font-size:9pt; font-family:Monaco; color:rgb(29,28,29)">srun: Force Terminated job 4631872 ...the killing hangs for 5-6minutes</span><u></u><u></u></pre>
<p class="x_MsoNormal">Is there anything wrong with my script? Why only when scancel an interactive session, I am seeing this hanging. I would like to remove the hanging<u></u><u></u></p>
</div>
<div>
<p class="x_MsoNormal">Thanks<br clear="all">
<u></u><u></u></p>
<div>
<div>
<div>
<p class="x_MsoNormal"><b><span style="font-size:10.5pt; font-family:Arial,sans-serif; color:rgb(156,29,33)">Fritz Ratnasamy</span></b><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Arial,sans-serif; color:rgb(156,29,33)">Data Scientist</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Arial,sans-serif; color:rgb(156,29,33)">Information Technology</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">The University of Chicago</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">Booth School of Business</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">5807 S. Woodlawn</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">Chicago,</span><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:rgb(51,51,51)"> </span><span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">Illinois</span><span style="font-size:10.5pt; font-family:"Helvetica Neue"; color:rgb(51,51,51)"> </span><span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">60637</span><u></u><u></u></p>
<p class="x_MsoNormal" style="background-image:initial; background-position:initial; background-repeat:initial">
<span style="font-size:10.5pt; font-family:Times; color:rgb(103,110,115)">Phone: +(1) 773-834-4556</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div style="border:1pt solid rgb(156,101,0); padding:2pt">
<p class="x_MsoNormal" style="line-height:12pt; background:rgb(255,235,156)"><span style="font-size:10pt; color:rgb(156,101,0)">CAUTION:</span><span style="font-size:10pt; color:black"> This email has originated outside of University email systems. Please do
not click links or open attachments unless you recognize the sender and trust the contents as safe.<u></u><u></u></span></p>
</div>
<p class="x_MsoNormal"><u></u> <u></u></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>