<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Helvetica Neue";
panose-1:2 0 5 3 0 0 0 2 0 4;}
@font-face
{font-family:Monaco;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Times;
panose-1:0 0 5 0 0 0 0 2 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Not a solution to your exact problem, but we document partitions for interactive, debug, and batch, and have a job_submit.lua [1] that routes GPU-reserving jobs to gpu-interactive, gpu-debug, and gpu partitions automatically. Since our
GPU nodes have extra memory slots, and have tended to run at less than 100% CPU usage during GPU jobs, they also serve as our large-memory and small interactive job targets.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[1] <a href="https://gist.github.com/mikerenfro/df89fac5052a45cc2c1651b9a30978e0">
https://gist.github.com/mikerenfro/df89fac5052a45cc2c1651b9a30978e0</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Ratnasamy, Fritz <fritz.ratnasamy@chicagobooth.edu><br>
<b>Date: </b>Tuesday, August 24, 2021 at 9:59 PM<br>
<b>To: </b>slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>[slurm-users] scancel gpu jobs when gpu is not requested<o:p></o:p></span></p>
</div>
<p align="center" style="margin:0in;text-align:center;background:white"><b><span style="font-size:12.0pt;color:red;background:white">External Email Warning</span></b><o:p></o:p></p>
<p align="center" style="mso-margin-top-alt:0in;margin-right:12.0pt;margin-bottom:0in;margin-left:12.0pt;text-align:center;background:white">
<b><span style="font-size:12.0pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div>
<div>
<p class="MsoNormal">Hello, <br>
<br>
I have written a script in my prolog.sh that cancels any slurm job if the parameter gres=gpu is not present. This is the script i added to my prolog.sh<br>
<br>
<span style="font-family:"Courier New"">if [ $SLURM_JOB_PARTITION == "gpu" ]; then<br>
if [ ! -z "${GPU_DEVICE_ORDINAL}" ]; then<br>
echo "GPU ID used is ID: $GPU_DEVICE_ORDINAL "<br>
list_gpu=$(echo "$GPU_DEVICE_ORDINAL" | sed -e "s/,//g")<br>
Ngpu=$(expr length $list_gpu)<br>
else<br>
echo "No GPU selected"<br>
Ngpu=0<br>
fi</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> # if 0 gpus were allocated, cancel the job</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> if [ "$Ngpu" -eq "0" ]; then<br>
scancel ${SLURM_JOB_ID} fi<br>
fi</span><br>
<br>
What the code does is look at the number of gpus allocated, and if it is 0, cancel the job ID. It working fine if a user use sbatch submit.sh (and the submit.sh do not have the value --gres=gpu:1). However, when requesting an interactive session without gpus,
the job is getting killed and the job hangs for 5-6 mins before getting killed. <br>
<br>
<o:p></o:p></p>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Monaco;color:#1D1C1D">jlo@mfe01:~ $ srun --partition=gpu --pty bash --login<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Monaco;color:#1D1C1D">srun: job 4631872 queued and waiting for resources<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Monaco;color:#1D1C1D">srun: job 4631872 has been allocated resources<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in"><span style="font-size:9.0pt;font-family:Monaco;color:#1D1C1D">srun: Force Terminated job 4631872 ...the killing hangs for 5-6minutes<o:p></o:p></span></pre>
<p class="MsoNormal">Is there anything wrong with my script? Why only when scancel an interactive session, I am seeing this hanging. I would like to remove the hanging<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks<br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#9C1D21">Fritz Ratnasamy</span></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#9C1D21">Data Scientist</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#9C1D21">Information Technology</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:#676E73">The University of Chicago</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:#676E73">Booth School of Business</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:#676E73">5807 S. Woodlawn</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:#676E73">Chicago,</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"> </span><span style="font-size:10.5pt;font-family:Times;color:#676E73">Illinois</span><span style="font-size:10.5pt;font-family:"Helvetica Neue";color:#333333"> </span><span style="font-size:10.5pt;font-family:Times;color:#676E73">60637</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background-image:initial;background-position:initial;background-repeat:initial">
<span style="font-size:10.5pt;font-family:Times;color:#676E73">Phone: +(1) 773-834-4556</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>