<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><div>Hi Thomas,</div><div><br></div><div>That value should be</div><div><br></div><div>sacctmgr modify qos gpujobs set MaxTRESPerUser=gres/gpu=4</div><div><br></div><div>Sean</div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">--<br>Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead<br>Research Computing Services | Business Services<br>The University of Melbourne, Victoria 3010 Australia<br><br></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 6 May 2020 at 04:53, Theis, Thomas <<a href="mailto:Thomas.Theis@teledyne.com">Thomas.Theis@teledyne.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US"><div><div style="color:rgb(0,0,0);font-size:12px;text-align:left;font-family:Helvetica,Arial,sans-serif"><strong>UoM notice: External email. Be cautious of links, attachments, or impersonation attempts.</strong><br></div><hr></div>
<div class="gmail-m_1741108795214560605WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">Hey Killian,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">I tried to limit the number of gpus a user can run on at a time by adding MaxTRESPerUser = gres:gpu4 to both the user and the qos.. I restarted slurm control
daemon and unfortunately I am still able to run on all the gpus in the partition. Any other ideas?
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:14pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">Thomas Theis</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(46,116,181)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>>
<b>On Behalf Of </b>Killian Murphy<br>
<b>Sent:</b> Thursday, April 23, 2020 1:33 PM<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> Re: [slurm-users] Limit the number of GPUS per user per partition<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div align="center">
<table style="width:112.5pt;border:1pt outset rgb(137,137,137)" width="0" cellspacing="0" cellpadding="0" border="1">
<tbody>
<tr>
<td style="border:1pt inset rgb(137,137,137);background:rgb(253,255,178) none repeat scroll 0% 0%;padding:3.75pt">
<p class="MsoNormal" style="text-align:center" align="center"><span style="font-size:8pt">External Email<u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif">Hi Thomas.<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif">We limit the maximum number of GPUs a user can have allocated in a partition through the MaxTRESPerUser field of a QoS for GPU jobs, which is set as the partition QoS on our
GPU partition. I.E:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif">We have a QOS `gpujobs` that sets MaxTRESPerUser => gres/gpu:4 to limit total number of allocated GPUs to 4, and set the GPU partition QoS to the `gpujobs` QoS.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif">There is a section in the Slurm documentation on the 'Resource Limits' page entitled 'QOS specific limits supported (<a href="https://slurm.schedmd.com/resource_limits.html" target="_blank">https://slurm.schedmd.com/resource_limits.html</a>)
that details some care needed when using this kind of limit setting with typed GRES. Although it seems like you are trying to do something with generic GRES, it's worth a read!<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif">Killian<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif">On Thu, 23 Apr 2020 at 18:19, Theis, Thomas <<a href="mailto:Thomas.Theis@teledyne.com" target="_blank">Thomas.Theis@teledyne.com</a>> wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Hi everyone,<u></u><u></u></p>
<p class="MsoNormal">First message, I am trying find a good way or multiple ways to limit the usage of jobs per node or use of gpus per node, without blocking a user from submitting them.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Example. We have 10 nodes each with 4 gpus in a partition. We allow a team of 6 people to submit jobs to any or all of the nodes. One job per gpu; thus we can hold a total of 40
jobs concurrently in the partition.<u></u><u></u></p>
<p class="MsoNormal">At the moment: each user usually submit 50- 100 jobs at once. Taking up all gpus, and all other users have to wait in pending..<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">What I am trying to setup is allow all users to submit as many jobs as they wish but only run on 1 out of the 4 gpus per node, or some number out of the total 40 gpus across the
entire partition. Using slurm 18.08.3..<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">This is roughly our slurm scripts.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">#SBATCH --job-name=Name # Job name<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --mem=5gb # Job memory request<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --ntasks=1<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --gres=gpu:1<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --partition=PART1<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --time=200:00:00 # Time limit hrs:min:sec<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --output=job _%j.log # Standard output and error log<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --nodes=1<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --qos=high<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">srun -n1 --gres=gpu:1 --exclusive --export=ALL bash -c "NV_GPU=$SLURM_JOB_GPUS nvidia-docker run --rm -e SLURM_JOB_ID=$SLURM_JOB_ID -e SLURM_OUTPUT=$SLURM_OUTPUT --name $SLURM_JOB_ID
do_job.sh"<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:14pt;color:rgb(31,73,125)">Thomas Theis</span></b><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif"><br clear="all">
<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Calibri",sans-serif">--
<u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Calibri",sans-serif">Killian Murphy<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Calibri",sans-serif">Research Software Engineer<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Calibri",sans-serif">Wolfson Atmospheric Chemistry Laboratories<br>
University of York<br>
Heslington<br>
York<br>
YO10 5DD<br>
+44 (0)1904 32 4753<br>
<br>
e-mail disclaimer: <a href="http://www.york.ac.uk/docs/disclaimer/email.htm" target="_blank"><span style="color:rgb(17,85,204)">http://www.york.ac.uk/docs/disclaimer/email.htm</span></a><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div>