<div dir="ltr">Hi Thomas.<div><br></div><div>We limit the maximum number of GPUs a user can have allocated in a partition through the MaxTRESPerUser field of a QoS for GPU jobs, which is set as the partition QoS on our GPU partition. I.E:<br></div><div><br></div><div>We have a QOS `gpujobs` that sets MaxTRESPerUser => gres/gpu:4 to limit total number of allocated GPUs to 4, and set the GPU partition QoS to the `gpujobs` QoS.</div><div><br></div><div>There is a section in the Slurm documentation on the 'Resource Limits' page entitled 'QOS specific limits supported (<a href="https://slurm.schedmd.com/resource_limits.html">https://slurm.schedmd.com/resource_limits.html</a>) that details some care needed when using this kind of limit setting with typed GRES. Although it seems like you are trying to do something with generic GRES, it's worth a read!</div><div><br></div><div>Killian</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 23 Apr 2020 at 18:19, Theis, Thomas <<a href="mailto:Thomas.Theis@teledyne.com">Thomas.Theis@teledyne.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_8502391332794380603WordSection1">
<p class="MsoNormal">Hi everyone,<u></u><u></u></p>
<p class="MsoNormal">First message, I am trying find a good way or multiple ways to limit the usage of jobs per node or use of gpus per node, without blocking a user from submitting them.
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Example. We have 10 nodes each with 4 gpus in a partition. We allow a team of 6 people to submit jobs to any or all of the nodes. One job per gpu; thus we can hold a total of 40 jobs concurrently in the partition.<u></u><u></u></p>
<p class="MsoNormal">At the moment: each user usually submit 50- 100 jobs at once. Taking up all gpus, and all other users have to wait in pending..<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">What I am trying to setup is allow all users to submit as many jobs as they wish but only run on 1 out of the 4 gpus per node, or some number out of the total 40 gpus across the entire partition. Using slurm 18.08.3..<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">This is roughly our slurm scripts.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">#SBATCH --job-name=Name # Job name<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --mem=5gb # Job memory request<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --ntasks=1<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --gres=gpu:1<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --partition=PART1<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --time=200:00:00 # Time limit hrs:min:sec<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --output=job _%j.log # Standard output and error log<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --nodes=1<u></u><u></u></p>
<p class="MsoNormal">#SBATCH --qos=high<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">srun -n1 --gres=gpu:1 --exclusive --export=ALL bash -c "NV_GPU=$SLURM_JOB_GPUS nvidia-docker run --rm -e SLURM_JOB_ID=$SLURM_JOB_ID -e SLURM_OUTPUT=$SLURM_OUTPUT --name $SLURM_JOB_ID do_job.sh"<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><b><span style="font-size:14pt;color:rgb(31,73,125)">Thomas Theis</span></b><span style="color:rgb(46,116,181)"><u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr" style="font-size:12.8px">Killian Murphy</div><div dir="ltr" style="font-size:12.8px">Research Software Engineer</div><div dir="ltr" style="font-size:12.8px"><br></div><div dir="ltr" style="font-size:12.8px">Wolfson Atmospheric Chemistry Laboratories<br>University of York<br>Heslington<br>York<br>YO10 5DD<br>+44 (0)1904 32 4753<br><br>e-mail disclaimer: <a href="http://www.york.ac.uk/docs/disclaimer/email.htm" style="color:rgb(17,85,204)" target="_blank">http://www.york.ac.uk/docs/disclaimer/email.htm</a></div></div></div></div></div></div></div></div></div></div></div></div></div>