<div dir="ltr"><div>Ryan,</div><div><br></div><div>That's good to know!<br></div><div>It would be great to get this working as users are used to checking via nvidia-smi. For now,</div><div>I have a few jobs ready for the coming weekend! Will check on this later.<br><br></div><div>Thanks for your help! <br></div><div><br></div><div>Abhiram<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 14, 2021 at 3:20 PM Ryan Novosielski <<a href="mailto:novosirj@rutgers.edu">novosirj@rutgers.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">
AFAIK, if you have this set up correctly, nvidia-smi will be restricted too, though I think we were seeing a bug there at one time in this version. <br>
<br>
<div dir="ltr"><span style="background-color:rgba(255,255,255,0)">--</span>
<div><span style="background-color:rgba(255,255,255,0)">#BlackLivesMatter<br>
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu" dir="ltr" target="_blank">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'</span></div>
</div>
<div dir="ltr"><br>
<blockquote type="cite">On Jan 14, 2021, at 18:05, Abhiram Chintangal <<a href="mailto:achintangal@berkeley.edu" target="_blank">achintangal@berkeley.edu</a>> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Sean, <br>
</div>
<div><br>
</div>
<div>Thanks for the clarification.I noticed that I am missing the "AllowedDevices" option in mine. After adding this, the GPU allocations started working. (Slurm version 18.08.8)
<br>
</div>
<div><br>
</div>
<div>I was also incorrectly using "nvidia-smi" as a check. <br>
</div>
<div><br>
</div>
<div>Regards, <br>
</div>
<div><br>
</div>
<div>Abhiram<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jan 14, 2021 at 12:22 AM Sean Crosby <<a href="mailto:scrosby@unimelb.edu.au" target="_blank">scrosby@unimelb.edu.au</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Hi Abhiram,</div>
<div><br>
</div>
<div>You need to configure cgroup.conf to constrain the devices a job has access to. See
<a href="https://slurm.schedmd.com/cgroup.conf.html" target="_blank">https://slurm.schedmd.com/cgroup.conf.html</a></div>
<div><br>
</div>
<div>My cgroup.conf is</div>
<div><br>
</div>
<div style="margin-left:40px">CgroupAutomount=yes<br>
AllowedDevicesFile="/usr/local/slurm/etc/cgroup_allowed_devices_file.conf"<br>
<br>
ConstrainCores=yes<br>
ConstrainRAMSpace=yes<br>
ConstrainSwapSpace=yes<br>
ConstrainDevices=yes<br>
<br>
TaskAffinity=no<br>
<br>
CgroupMountpoint=/sys/fs/cgroup</div>
<div style="margin-left:40px"><br>
</div>
<div>The ConstrainDevices=yes is the key to stopping jobs from having access to GPUs they didn't request.</div>
<div><br>
</div>
<div>Sean</div>
<div><br>
</div>
<div>
<div>
<div>
<div>
<div dir="ltr">--<br>
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead<br>
Research Computing Services | Business Services<br>
The University of Melbourne, Victoria 3010 Australia<br>
<br>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, 14 Jan 2021 at 18:36, Abhiram Chintangal <<a href="mailto:achintangal@berkeley.edu" target="_blank">achintangal@berkeley.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div style="color:rgb(0,0,0);font-size:12px;text-align:left;font-family:Helvetica,Arial,sans-serif">
<strong>
<table style="width:100%;float:left;background-color:lemonchiffon" cellspacing="0" cellpadding="5" border="1">
<tbody>
<tr>
<td><b>UoM notice: </b>External email. Be cautious of links, attachments, or impersonation attempts</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
Hello, <br>
</div>
<div><br>
</div>
<div>I recently set up a small cluster at work using Warewulf/Slurm. Currently, I am not able to get the scheduler to
<br>
</div>
<div>work well with GPU's (Gres). <br>
</div>
<div><br>
</div>
<div>While slurm is able to filter by GPU type, it allocates all the GPU's on the node. See below:</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span style="font-family:monospace">[abhiram@whale ~]$ srun --gres=gpu:p100:2 -n 1 --partition=gpu nvidia-smi --query-gpu=index,name --format=csv</span><br>
<span style="font-family:monospace">index, name</span><br>
<span style="font-family:monospace">0, Tesla P100-PCIE-16GB</span><br>
<span style="font-family:monospace">1, Tesla P100-PCIE-16GB</span><br>
<span style="font-family:monospace">2, Tesla P100-PCIE-16GB</span><br>
<span style="font-family:monospace">3, Tesla P100-PCIE-16GB</span><br>
<span style="font-family:monospace">[abhiram@whale ~]$ srun --gres=gpu:titanrtx:2 -n 1 --partition=gpu nvidia-smi --query-gpu=index,name --format=csv</span><br>
<span style="font-family:monospace">index, name</span><br>
<span style="font-family:monospace">0, TITAN RTX</span><br>
<span style="font-family:monospace">1, TITAN RTX</span><br>
<span style="font-family:monospace">2, TITAN RTX</span><br>
<span style="font-family:monospace">3, TITAN RTX</span><br>
<span style="font-family:monospace">4, TITAN RTX</span><br>
<span style="font-family:monospace">5, TITAN RTX</span><br>
<span style="font-family:monospace">6, TITAN RTX</span><br>
<span style="font-family:monospace">7, TITAN RTX</span><br>
<span style="font-family:monospace"></span></blockquote>
<div><br>
</div>
<div>I am fairly new to Slurm and still figuring out my way around it. I would really appreciate any help with this.</div>
<div><br>
</div>
<div>For your reference, I attached the slurm.conf and gres.conf files. <br>
</div>
<div><br>
</div>
<div>Best, <br>
</div>
<div><br>
</div>
<div>Abhiram <br>
</div>
<div><br>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<pre cols="72">Abhiram Chintangal
QB3 Nogales Lab
Bioinformatics Specialist @ Howard Hughes Medical Institute
University of California Berkeley
708D Stanley Hall, Berkeley, CA 94720
Phone (510)666-3344<br></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<pre cols="72">Abhiram Chintangal
QB3 Nogales Lab
Bioinformatics Specialist @ Howard Hughes Medical Institute
University of California Berkeley
708D Stanley Hall, Berkeley, CA 94720
Phone (510)666-3344<br></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><pre cols="72">Abhiram Chintangal
QB3 Nogales Lab
Bioinformatics Specialist @ Howard Hughes Medical Institute
University of California Berkeley
708D Stanley Hall, Berkeley, CA 94720
Phone (510)666-3344<br></pre>
</div></div></div></div></div></div>