<div dir="ltr">Hi,<div><br></div><div>Thanks for the reply. Yes, your advice helped! Much obliged. Not only was cgroups config necessary, but the option</div><div><br></div><div>ConstrainDevices=yes<br></div><div><br></div><div>in cgroup.conf was necessary to enforce the gpu gres. Now, not adding a gres parameter to srun causes gpu jobs to fail. An improvement!</div><div><br></div><div>Although, I still can't keep out gpu jobs from the "CPU" partition. Is there a way to link a partition to a GRES or something?</div><div><br></div><div>Alternatively, can I define two nodenames in slurm.conf that point to the same physical node, but only one of them has the gpu GRES? That way, I can link the GPU partition to the gres-configged nodename only.</div><div><br></div><div>Thanks in advance,</div><div>AR</div><div><br></div><div><b>PS</b>: If the slurm devs are reading this, may I suggest that perhaps it would be a good idea to add a reference to cgroups in the gres documentation page?</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div> </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 2 Feb 2023 at 16:52, Holtgrewe, Manuel <<a href="mailto:manuel.holtgrewe@bih-charite.de">manuel.holtgrewe@bih-charite.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div id="m_-3115091350617040803divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>if by "share the GPU" you mean exclusive allocation to a single job then, I believe, you are missing cgroup configuration for isolating access to the GPU.</p>
<p><br>
</p>
<p>Below the relevant parts (I believe) of our configuration.</p>
<p><br>
</p>
<p>There also is a way of time- and space-slice GPUs but I guess you should get things setup without slicing.<br>
</p>
<p><br>
</p>
<p>I hope this helps.</p>
<p><br>
</p>
<p>Manuel<br>
</p>
<p><br>
</p>
<p></p>
<div>==> /etc/slurm/cgroup.conf <==<br>
# <a href="https://bugs.schedmd.com/show_bug.cgi?id=3701" target="_blank">https://bugs.schedmd.com/show_bug.cgi?id=3701</a><br>
CgroupMountpoint="/sys/fs/cgroup"<br>
</div>
<div>CgroupAutomount=yes<br>
</div>
<div>AllowedDevicesFile="/etc/slurm/cgroup_allowed_devices_file.conf"<br>
<br>
==> /etc/slurm/cgroup_allowed_devices_file.conf <==<br>
/dev/null<br>
/dev/urandom<br>
/dev/zero<br>
/dev/sda*<br>
/dev/cpu/*/*<br>
/dev/pts/*<br>
/dev/nvidia*<br>
<br>
</div>
<p>==> /etc/slurm/slurm.conf <==</p>
<p></p>
<div>ProctrackType=proctrack/cgroup<br>
<br>
</div>
<div># Memory is enforced via cgroups, so we should not do this here by [*]<br>
#<br>
# /etc/slurm/cgroup.conf: ConstrainRAMSpace=yes<br>
#<br>
# [*] <a href="https://bugs.schedmd.com/show_bug.cgi?id=5262" target="_blank">https://bugs.schedmd.com/show_bug.cgi?id=5262</a><br>
JobAcctGatherParams=NoOverMemoryKill<br>
<br>
</div>
<div>TaskPlugin=task/cgroup<br>
</div>
<div><br>
</div>
JobAcctGatherType=jobacct_gather/cgroup<br>
<p></p>
<p></p>
<p><br>
</p>
<div id="m_-3115091350617040803Signature">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px">
<div><font size="2"><span style="font-size:10pt">-- <br>
Dr. Manuel Holtgrewe, Dipl.-Inform.<br>
Bioinformatician<br>
Core Unit Bioinformatics – CUBI<br>
Berlin Institute of Health / Max Delbrück Center for Molecular Medicine in the Helmholtz Association / Charité – Universitätsmedizin Berlin<br>
<br>
Visiting Address: Invalidenstr. 80, 3rd Floor, Room 03 028, 10117 Berlin<br>
Postal Address: Chariteplatz 1, 10117 Berlin<br>
<br>
E-Mail: <a href="mailto:manuel.holtgrewe@bihealth.de" target="_blank">manuel.holtgrewe@bihealth.de</a><br>
Phone: +49 30 450 543 607<br>
Fax: +49 30 450 7 543 901<br>
Web: <a href="http://cubi.bihealth.org" target="_blank">cubi.bihealth.org</a> <a href="http://www.bihealth.org" target="_blank">www.bihealth.org</a> <a href="http://www.mdc-berlin.de" target="_blank">www.mdc-berlin.de</a> <a href="http://www.charite.de" target="_blank">www.charite.de</a></span></font><font size="2"><span style="font-size:10pt">
</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-3115091350617040803divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Analabha Roy <<a href="mailto:hariseldon99@gmail.com" target="_blank">hariseldon99@gmail.com</a>><br>
<b>Sent:</b> Wednesday, February 1, 2023 6:12:40 PM<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> [ext] [slurm-users] Enforce gpu usage limits (with GRES?)</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi,
<div><br>
</div>
<div>I'm new to slurm, so I apologize in advance if my question seems basic.</div>
<div><br>
</div>
<div>I just purchased a single node 'cluster' consisting of one 64-core cpu and an nvidia rtx5k gpu (Turing architecture, I think). The vendor supplied it with ubuntu 20.04 and slurm-wlm 19.05.5. Now I'm trying to adjust the config to suit the needs of my department.</div>
<div><br>
</div>
<div>I'm trying to bone up on GRES scheduling by reading <a href="https://slurm.schedmd.com/gres.html" target="_blank">
this manual page</a>, but am confused about some things.</div>
<div><br>
</div>
<div>My slurm.conf file has the following lines put in it by the vendor:</div>
<div><br>
</div>
<div>###################</div>
# COMPUTE NODES<br>
GresTypes=gpu<br>
NodeName=shavak-DIT400TR-55L CPUs=64 SocketsPerBoard=2 CoresPerSocket=32 ThreadsPerCore=1 RealMemory=95311 Gres=gpu:1<br>
#PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP<br>
<br>
PartitionName=CPU Nodes=ALL Default=Yes MaxTime=INFINITE State=UP<br>
<br>
PartitionName=GPU Nodes=ALL Default=NO MaxTime=INFINITE State=UP<br>
<div>#####################</div>
<div><br>
</div>
<div>So they created two partitions that are essentially identical. Secondly, they put just the following line in gres.conf:</div>
<div><br>
</div>
<div>###################<br>
</div>
<div>NodeName=shavak-DIT400TR-55L Name=gpu File=/dev/nvidia0<br>
</div>
<div>###################<br>
</div>
<div><br>
</div>
<div>That's all. However, this configuration does not appear to constrain anyone in any manner. As a regular user, I can still use srun or sbatch to start GPU jobs from the "CPU partition," and nvidia-smi says that a simple
<a href="https://cupy.dev/" target="_blank">cupy</a> script that multiplies matrices and starts as an sbatch job in the CPU partition can access the gpu just fine. Note that the environment variable "CUDA_VISIBLE_DEVICES" does not appear to be set in any job step. I tested
this by starting an interactive srun shell in both CPU and GPU partition and running ''echo $CUDA_VISIBLE_DEVICES" and got bupkis for both.<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>What I need to do is constrain jobs to using chunks of GPU Cores/RAM so that multiple jobs can share the GPU. </div>
<div><br>
</div>
<div>As I understand from the gres manpage, simply adding "AutoDetect=nvml" (NVML should be installed with the NVIDIA HPC SDK, right? I installed it with apt-get...) in gres.conf should allow Slurm to detect the GPU's internal specifications automatically.
Is that all, or do I need to config an mps GRES as well? Will that succeed in jailing out the GPU from jobs that don't mention any gres parameters (perhaps by setting CUDA_VISIBLE_DEVICES), or is there any additional config for that? Do I really need that
extra "GPU" partition that the vendor put in for any of this, or is there a way to bind GRES resources to a particular partition in such a way that simply launching jobs in that partition will be enough?</div>
<div><br>
</div>
<div>Thanks for your attention.</div>
<div>Regards</div>
<div>AR</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> </div>
<div><br>
</div>
<div><br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>Analabha Roy<br>
</div>
<div>Assistant Professor</div>
<div><a href="http://www.buruniv.ac.in/academics/department/physics" target="_blank">Department of Physics</a></div>
<div><a href="http://www.buruniv.ac.in/" target="_blank">The University of Burdwan</a></div>
<div>Golapbag Campus, Barddhaman 713104</div>
<div>West Bengal, India</div>
<div>Emails: <a href="mailto:daneel@utexas.edu" target="_blank">daneel@utexas.edu</a>,
<a href="mailto:aroy@phys.buruniv.ac.in" target="_blank">aroy@phys.buruniv.ac.in</a>,
<a href="mailto:hariseldon99@gmail.com" target="_blank">hariseldon99@gmail.com</a><br>
<div><font face="tahoma, sans-serif">Webpage: <a href="http://www.ph.utexas.edu/~daneel/" target="_blank">
http://www.ph.utexas.edu/~daneel/</a></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Analabha Roy<br></div><div>Assistant Professor</div><div><a href="http://www.buruniv.ac.in/academics/department/physics" target="_blank">Department of Physics</a></div><div><a href="http://www.buruniv.ac.in/" target="_blank">The University of Burdwan</a></div><div>Golapbag Campus, Barddhaman 713104</div><div>West Bengal, India</div><div>Emails: <a href="mailto:daneel@utexas.edu" target="_blank">daneel@utexas.edu</a>, <a href="mailto:aroy@phys.buruniv.ac.in" target="_blank">aroy@phys.buruniv.ac.in</a>, <a href="mailto:hariseldon99@gmail.com" target="_blank">hariseldon99@gmail.com</a><br><div><font face="tahoma, sans-serif">Webpage: <a href="http://www.ph.utexas.edu/~daneel/" target="_blank">http://www.ph.utexas.edu/~daneel/</a></font></div></div></div></div></div></div>