<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:等线;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@等线";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:宋体;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:等线;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=ZH-CN link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-size:10.5pt;font-family:等线'>Well, this is indeed the point. We didn’t set <b>ConstrainDevices=yes </b>in cgroup.conf. After adding this, gpu restriction works as expected.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.5pt;font-family:等线'>But what is the relation between gpu restriction and cgroup? I never heard that cgroup can limit gpu card usage. Isn’t it a feature of cuda or nvidia driver? <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.5pt;font-family:等线'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:等线'>发件人<span lang=EN-US>:</span></span></b><span lang=EN-US style='font-size:11.0pt;font-family:等线'> Sean Maxwell <stm@case.edu> <br></span><b><span style='font-size:11.0pt;font-family:等线'>发送时间<span lang=EN-US>:</span></span></b><span lang=EN-US style='font-size:11.0pt;font-family:等线'> 2022</span><span style='font-size:11.0pt;font-family:等线'>年<span lang=EN-US>3</span>月<span lang=EN-US>23</span>日<span lang=EN-US> 23:05<br></span><b>收件人<span lang=EN-US>:</span></b><span lang=EN-US> Slurm User Community List <slurm-users@lists.schedmd.com><br></span><b>主题<span lang=EN-US>:</span></b><span lang=EN-US> Re: [slurm-users] how to locate the problem when slurm failed to restrict gpu usage of user jobs<o:p></o:p></span></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><div><div><p class=MsoNormal><span lang=EN-US>Hi,<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US>If you are using cgroups for task/process management, you should verify that your /etc/slurm/cgroup.conf has the following line:<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US>ConstrainDevices=yes<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US>I'm not sure about the missing environment variable, but the absence of the above in cgroup.conf is one way the GPU devices can be unconstrained in the jobs.<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US>-Sean<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div></div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><div><div><p class=MsoNormal><span lang=EN-US>On Wed, Mar 23, 2022 at 10:46 AM <<a href="mailto:taleintervenor@sjtu.edu.cn" target="_blank">taleintervenor@sjtu.edu.cn</a>> wrote:<o:p></o:p></span></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Hi, all:<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>We found a problem that slurm job with argument such as <b>--gres gpu:1 </b>didn</span>’<span lang=EN-US>t be restricted with gpu usage, user still can see all gpu card on allocated nodes.<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Our gpu node has 4 cards with their gres.conf to be:<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>> cat /etc/slurm/gres.conf<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Name=gpu Type=NVlink_A100_40GB File=/dev/nvidia0 CPUs=0-15<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Name=gpu Type=NVlink_A100_40GB File=/dev/nvidia1 CPUs=16-31<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Name=gpu Type=NVlink_A100_40GB File=/dev/nvidia2 CPUs=32-47<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Name=gpu Type=NVlink_A100_40GB File=/dev/nvidia3 CPUs=48-63<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>And for test, we submit simple job batch like:<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>#!/bin/bash<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>#SBATCH --job-name=test<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>#SBATCH --partition=a100<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>#SBATCH --nodes=1<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>#SBATCH --ntasks=6<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>#SBATCH --gres=gpu:1<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>#SBATCH --reservation="gpu test"<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>hostname<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>nvidia-smi<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>echo end<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Then in the out file the nvidia-smi showed all 4 gpu cards. But we expect to see only 1 allocated gpu card.<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Official document of slurm said it will set <b>CUDA_VISIBLE_DEVICES </b>env var to restrict the gpu card available to user. But we didn</span>’<span lang=EN-US>t find such variable exists in job environment. We only confirmed it do exist in prolog script environment by adding debug command </span>“<span lang=EN-US>echo $CUDA_VISIBLE_DEVICES</span>”<span lang=EN-US> to slurm prolog script.<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>So how do slurm co-operate with nvidia tools to make job user only see its allocated gpu card? What is the requirement on nvidia gpu drivers, CUDA toolkit or any other part to help slurm correctly restrict the gpu usage?<o:p></o:p></span></p></div></div></blockquote></div></div></body></html>