<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div><span data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="x_elementToProof elementToProof" style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;">I
have the following lines in my <code>slurm.conf</code></span></div>
<div data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="x_elementToProof" style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;">
<pre data-darkreader-inline-bgcolor="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="" data-darkreader-inline-border-left="" style="font-size: 13px; background-color: rgb(248, 248, 248); border-radius: 3px; max-width: 100%; margin: 15px 0px; padding: 6px 10px; border: 1px solid rgb(204, 204, 204); line-height: 19px; --darkreader-inline-bgcolor: #faefbe; --darkreader-inline-border-top: #f7e188; --darkreader-inline-border-right: #f7e188; --darkreader-inline-border-bottom: #f7e188; --darkreader-inline-border-left: #f7e188;"><code data-darkreader-inline-bgcolor="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="" data-darkreader-inline-border-left="" class="x_ContentPasted0 x_ContentPasted1" style="background-color: rgb(248, 248, 248); display: block; max-width: 100%; margin: 0px; border: 0px none transparent; --darkreader-inline-bgcolor: #faefbe; --darkreader-inline-border-top: transparent; --darkreader-inline-border-right: transparent; --darkreader-inline-border-bottom: transparent; --darkreader-inline-border-left: transparent;">ProctrackType=proctrack/cgroup<div class="x_ContentPasted1">TaskPlugin=task/cgroup,task/affinity</div>PrologFlags=Contain<br></code></pre>
and the following lines in my <code>cgroups.conf</code></div>
<div data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="x_elementToProof elementToProof" style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;">
<pre data-darkreader-inline-bgcolor="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="" data-darkreader-inline-border-left="" style="font-size: 13px; background-color: rgb(248, 248, 248); border-radius: 3px; max-width: 100%; margin: 15px 0px; padding: 6px 10px; border: 1px solid rgb(204, 204, 204); line-height: 19px; --darkreader-inline-bgcolor: #faefbe; --darkreader-inline-border-top: #f7e188; --darkreader-inline-border-right: #f7e188; --darkreader-inline-border-bottom: #f7e188; --darkreader-inline-border-left: #f7e188;"><code data-darkreader-inline-bgcolor="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="" data-darkreader-inline-border-left="" class="x_ContentPasted0 x_ContentPasted1 x_ContentPasted2 x_ContentPasted3" style="background-color: rgb(248, 248, 248); display: block; max-width: 100%; margin: 0px; border: 0px none transparent; --darkreader-inline-bgcolor: #faefbe; --darkreader-inline-border-top: transparent; --darkreader-inline-border-right: transparent; --darkreader-inline-border-bottom: transparent; --darkreader-inline-border-left: transparent;">CgroupPlugin=cgroup/v2<div class="x_ContentPasted3">CgroupAutomount=yes</div><div class="x_ContentPasted3">ConstrainDevices=yes</div><div class="x_ContentPasted3">ConstrainCores=yes</div>ConstrainRAMSpace=yes</code></pre>
My small cluster consists of three servers running Slurm 22.05.8, each of which allows users to directly
<code>ssh</code> to them. One of the servers is used as both a head node and a compute node.</div>
<div data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="x_elementToProof" style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;">
<br>
</div>
<div data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="x_elementToProof" style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;">
When I run a job with <code>sbatch</code> or <code>srun</code>, I see the GPU constraints are enforced correctly, but all the CPU cores and memory are visible to the user.</div>
<div data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="x_elementToProof" style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;">
<br>
</div>
<div data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="x_elementToProof x_ContentPasted4" style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;">
Is <code>PrologFlags=Contain</code> compatible with <code>cgroup/v2</code>? If yes, what could be the cause of this issue?</div>
<span data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="x_elementToProof x_ContentPasted4" style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;"></span>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;" data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="elementToProof">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;" data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="elementToProof ContentPasted0">
The full contents of my <code class="ContentPasted0">slurm.conf</code>, <code class="ContentPasted0">
gres.conf</code>, and <code class="ContentPasted0">cgroup.conf</code> files are shown below.</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;" data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="elementToProof ContentPasted0">
<pre data-darkreader-inline-bgcolor="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="" data-darkreader-inline-border-left="" style="font-size: 13px; background-color: rgb(248, 248, 248); border-radius: 3px; max-width: 100%; margin: 15px 0px; padding: 6px 10px; border: 1px solid rgb(204, 204, 204); line-height: 19px; --darkreader-inline-bgcolor: #faefbe; --darkreader-inline-border-top: #f7e188; --darkreader-inline-border-right: #f7e188; --darkreader-inline-border-bottom: #f7e188; --darkreader-inline-border-left: #f7e188;"><code data-darkreader-inline-bgcolor="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="" data-darkreader-inline-border-left="" class="x_ContentPasted0 x_ContentPasted1 ContentPasted1 ContentPasted4" style="background-color: rgb(248, 248, 248); display: block; max-width: 100%; margin: 0px; border: 0px none transparent; --darkreader-inline-bgcolor: #faefbe; --darkreader-inline-border-top: transparent; --darkreader-inline-border-right: transparent; --darkreader-inline-border-bottom: transparent; --darkreader-inline-border-left: transparent;">ClusterName=sportlab<div><br class="ContentPasted4"></div><div class="ContentPasted4">ControlMachine=kaveh.usc.edu</div><div class="ContentPasted4">SlurmUser=root</div><div class="ContentPasted4">SlurmdUser=root</div><div class="ContentPasted4">SlurmctldPort=6817</div><div class="ContentPasted4">SlurmdPort=6818</div><div class="ContentPasted4">AuthType=auth/munge</div><div class="ContentPasted4">StateSaveLocation=/var/lib/slurm</div><div class="ContentPasted4">SlurmdSpoolDir=/var/spool/slurm</div><div><br class="ContentPasted4"></div><div class="ContentPasted4">SlurmctldPidFile=/var/run/slurmctld.pid</div><div class="ContentPasted4">SlurmdPidFile=/var/run/slurmd.pid</div><div><br class="ContentPasted4"></div><div class="ContentPasted4">ProctrackType=proctrack/cgroup</div><div class="ContentPasted4">TaskPlugin=task/cgroup,task/affinity</div><div class="ContentPasted4">PrologFlags=Contain</div><div><br class="ContentPasted4"></div><div class="ContentPasted4">MessageTimeout=30</div><div class="ContentPasted4">SlurmctldTimeout=300</div><div class="ContentPasted4">SlurmdTimeout=300</div><div class="ContentPasted4">InactiveLimit=0</div><div class="ContentPasted4">MinJobAge=300</div><div class="ContentPasted4">KillWait=30</div><div class="ContentPasted4">Waittime=0</div><div><br class="ContentPasted4"></div><div class="ContentPasted4">SchedulerType=sched/backfill</div><div class="ContentPasted4">SchedulerParameters=preemptstrict_order,preempt_reorder_count=3,max_rpc_cnt=160</div><div class="ContentPasted4">SelectType=select/cons_res</div><div class="ContentPasted4">SelectTypeParameters=CR_Core_Memory</div><div class="ContentPasted4">GresTypes=gpu</div><div><br class="ContentPasted4"></div><div class="ContentPasted4">PreemptType=preempt/partition_prio</div><div class="ContentPasted4">PreemptExemptTime=-1</div><div class="ContentPasted4">PreemptMode=CANCEL</div><div><br class="ContentPasted4"></div><div class="ContentPasted4">PriorityType=priority/multifactor</div><div class="ContentPasted4">PriorityDecayHalfLife=30-0</div><div class="ContentPasted4">PriorityFavorSmall=NO</div><div><br class="ContentPasted4"></div><div class="ContentPasted4">PriorityWeightPartition=1000</div><div class="ContentPasted4">PriorityWeightJobSize=1000</div><div class="ContentPasted4">PriorityMaxAge=14-0</div><div class="ContentPasted4">PriorityWeightQOS=1000</div><div class="ContentPasted4">PropagateResourceLimitsExcept=MEMLOCK</div><div class="ContentPasted4">PriorityFlags=FAIR_TREE</div><div><br class="ContentPasted4"></div><div class="ContentPasted4">SlurmctldDebug=verbose</div><div class="ContentPasted4">SlurmctldLogFile=/var/log/slurmctld.log</div><div class="ContentPasted4">SlurmdDebug=verbose</div><div class="ContentPasted4">SlurmdLogFile=/var/log/slurmd.log</div><div class="ContentPasted4">JobCompType=jobcomp/none</div><div><br class="ContentPasted4"></div><div class="ContentPasted4">MpiDefault=none</div><div><br class="ContentPasted4"></div><div class="ContentPasted4">NodeName=kaveh Gres=gpu:a6000:8 CPUs=256 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=1031870 Feature=a6000</div><div class="ContentPasted4">NodeName=arvand Gres=gpu:2080ti:4 CPUs=24 Boards=1 SocketsPerBoard=1 CoresPerSocket=12 ThreadsPerCore=2 RealMemory=128488 Feature=2080ti</div><div class="ContentPasted4">NodeName=haraz Gres=gpu:1080ti:4 CPUs=12 Boards=1 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=80350 Feature=1080ti</div><div><br class="ContentPasted4"></div><div class="ContentPasted4">PartitionName=gpu Nodes=kaveh,arvand,haraz Default=YES DefaultTime=12:00:00 MaxTime=UNLIMITED OverSubscribe=NO State=UP DefMemPerCPU=2048 GraceTime=10</div></code></pre>
<pre data-darkreader-inline-bgcolor="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="" data-darkreader-inline-border-left="" style="font-size: 13px; background-color: rgb(248, 248, 248); border-radius: 3px; max-width: 100%; margin: 15px 0px; padding: 6px 10px; border: 1px solid rgb(204, 204, 204); line-height: 19px; --darkreader-inline-bgcolor: #faefbe; --darkreader-inline-border-top: #f7e188; --darkreader-inline-border-right: #f7e188; --darkreader-inline-border-bottom: #f7e188; --darkreader-inline-border-left: #f7e188;"><code data-darkreader-inline-bgcolor="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="" data-darkreader-inline-border-left="" class="x_ContentPasted0 x_ContentPasted1 ContentPasted2" style="background-color: rgb(248, 248, 248); display: block; max-width: 100%; margin: 0px; border: 0px none transparent; --darkreader-inline-bgcolor: #faefbe; --darkreader-inline-border-top: transparent; --darkreader-inline-border-right: transparent; --darkreader-inline-border-bottom: transparent; --darkreader-inline-border-left: transparent;">NodeName=kaveh Name=gpu Type=a6000 File=/dev/nvidia[0-7]<div>NodeName=arvand Name=gpu Type=2080ti File=/dev/nvidia[0-3]</div><div>NodeName=haraz Name=gpu Type=1080ti File=/dev/nvidia[0-3]</div></code></pre>
<pre data-darkreader-inline-bgcolor="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="" data-darkreader-inline-border-left="" style="font-size: 13px; background-color: rgb(248, 248, 248); border-radius: 3px; max-width: 100%; margin: 15px 0px; padding: 6px 10px; border: 1px solid rgb(204, 204, 204); line-height: 19px; --darkreader-inline-bgcolor: #faefbe; --darkreader-inline-border-top: #f7e188; --darkreader-inline-border-right: #f7e188; --darkreader-inline-border-bottom: #f7e188; --darkreader-inline-border-left: #f7e188;"><code data-darkreader-inline-bgcolor="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="" data-darkreader-inline-border-left="" class="x_ContentPasted0 x_ContentPasted1 ContentPasted3 ContentPasted5" style="background-color: rgb(248, 248, 248); display: block; max-width: 100%; margin: 0px; border: 0px none transparent; --darkreader-inline-bgcolor: #faefbe; --darkreader-inline-border-top: transparent; --darkreader-inline-border-right: transparent; --darkreader-inline-border-bottom: transparent; --darkreader-inline-border-left: transparent;">CgroupPlugin=cgroup/v2<div class="ContentPasted5">CgroupAutomount=yes </div><div class="ContentPasted5">ConstrainDevices=yes</div><div class="ContentPasted5">ConstrainCores=yes </div><div class="ContentPasted5">ConstrainRAMSpace=yes</div><div class="ContentPasted5">#AllowedDevicesFile=/etc/slurm/cgroup_allowed_devices_file.conf</div></code></pre>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); --darkreader-inline-color: #3c3836; --darkreader-inline-bgcolor: #fbf1c7;" data-darkreader-inline-color="" data-darkreader-inline-bgcolor="" class="elementToProof ContentPasted0">
Thank you!<br>
</div>
<div class="elementToProof">
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); --darkreader-inline-color: #3c3836;" data-darkreader-inline-color="">
<br>
</div>
<div id="Signature">
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); --darkreader-inline-color: #3c3836;" data-darkreader-inline-color="">
Best Regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); --darkreader-inline-color: #3c3836;" data-darkreader-inline-color="">
Mahdi Nazemi<br>
</div>
</div>
</div>
</div>
</body>
</html>