<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Amjad,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Make sure you have qos in the config entry AccountingStorageEnforce<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
e.g.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
AccountingStorageEnforce=associations,limits,qos,safe<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Sean<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Amjad Syed <amjadcsu@gmail.com><br>
<b>Sent:</b> Friday, 27 August 2021 20:28<br>
<b>To:</b> slurm-users@schedmd.com <slurm-users@schedmd.com><br>
<b>Subject:</b> [EXT] [slurm-users] User association with partition and Qos</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div class="x_mc-ip-hide">
<div style="color:#000000; font-size:12px; text-align:left; font-family:Helvetica,Arial,sans-serif">
<strong>
<table border="0" cellspacing="0" cellpadding="0" style="width:100%; float:left">
<tbody>
<tr>
<td style="color:red"><b>External email: </b>Please exercise caution</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
Hello all<br>
<div><br>
</div>
<div>We are having an issue understanding user association and partition.</div>
<div><br>
</div>
<div>Currently we have a partition with 30 GPU cards .</div>
<div><br>
</div>
<div>We have defined a qos gpu-rtx that allows user to reserve 2 cards </div>
<div><br>
</div>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">sacctmgr show qos gpu-rtx format=MaxTRESPU%60</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space">
</span>MaxTRESPU<span class="x_gmail-Apple-converted-space"> </span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space">
</span>-----------------------------------------------------<span class="x_gmail-Apple-converted-space"> </span></span></p>
<div><span class="x_gmail-Apple-converted-space" style="font-variant-ligatures:no-common-ligatures; color:rgb(0,0,0); font-family:Menlo; font-size:11px"> </span><span style="font-variant-ligatures:no-common-ligatures; color:rgb(0,0,0); font-family:Menlo; font-size:11px">cpu=96,gres/gpu=2</span><span class="x_gmail-Apple-converted-space" style="font-variant-ligatures:no-common-ligatures; color:rgb(0,0,0); font-family:Menlo; font-size:11px"> </span> </div>
<div>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space"><br>
</span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space"><br>
</span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space"><br>
</span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-style:normal; font-variant:normal; font-weight:normal; font-stretch:normal; line-height:normal">
We have defined a user test that is assoc with this qos</p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space"></span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space"><br>
</span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space">sacctmgr show assoc user=test format=user,qos</span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space"><br>
</span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space">Qos</span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space">gpu-rtx</span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space"><br>
</span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space"><br>
</span></span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; line-height:normal">
Now we define another qos gpu-rtx-reserved that allows gpu=8</p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; line-height:normal">
<br>
</p>
</div>
<div>
<p class="x_gmail-p1" style="font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; margin:0px; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">sacctmgr show qos gpu-rtx-reserved format=MaxTRESPU%60</span></p>
<p class="x_gmail-p1" style="font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; margin:0px; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space"> </span>MaxTRESPU<span class="x_gmail-Apple-converted-space"> </span></span></p>
<p class="x_gmail-p1" style="font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; margin:0px; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="x_gmail-Apple-converted-space"> </span>-----------------------------------------------------<span class="x_gmail-Apple-converted-space"> </span></span></p>
<div><span class="x_gmail-Apple-converted-space" style="font-variant-ligatures:no-common-ligatures; color:rgb(0,0,0); font-family:Menlo; font-size:11px"> </span><span style="font-variant-ligatures:no-common-ligatures; color:rgb(0,0,0); font-family:Menlo; font-size:11px">cpu=192,gres/gpu=8</span> </div>
</div>
<div><br>
</div>
<div>User test is not associated with gpu-rtx-reserved qos. So he should not be able to use more then gpu=2 .</div>
<div>Both of these qos are now in slurm.conf for the partition</div>
<div><br>
</div>
<div>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">parrtitionName=gpu-rtx6000-2 State=UP Nodes=g[15-29] MaxNodes=9 MaxTime=168:00:00 DefMemPerCPU=3996 AllowQos=gpu-rtx,gpu-rtx-reserved</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">But we found out that even though user is not assoc with gpu-rtx-reserved if the user uses gpu-rtx-reserved in his slurm script , he can reserve 8 gpu cards </span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">So our question is , can the users assoc with one partition qos can use the other qos in the partition even if they are not associated with it . or in other words , we can only define
one partition qos and not more then one.?</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">Hope i was able to explain ?</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">Any advice if we want partition to use more then one qos with different limits and users associated with one qos should not use other qos ?</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures">Majid</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p class="x_gmail-p1" style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span class="x_gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
</div>
<div>
<p class="x_gmail-p1" style="font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; margin:0px; color:rgb(0,0,0)">
<br>
</p>
</div>
</div>
</div>
</body>
</html>