<div dir="ltr">Hello me again<div><br></div><div>Just found out that when our slurmctld restarts all qos are gone. </div><div><br></div><div>I mean users who have association with the qos can not submit job with sbatch, they get error as</div><div><br></div><div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px">sbatch: error: Batch job submission failed: Invalid qos specification</span><br></div><div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px"><br></span></div><div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px"><br></span></div><div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px">Do we need to make anymore changes in slurm.conf so that qos becomes permanent ?</span></div><div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px"><br></span></div><div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px">Amjad</span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 27, 2021 at 3:32 PM Amjad Syed <<a href="mailto:amjadcsu@gmail.com">amjadcsu@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Sean,<div><br></div><div>Thanks for the suggestion, seems to work now. </div><div><br></div><div>Majid</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 27, 2021 at 12:56 PM Sean Crosby <<a href="mailto:scrosby@unimelb.edu.au" target="_blank">scrosby@unimelb.edu.au</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Amjad,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Make sure you have qos in the config entry AccountingStorageEnforce<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
e.g.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
AccountingStorageEnforce=associations,limits,qos,safe<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Sean<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_641854653930490418gmail-m_-5581969826299184476appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_641854653930490418gmail-m_-5581969826299184476divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Amjad Syed <<a href="mailto:amjadcsu@gmail.com" target="_blank">amjadcsu@gmail.com</a>><br>
<b>Sent:</b> Friday, 27 August 2021 20:28<br>
<b>To:</b> <a href="mailto:slurm-users@schedmd.com" target="_blank">slurm-users@schedmd.com</a> <<a href="mailto:slurm-users@schedmd.com" target="_blank">slurm-users@schedmd.com</a>><br>
<b>Subject:</b> [EXT] [slurm-users] User association with partition and Qos</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div style="color:rgb(0,0,0);font-size:12px;text-align:left;font-family:Helvetica,Arial,sans-serif">
<strong>
<table border="0" cellspacing="0" cellpadding="0" style="width:100%;float:left">
<tbody>
<tr>
<td style="color:red"><b>External email: </b>Please exercise caution</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
Hello all<br>
<div><br>
</div>
<div>We are having an issue understanding user association and partition.</div>
<div><br>
</div>
<div>Currently we have a partition with 30 GPU cards .</div>
<div><br>
</div>
<div>We have defined a qos gpu-rtx that allows user to reserve 2 cards </div>
<div><br>
</div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">sacctmgr show qos gpu-rtx format=MaxTRESPU%60</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>
</span>MaxTRESPU<span> </span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>
</span>-----------------------------------------------------<span> </span></span></p>
<div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px">cpu=96,gres/gpu=2</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px"> </span> </div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;line-height:normal">
We have defined a user test that is assoc with this qos</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span></span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>sacctmgr show assoc user=test format=user,qos</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>Qos</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>gpu-rtx</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal">
Now we define another qos gpu-rtx-reserved that allows gpu=8</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal">
<br>
</p>
</div>
<div>
<p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">sacctmgr show qos gpu-rtx-reserved format=MaxTRESPU%60</span></p>
<p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span> </span>MaxTRESPU<span> </span></span></p>
<p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span> </span>-----------------------------------------------------<span> </span></span></p>
<div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px">cpu=192,gres/gpu=8</span> </div>
</div>
<div><br>
</div>
<div>User test is not associated with gpu-rtx-reserved qos. So he should not be able to use more then gpu=2 .</div>
<div>Both of these qos are now in slurm.conf for the partition</div>
<div><br>
</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">parrtitionName=gpu-rtx6000-2 State=UP Nodes=g[15-29] MaxNodes=9 MaxTime=168:00:00 DefMemPerCPU=3996 AllowQos=gpu-rtx,gpu-rtx-reserved</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">But we found out that even though user is not assoc with gpu-rtx-reserved if the user uses gpu-rtx-reserved in his slurm script , he can reserve 8 gpu cards </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">So our question is , can the users assoc with one partition qos can use the other qos in the partition even if they are not associated with it . or in other words , we can only define
one partition qos and not more then one.?</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">Hope i was able to explain ?</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">Any advice if we want partition to use more then one qos with different limits and users associated with one qos should not use other qos ?</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">Majid</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
</div>
<div>
<p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)">
<br>
</p>
</div>
</div>
</div>
</div>
</blockquote></div>
</blockquote></div>