<div dir="ltr">Just a correction<div><br></div><div>We use </div><div>sacctmgr modify user=<username> set qos+=gpu-rtx6000-2</div><div><br></div><div>Amjad</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 31, 2021 at 10:17 AM Amjad Syed <<a href="mailto:amjadcsu@gmail.com">amjadcsu@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Sean<div><br></div><div>We have been adding by using the following command</div><div><br></div><div>sacctmgr modify user set qos+=gpu-rtx-reserved</div><div><br></div><div>We have a single account that is associated with all our users and root account for admin</div><div><br></div><div><br></div><div><br></div><div>Is that the issue, we need to associate user with account?</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 31, 2021 at 9:38 AM Sean Crosby <<a href="mailto:scrosby@unimelb.edu.au" target="_blank">scrosby@unimelb.edu.au</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">




<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Amjad,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
AccountingStorageUser is the user used to connect to the accounting database. If you have it defined in slurm.conf, it is ignored.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
>From the output you showed, it says the user <span style="font-variant-ligatures:no-common-ligatures">
cjr13geu</span> in the cluster <span style="font-variant-ligatures:no-common-ligatures">
uea_cluster</span> has access to the QoS.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
How are you adding the QoS to other users? The way you would do it would be</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
sacctmgr modify account <accountname> user=<username> set qos+=<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">gpu-rtx-reserved</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">or</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">sacctmgr modify account <accountname> set qos+=</span><span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">gpu-rtx-reserved</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">if you want to give it to every user in <accountname></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"></span><span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"></span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Sean<br>
</div>
<div id="gmail-m_-1640889757752330806gmail-m_5884111995180100821appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-1640889757752330806gmail-m_5884111995180100821divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Amjad Syed <<a href="mailto:amjadcsu@gmail.com" target="_blank">amjadcsu@gmail.com</a>><br>
<b>Sent:</b> Tuesday, 31 August 2021 17:46<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> Re: [slurm-users] [EXT] User association with partition and Qos</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div style="color:rgb(0,0,0);font-size:12px;text-align:left;font-family:Helvetica,Arial,sans-serif">
<strong>
<table border="0" cellspacing="0" cellpadding="0" style="width:100%;float:left">
<tbody>
<tr>
<td style="color:red"><b>External email: </b>Please exercise caution</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
Hi Sean
<div><br>
</div>
<div>Here is the output for gpu-rtx-reserved qos</div>
<div><br>
</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">sacctmgr show account withassoc -p | grep gpu-rtx-reserved</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">default|default|default|uea_cluster||cjr13geu|1|||||||||||||||gpu,gpu-k40-1,gpu-rtx,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>gpu-rtx-reserved</b></span><span style="font-variant-ligatures:no-common-ligatures">,hmem,ht,uea_def_qos|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">sontrol show part gpu-rtx6000-2</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">PartitionName=gpu-rtx6000-2</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span>  
</span>AllowGroups=ALL AllowAccounts=ALL AllowQos=gpu-rtx,gpu-rtx-reserved,jakeuea</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span>  
</span>AllocNodes=ALL Default=NO QoS=N/A</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span>  
</span>DefaultTime=1-00:00:00 DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span>  
</span>MaxNodes=9 MaxTime=7-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span>  
</span>Nodes=g[15-29]</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span>  
</span>PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span>  
</span>OverTimeLimit=NONE PreemptMode=GANG,SUSPEND</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span>  
</span>State=UP TotalCPUs=720 TotalNodes=15 SelectTypeParameters=NONE</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span>  
</span>JobDefaults=(null)</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>   </span>DefMemPerCPU=3996 MaxMemPerNode=UNLIMITED</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;font-size:small;font-variant-ligatures:normal">On a different note we have the following in  slurm.conf</span><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;font-size:small;font-variant-ligatures:normal"><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">AccountingStorageUser=slurm</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">But we have been adding qos and assigning users as root ? Can this be an issue</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">Amjad</span></p>
</div>
</div>
<br>
<div>
<div dir="ltr">On Tue, Aug 31, 2021 at 8:22 AM Sean Crosby <<a href="mailto:scrosby@unimelb.edu.au" target="_blank">scrosby@unimelb.edu.au</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
What does sacctmgr show for the user you added to have access to the QoS, and what does Slurm show for the partition config?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
sacctmgr show account withassoc -p</div>
<div><span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">scontrol show part </span><span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);display:inline">gpu-rtx6000-2</span></div>
<div><span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);display:inline"><br>
</span></div>
<div><span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);display:inline">Sean</span></div>
<div id="gmail-m_-1640889757752330806gmail-m_5884111995180100821x_gmail-m_9027024710706152232appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-1640889757752330806gmail-m_5884111995180100821x_gmail-m_9027024710706152232divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>>
 on behalf of Amjad Syed <<a href="mailto:amjadcsu@gmail.com" target="_blank">amjadcsu@gmail.com</a>><br>
<b>Sent:</b> Tuesday, 31 August 2021 17:03<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> Re: [slurm-users] [EXT] User association with partition and Qos</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div style="color:rgb(0,0,0);font-size:12px;text-align:left;font-family:Helvetica,Arial,sans-serif">
<strong>
<table border="0" cellspacing="0" cellpadding="0" style="width:100%;float:left">
<tbody>
<tr>
<td style="color:red"><b>External email: </b>Please exercise caution</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
Hello me again
<div><br>
</div>
<div>Just found out that when our slurmctld restarts all qos are gone. </div>
<div><br>
</div>
<div>I mean users who have association with the qos can not submit job with sbatch, they get error as</div>
<div><br>
</div>
<div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px">sbatch: error: Batch job submission failed: Invalid qos specification</span><br>
</div>
<div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px"><br>
</span></div>
<div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px"><br>
</span></div>
<div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px">Do we need to make anymore changes in slurm.conf so that qos becomes permanent ?</span></div>
<div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px"><br>
</span></div>
<div><span style="color:rgb(32,31,30);font-family:Calibri,sans-serif;font-size:14.6667px">Amjad</span></div>
</div>
<br>
<div>
<div dir="ltr">On Fri, Aug 27, 2021 at 3:32 PM Amjad Syed <<a href="mailto:amjadcsu@gmail.com" target="_blank">amjadcsu@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi Sean,
<div><br>
</div>
<div>Thanks for the suggestion, seems to work now. </div>
<div><br>
</div>
<div>Majid</div>
</div>
<br>
<div>
<div dir="ltr">On Fri, Aug 27, 2021 at 12:56 PM Sean Crosby <<a href="mailto:scrosby@unimelb.edu.au" target="_blank">scrosby@unimelb.edu.au</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Amjad,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Make sure you have qos in the config entry AccountingStorageEnforce<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
e.g.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
AccountingStorageEnforce=associations,limits,qos,safe<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Sean<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-1640889757752330806gmail-m_5884111995180100821x_gmail-m_9027024710706152232x_gmail-m_641854653930490418gmail-m_-5581969826299184476appendonsend">
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-1640889757752330806gmail-m_5884111995180100821x_gmail-m_9027024710706152232x_gmail-m_641854653930490418gmail-m_-5581969826299184476divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Amjad Syed <<a href="mailto:amjadcsu@gmail.com" target="_blank">amjadcsu@gmail.com</a>><br>
<b>Sent:</b> Friday, 27 August 2021 20:28<br>
<b>To:</b> <a href="mailto:slurm-users@schedmd.com" target="_blank">slurm-users@schedmd.com</a> <<a href="mailto:slurm-users@schedmd.com" target="_blank">slurm-users@schedmd.com</a>><br>
<b>Subject:</b> [EXT] [slurm-users] User association with partition and Qos</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div style="color:rgb(0,0,0);font-size:12px;text-align:left;font-family:Helvetica,Arial,sans-serif">
<strong>
<table border="0" cellspacing="0" cellpadding="0" style="width:100%;float:left">
<tbody>
<tr>
<td style="color:red"><b>External email: </b>Please exercise caution</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
Hello all<br>
<div><br>
</div>
<div>We are having an issue understanding user association and partition.</div>
<div><br>
</div>
<div>Currently we have a partition with 30 GPU cards .</div>
<div><br>
</div>
<div>We have defined a qos gpu-rtx that allows user to reserve 2 cards </div>
<div><br>
</div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">sacctmgr show qos gpu-rtx format=MaxTRESPU%60</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>                                                  
</span>MaxTRESPU<span> </span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>       </span>-----------------------------------------------------<span> </span></span></p>
<div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px">                                           </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px">cpu=96,gres/gpu=2</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px"> </span> </div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;line-height:normal">
We have defined a user test that is assoc with this qos</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span></span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>sacctmgr show assoc user=test format=user,qos</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>Qos</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>gpu-rtx</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal">
Now we define another qos  gpu-rtx-reserved  that allows gpu=8</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal">
<br>
</p>
</div>
<div>
<p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">sacctmgr show qos gpu-rtx-reserved format=MaxTRESPU%60</span></p>
<p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>                                                   </span>MaxTRESPU<span> </span></span></p>
<p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>       </span>-----------------------------------------------------<span> </span></span></p>
<div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px">                                           </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px">cpu=192,gres/gpu=8</span> </div>
</div>
<div><br>
</div>
<div>User test is not associated with gpu-rtx-reserved qos. So he should not be able to use more then gpu=2 .</div>
<div>Both of these qos are now in slurm.conf for the partition</div>
<div><br>
</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">parrtitionName=gpu-rtx6000-2 State=UP Nodes=g[15-29] MaxNodes=9 MaxTime=168:00:00 DefMemPerCPU=3996 AllowQos=gpu-rtx,gpu-rtx-reserved</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">But we found out that even though user is not assoc with gpu-rtx-reserved if the user uses gpu-rtx-reserved  in his slurm script , he can reserve 8 gpu cards </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">So our question is , can the users assoc with one partition qos can use the other qos in the partition  even if they are not associated with it . or in other words , we can only define one partition qos
 and not more then one.?</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">Hope i was able to explain ?</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">Any advice if we want partition to use more then one qos with different limits and users associated with one qos should not use other qos ?</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">Majid</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
</div>
<div>
<p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)">
<br>
</p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>

</blockquote></div>
</blockquote></div>