<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
What does sacctmgr show for the user you added to have access to the QoS, and what does Slurm show for the partition config?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
sacctmgr show account withassoc -p</div>
<div style=""><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">scontrol show part </span><span style="display: inline !important; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">gpu-rtx6000-2</span></div>
<div style=""><span style="display: inline !important; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style=""><span style="display: inline !important; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Sean</span></div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Amjad Syed <amjadcsu@gmail.com><br>
<b>Sent:</b> Tuesday, 31 August 2021 17:03<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] [EXT] User association with partition and Qos</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div class="x_mc-ip-hide">
<div style="color:#000000; font-size:12px; text-align:left; font-family:Helvetica,Arial,sans-serif">
<strong>
<table border="0" cellspacing="0" cellpadding="0" style="width:100%; float:left">
<tbody>
<tr>
<td style="color:red"><b>External email: </b>Please exercise caution</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
Hello me again
<div><br>
</div>
<div>Just found out that when our slurmctld restarts all qos are gone. </div>
<div><br>
</div>
<div>I mean users who have association with the qos can not submit job with sbatch, they get error as</div>
<div><br>
</div>
<div><span style="color:rgb(32,31,30); font-family:Calibri,sans-serif; font-size:14.6667px">sbatch: error: Batch job submission failed: Invalid qos specification</span><br>
</div>
<div><span style="color:rgb(32,31,30); font-family:Calibri,sans-serif; font-size:14.6667px"><br>
</span></div>
<div><span style="color:rgb(32,31,30); font-family:Calibri,sans-serif; font-size:14.6667px"><br>
</span></div>
<div><span style="color:rgb(32,31,30); font-family:Calibri,sans-serif; font-size:14.6667px">Do we need to make anymore changes in slurm.conf so that qos becomes permanent ?</span></div>
<div><span style="color:rgb(32,31,30); font-family:Calibri,sans-serif; font-size:14.6667px"><br>
</span></div>
<div><span style="color:rgb(32,31,30); font-family:Calibri,sans-serif; font-size:14.6667px">Amjad</span></div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Fri, Aug 27, 2021 at 3:32 PM Amjad Syed <<a href="mailto:amjadcsu@gmail.com">amjadcsu@gmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">Hi Sean,
<div><br>
</div>
<div>Thanks for the suggestion, seems to work now. </div>
<div><br>
</div>
<div>Majid</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Fri, Aug 27, 2021 at 12:56 PM Sean Crosby <<a href="mailto:scrosby@unimelb.edu.au" target="_blank">scrosby@unimelb.edu.au</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Amjad,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Make sure you have qos in the config entry AccountingStorageEnforce<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
e.g.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
AccountingStorageEnforce=associations,limits,qos,safe<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Sean<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_gmail-m_641854653930490418gmail-m_-5581969826299184476appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_641854653930490418gmail-m_-5581969826299184476divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Amjad Syed <<a href="mailto:amjadcsu@gmail.com" target="_blank">amjadcsu@gmail.com</a>><br>
<b>Sent:</b> Friday, 27 August 2021 20:28<br>
<b>To:</b> <a href="mailto:slurm-users@schedmd.com" target="_blank">slurm-users@schedmd.com</a> <<a href="mailto:slurm-users@schedmd.com" target="_blank">slurm-users@schedmd.com</a>><br>
<b>Subject:</b> [EXT] [slurm-users] User association with partition and Qos</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div style="color:rgb(0,0,0); font-size:12px; text-align:left; font-family:Helvetica,Arial,sans-serif">
<strong>
<table border="0" cellspacing="0" cellpadding="0" style="width:100%; float:left">
<tbody>
<tr>
<td style="color:red"><b>External email: </b>Please exercise caution</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
Hello all<br>
<div><br>
</div>
<div>We are having an issue understanding user association and partition.</div>
<div><br>
</div>
<div>Currently we have a partition with 30 GPU cards .</div>
<div><br>
</div>
<div>We have defined a qos gpu-rtx that allows user to reserve 2 cards </div>
<div><br>
</div>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">sacctmgr show qos gpu-rtx format=MaxTRESPU%60</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>
</span>MaxTRESPU<span> </span></span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span> </span>-----------------------------------------------------<span> </span></span></p>
<div><span style="font-variant-ligatures:no-common-ligatures; color:rgb(0,0,0); font-family:Menlo; font-size:11px"> </span><span style="font-variant-ligatures:no-common-ligatures; color:rgb(0,0,0); font-family:Menlo; font-size:11px">cpu=96,gres/gpu=2</span><span style="font-variant-ligatures:no-common-ligatures; color:rgb(0,0,0); font-family:Menlo; font-size:11px"> </span> </div>
<div>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px; font-style:normal; font-variant:normal; font-weight:normal; font-stretch:normal; line-height:normal">
We have defined a user test that is assoc with this qos</p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span></span></span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>sacctmgr show assoc user=test format=user,qos</span></span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>Qos</span></span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span>gpu-rtx</span></span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span><br>
</span></span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; line-height:normal">
Now we define another qos gpu-rtx-reserved that allows gpu=8</p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; line-height:normal">
<br>
</p>
</div>
<div>
<p style="font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; margin:0px; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">sacctmgr show qos gpu-rtx-reserved format=MaxTRESPU%60</span></p>
<p style="font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; margin:0px; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span> </span>MaxTRESPU<span> </span></span></p>
<p style="font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; margin:0px; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><span> </span>-----------------------------------------------------<span> </span></span></p>
<div><span style="font-variant-ligatures:no-common-ligatures; color:rgb(0,0,0); font-family:Menlo; font-size:11px"> </span><span style="font-variant-ligatures:no-common-ligatures; color:rgb(0,0,0); font-family:Menlo; font-size:11px">cpu=192,gres/gpu=8</span> </div>
</div>
<div><br>
</div>
<div>User test is not associated with gpu-rtx-reserved qos. So he should not be able to use more then gpu=2 .</div>
<div>Both of these qos are now in slurm.conf for the partition</div>
<div><br>
</div>
<div>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">parrtitionName=gpu-rtx6000-2 State=UP Nodes=g[15-29] MaxNodes=9 MaxTime=168:00:00 DefMemPerCPU=3996 AllowQos=gpu-rtx,gpu-rtx-reserved</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">But we found out that even though user is not assoc with gpu-rtx-reserved if the user uses gpu-rtx-reserved in his slurm script , he can reserve 8 gpu cards </span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">So our question is , can the users assoc with one partition qos can use the other qos in the partition even if they are not associated with it . or in other words , we can only define one partition qos
and not more then one.?</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">Hope i was able to explain ?</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">Any advice if we want partition to use more then one qos with different limits and users associated with one qos should not use other qos ?</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">Majid</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px; font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
</div>
<div>
<p style="font-variant-numeric:normal; font-variant-east-asian:normal; font-stretch:normal; font-size:11px; line-height:normal; font-family:Menlo; margin:0px; color:rgb(0,0,0)">
<br>
</p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</body>
</html>