<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I thought about that, and on a GENERAL level I think that would work. But I was thinking more along the lines of the scenario I had described where the coordinator wanted control of what jobs from their account would run next, right now. So, it would be changing
the priority of jobs already assigned a priority...so this would be a post-fairshare-affected priority I'd be trying to change.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Basically, I was looking for the ability of the coordinator to change the priority of a job from their account after it has been calculated by slurm, to move it ahead of other jobs from the same account. Not ahead of all other jobs on the system...just ahead
of jobs from the same account. </div>
<div id="appendonsend"></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Renfro, Michael <Renfro@tntech.edu><br>
<b>Sent:</b> Wednesday, May 17, 2023 2:11 PM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] On the ability of coordinators</font>
<div> </div>
</div>
<div lang="EN-US" style="word-wrap:break-word">
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="border:0; display:table; width:100%; table-layout:fixed; border-collapse:seperate; float:none">
<tbody style="display:block">
<tr>
<td valign="middle" width="1px" cellpadding="7px 2px 7px 2px" style="padding: 7px 2px; background-color: rgb(166, 166, 166);">
</td>
<td valign="middle" width="100%" cellpadding="7px 5px 7px 15px" style="width: 100%; padding: 7px 5px 7px 15px; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; overflow-wrap: break-word; color: rgb(33, 33, 33); background-color: rgb(234, 234, 234);">
<div>You don't often get email from renfro@tntech.edu. <a href="https://aka.ms/LearnAboutSenderIdentification" data-auth="NotApplicable" data-loopstyle="link">
Learn why this is important</a></div>
</td>
<td valign="middle" align="left" width="75px" cellpadding="7px 5px 7px 5px" style="width: 75px; padding: 7px 5px; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; overflow-wrap: break-word; color: rgb(33, 33, 33); background-color: rgb(234, 234, 234);">
</td>
</tr>
</tbody>
</table>
<div>
<div class="x_WordSection1">
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
If there’s a fairshare component to job priorities, and there’s a share assigned to each user under the account, wouldn’t the light user’s jobs move ahead of any of the heavy user’s pending jobs automatically?</p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
</p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<b><span style="font-size: 12pt; color: black;">From: </span></b><span style="font-size: 12pt; color: black;">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of "Groner, Rob" <rug262@psu.edu><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Wednesday, May 17, 2023 at 1:09 PM<br>
<b>To: </b>"slurm-users@lists.schedmd.com" <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] On the ability of coordinators</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
</p>
</div>
<p align="center" style="text-align: center; background: white;"><b><span style="font-size: 12pt; background: white; color: red;">External Email Warning</span></b></p>
<p align="center" style="margin-right: 12pt; margin-left: 12pt; margin-bottom: 0.0001pt; text-align: center; background: white;">
<b><span style="font-size: 12pt; color: red;">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b></p>
<div class="x_MsoNormal" align="center" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;">Ya, I found they had the power to hold jobs just be experimentation. Maybe it will turn out I had something misconfigured and coordinators don't have that ability either. I hope that's not the case, since being
able to hold jobs in their account gives them some usefulness.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;">My interest in this was solely focused on what coordinators could do to jobs within their account. So, I accepted as ok that a coordinator couldn't move jobs in their account to a higher priority than jobs in other
accounts. I just wanted the coordinator to be able to move jobs in their account to a higher priority over other jobs within the same account. Being able to use hold/release seems like what we're looking for. I just wonder why coordinators can't use "top"
as well, for jobs within their coordinated account. I guess "top" is meant to move them to the top of the entire pending queue, and in my case, I was only interested in the coordinator moving certain jobs in their accounts to the top of the account-related
queue. But of course, there ISN'T an account-related queue, so maybe that's why top doesn't work for a coordinator. I think I just answered my own question.....</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<b><span style="color: black;">From:</span></b><span style="color: black;"> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Brian Andrus <toomuchit@gmail.com><br>
<b>Sent:</b> Wednesday, May 17, 2023 2:00 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] On the ability of coordinators</span> </p>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
</p>
</div>
</div>
<div>
<p>Coordinator permissions from the man pages:</p>
<blockquote style="margin-top:5.0pt; margin-bottom:5.0pt">
<p>coordinator<br>
A special privileged user, usually an account manager, that can add users or sub-accounts to the account they are coordinator over. This should be a trusted person since they can change limits on account and user associations, as well as cancel, requeue or
reassign accounts of jobs inside their realm.</p>
</blockquote>
<p>So, I read that as it manages accounts in slurmdb with minimal access to the jobs themselves. So you would be stuck with cancel/requeue. I see no mention of hold, but if that is one of the permissions, I would say, yes, our approach does what you want within
the limits of what the default permissions of a coordinator can do.</p>
<p> </p>
<p>Of course, that still may not work if there are other accounts/partitions/users with higher priority jobs than User B. Specifically if those jobs can use the same resources A's jobs are running on.</p>
<p> </p>
<p>Brian Andrus</p>
<p> </p>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
On 5/17/2023 10:49 AM, Groner, Rob wrote:</p>
</div>
<blockquote style="margin-top:5.0pt; margin-bottom:5.0pt">
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;">I'm not sure what you mean by "if they have the permissions". I'm talking about someone who is specifically designated as "coordinator" of an account in slurm. With that designation, and no other admin level changes,
I'm not aware that they can directly change the priority of jobs associated with the account.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;">If you're talking about additional permissions or admin levels...we're not looking into that as an option. We want to purely use the coordinator role to have them manipulate stuff.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_divRplyFwdMsg">
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<b><span style="color: black;">From:</span></b><span style="color: black;"> slurm-users
<a href="mailto:slurm-users-bounces@lists.schedmd.com" data-auth="NotApplicable" data-loopstyle="link">
<slurm-users-bounces@lists.schedmd.com></a> on behalf of Brian Andrus <a href="mailto:toomuchit@gmail.com" data-auth="NotApplicable" data-loopstyle="link">
<toomuchit@gmail.com></a><br>
<b>Sent:</b> Wednesday, May 17, 2023 12:58 PM<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" data-auth="NotApplicable" data-loopstyle="link">
slurm-users@lists.schedmd.com</a> <a href="mailto:slurm-users@lists.schedmd.com" data-auth="NotApplicable" data-loopstyle="link">
<slurm-users@lists.schedmd.com></a><br>
<b>Subject:</b> Re: [slurm-users] On the ability of coordinators</span> </p>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
</p>
</div>
</div>
<div>
<p>If they have the permissions, you can just raise the priority of user B's jobs to be higher than whatever A's currently are. Then they will run next.</p>
<p>That will work if you are able to wait for some jobs to finish and you can 'skip the line' for the priority jobs.</p>
<p>If you need to preempt running jobs, that would take a bit more effort to set up, but is an alternative.</p>
<p> </p>
<p>Brian Andrus</p>
<p> </p>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
On 5/17/2023 6:40 AM, Groner, Rob wrote:</p>
</div>
<blockquote style="margin-top:5.0pt; margin-bottom:5.0pt">
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;">I was asked to see if coordinators could do anything in this scenario:</span></p>
</div>
<div>
<ul type="disc" style="margin-bottom: 0in;">
<li class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; color: black;">
<span style="font-size:12.0pt">Within the account that they coordinated, User A submitted 1000s of jobs and left for the day.</span></li><li class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; color: black;">
<span style="font-size:12.0pt">Within the same account, User B wanted to run a few jobs really quickly. Once submitted, his jobs were of course behind User A's jobs.</span></li><li class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; color: black;">
<span style="font-size:12.0pt">The coordinator wanted to see the results of User B's runs.</span></li></ul>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;">Reading the docs and doing some experiments, here is what I determined:</span></p>
</div>
<div>
<ul type="disc" style="margin-bottom: 0in;">
<li class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; color: black;">
<span style="font-size:12.0pt">The coordinator could put a hold on all of User A's jobs in the pending queue. This won't affect any jobs User A has that aren't tied to the coordinated account.</span></li><li class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; color: black;">
<span style="font-size:12.0pt">With User A's jobs held, then User B's jobs would be next to run.</span></li><li class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; color: black;">
<span style="font-size:12.0pt">If the coordinator was particularly impatient, he could scancel User A's currently running jobs so that User B's jobs immediately started.</span></li><li class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; color: black;">
<span style="font-size:12.0pt">The coordinator would need to remember to release the held jobs, or put them in a uhold so that User A could release them eventually.</span></li></ul>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;">It seems like the easiest way for the coordinator to elevate User B's jobs to the top of the queue would be if he could "scontrol top" those jobs. But my testing indicates that the coordinator doesn't have that
permission. Is there some reason that a coordinator can't use "scontrol top" to change the priority of jobs within the account that he coordinates?</span></p>
</div>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;">Thanks.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;">Rob</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</body>
</html>