<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div>It could probably be implemented in the job submit Lua script if your jobs are being rejected at submit time. https://slurm.schedmd.com/job_submit_plugins.html that would need to be implemented by the cluster admins, and then modified in a way that would
address your specific scenario. </div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size: 11pt;" color="#000000" data-ogsc=""><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Ebbert, Mark T. <Mark.Ebbert@uky.edu><br>
<b>Sent:</b> Thursday, August 3, 2023 4:37 AM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Automatically converting jobs to regular priority when high-priority is exhausted</font>
<div> </div>
</div>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div style="background-color: rgb(255, 199, 206); width: 100%; border-style: solid; border-color: rgb(156, 0, 6); border-width: 1pt; padding: 2pt; font-size: 10pt; line-height: 12pt; font-family: Calibri; color: black; text-align: left; margin-bottom: 2pt;">
<span style="color: rgb(156, 0, 6); font-weight: bold;">WARNING:</span> This message has originated from an External Source. This may be a phishing expedition that can result in unauthorized access to our IT System. Please use proper judgment and caution when
opening attachments, clicking links, or responding to this email.</div>
<div>
<div class="x_WordSection1">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 11pt; color: rgb(33, 33, 33);">Hello,</span><span style="color: rgb(33, 33, 33);"></span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 11pt; color: rgb(33, 33, 33);"> </span><span style="color: rgb(33, 33, 33);"></span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 11pt; color: rgb(33, 33, 33);">Just wanted to give another go at this question.</span><span style="color: rgb(33, 33, 33);"></span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 11pt; color: rgb(33, 33, 33);"> </span><span style="color: rgb(33, 33, 33);"></span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<b><span style="font-size: 11pt; color: rgb(33, 33, 33);">If no one has input, is there somewhere I can submit feature requests?</span></b><span style="font-size: 11pt; color: rgb(33, 33, 33);"> I didn’t see a way to do that on the schedmd website.</span><span style="color: rgb(33, 33, 33);"></span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 11pt; color: black;">____________________________________________</span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 11pt; color: black;"> </span></p>
<table class="x_MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="225" style="width: 168.55pt; background: white; border-collapse: collapse;">
<tbody>
<tr>
<td width="225" valign="top" style="width:168.55pt; padding:0in 7.5pt 0in 0in">
<table class="x_MsoNormalTable" border="0" cellspacing="0" cellpadding="0" style="border-collapse:collapse">
<tbody>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 13pt; font-family: inherit; color: black; border: 1pt none windowtext; padding: 0in;">Mark T. W. Ebbert, Ph.D.</span><span style="font-size: 13pt; font-family: Georgia, serif; color: black;"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in"></td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black; border: 1pt none windowtext; padding: 0in;">Assistant Professor</span><span style="font-size: 10.5pt; font-family: Georgia, serif; color: black;"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black; border: 1pt none windowtext; padding: 0in;">University of Kentucky</span><span style="font-size: 10.5pt; font-family: Georgia, serif; color: black;"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black; border: 1pt none windowtext; padding: 0in;">Sanders-Brown Center on Aging</span><span style="font-size: 10.5pt; font-family: Georgia, serif; color: black;"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in"></td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black; border: 1pt none windowtext; padding: 0in;">545 Lee T. Todd Jr. Building</span><span style="font-size: 10.5pt; font-family: Georgia, serif; color: black;"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black; border: 1pt none windowtext; padding: 0in;">Lexington, KY 40536</span><span style="font-size: 10.5pt; font-family: Georgia, serif; color: black;"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black;"><a href="tel:859-218-0125" data-auth="NotApplicable" data-ogsc="" style=""><span style="color: black; border: 1pt none windowtext; padding: 0in; text-decoration: none;">859.218.0125</span></a></span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black;">ebbertlab.com</span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
</div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<div id="x_mail-editor-reference-message-container">
<div>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;margin-bottom:12.0pt">
<b><span style="font-size: 12pt; color: black;">From: </span></b><span style="font-size: 12pt; color: black;">Ebbert, Mark T. <Mark.Ebbert@uky.edu><br>
<b>Date: </b>Wednesday, July 26, 2023 at 6:49 PM<br>
<b>To: </b>slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Automatically converting jobs to regular priority when high-priority is exhausted</span></p>
</div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt">Hello,</span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt">I’m a SLURM user (not administrator) so I may not use the correct terms, but hopefully I can get my question across.</span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<b><span style="font-size:11.0pt">Ultimate question:</span></b><span style="font-size:11.0pt"> Is there a way to have SLURM automatically convert job submissions from a “high-priority” account to regular priority when that account’s submissions have been maxed
out? If not, is there a way to request such a feature?</span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<b><span style="font-size:11.0pt">More background</span></b><span style="font-size:11.0pt">: Essentially, my lab as two accounts that we can use on our campus cluster. One is given higher priority in the queue because we directly funded a certain number of
compute nodes, while the other has regular priority. Once we exhaust the number of job units allotted to our high-priority account, any new jobs submitted from that account get straight-up rejected from the scheduler. We don’t want to have to track how many
units we’ve used from a certain account and then switch to a different account. We’d like to always be able to submit under a single account and let SLURM first use our allotted high-priority units, but once those are used up, simply set the rest of the jobs
as regular priority (and not reject them).</span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt">We use Nextflow to manage our job submissions, so in the middle of a large and complicated run, all of our jobs start getting rejected. Going through our scripts, etc. and changing accounts is a waste of time and error prone.
It’s gotten to the point where we’re only submitting jobs under our regular account so we don’t have to deal with our jobs getting rejected, which defeats the purpose of buying into the high-priority to begin with.</span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt">Thanks for your help.</span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 11pt; color: black;">____________________________________________</span><span style="font-size:11.0pt"></span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 11pt; color: black;"> </span><span style="font-size:11.0pt"></span></p>
<table class="x_MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="225" style="width: 168.55pt; background: white; border-collapse: collapse;">
<tbody>
<tr>
<td width="225" valign="top" style="width:168.55pt; padding:0in 7.5pt 0in 0in">
<table class="x_MsoNormalTable" border="0" cellspacing="0" cellpadding="0" style="border-collapse:collapse">
<tbody>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 13pt; font-family: inherit; color: black; border: 1pt none windowtext; padding: 0in;">Mark T. W. Ebbert, Ph.D.</span><span style="font-size:11.0pt"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in"></td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black; border: 1pt none windowtext; padding: 0in;">Assistant Professor</span><span style="font-size:11.0pt"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black; border: 1pt none windowtext; padding: 0in;">University of Kentucky</span><span style="font-size:11.0pt"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black; border: 1pt none windowtext; padding: 0in;">Sanders-Brown Center on Aging</span><span style="font-size:11.0pt"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in"></td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black; border: 1pt none windowtext; padding: 0in;">545 Lee T. Todd Jr. Building</span><span style="font-size:11.0pt"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black; border: 1pt none windowtext; padding: 0in;">Lexington, KY 40536</span><span style="font-size:11.0pt"></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="bottom" style="padding:0in 0in 0in 0in">
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black;"><a href="tel:859-218-0125" data-auth="NotApplicable" data-ogsc="" style=""><span style="color: black; border: 1pt none windowtext; padding: 0in; text-decoration: none;">859.218.0125</span></a></span><span style="font-size:11.0pt"></span></p>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 10.5pt; font-family: Georgia, serif; color: black;">ebbertlab.com</span><span style="font-size:11.0pt"></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<p class="x_MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size:11.0pt"> <span style=""></span></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>