<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Well again, I don't want to tweak things just to get the test to happen quicker. I DO have to keep in mind the scheduler and backfill settings, though. For instance, I think the default scheduler and backfill interval is 60 and 30 seconds...or vice versa.
So, before I check the Scheduler value for the high priority job via scontrol, I wait 90 seconds and then some. In a perfect world, that SHOULD have given the scheduler and backfill scheduler time to get to it. I THINK, however, that in a sufficiently busy
system, there's no guarantee even after that amount of time that the new high priority job has been evaluated.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I'll take a look at sdiag and see if it can tell me where the job is at, thanks for the suggestion.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Rob</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Ryan Novosielski <novosirj@rutgers.edu><br>
<b>Sent:</b> Friday, September 29, 2023 4:19 PM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Verifying preemption WON'T happen</font>
<div> </div>
</div>
<div dir="auto">You can get some information on that from sdiag, and there are tweaks you can make to backfill scheduling that affect how quickly it will get to a job.
<div><br>
</div>
<div>That doesn’t really answer your real question, but might help you when you are looking into this.<br>
<br>
<div dir="ltr">Sent from my iPhone</div>
<div dir="ltr"><br>
<blockquote type="cite">On Sep 29, 2023, at 16:10, Groner, Rob <rug262@psu.edu> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I'm not looking for a one-time answer. We run these tests anytime we change anything related to slurm....version, configuration, etc. We certainly run the test after the system comes back up after an outage, and an hour would be a long time to wait for
that. That's certainly the brute-force approach, but I'm hoping there's a definitive way to show, through scontrol job output, that the job won't preempt.</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I could set the preemptexempttime to a smaller value, say 5 minutes instead of 1 hour, that is true, but there's a few issues with that. </div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<ol data-editing-info="{"orderedStyleType":3,"unorderedStyleType":1}">
<li style="list-style-type:"1) "">I would then no longer be testing the system as it actually is. I want to test the system in its actual production configuration.</li><li style="list-style-type:"2) "">If I did lower its value, what would be a safe value? 5 minutes? Does running for 5 minutes guarantee that the higher priority job had a chance to preempt it but didn't? Or did the scheduler even ever get to it? On a test
cluster with few jobs, you could be reasonably assured it did, but running tests on the production cluster...isn't it possible the scheduler hasn't yet had a chance to process it, even after 5 minutes? Depends on the slurm scheduler settings I suppose....</li></ol>
<div>rob</div>
<div><br>
</div>
</div>
<div id="x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Bernstein, Noam CIV USN NRL (6393) Washington DC (USA) <noam.bernstein@nrl.navy.mil><br>
<b>Sent:</b> Friday, September 29, 2023 3:14 PM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Verifying preemption WON'T happen</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="border:0; display:table; width:100%; table-layout:fixed; border-collapse:seperate; float:none">
<tbody style="display:block">
<tr>
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="padding:7px 2px 7px 2px; background-color:#A6A6A6">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" color="#212121" style="width:100%; background-color:#EAEAEA; padding:7px 5px 7px 15px; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif; font-size:12px; font-weight:normal; color:#212121; text-align:left; word-wrap:break-word">
<div>You don't often get email from noam.bernstein@nrl.navy.mil. <a href="https://aka.ms/LearnAboutSenderIdentification">
Learn why this is important</a></div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" color="#212121" style="width:75px; background-color:#EAEAEA; padding:7px 5px 7px 5px; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif; font-size:12px; font-weight:normal; color:#212121; text-align:left; word-wrap:break-word">
</td>
</tr>
</tbody>
</table>
<div>
<div>
<blockquote type="cite" class="">
<div class="">On Sep 29, 2023, at 2:51 PM, Davide DelVento <<a href="mailto:davide.quantum@gmail.com" class="">davide.quantum@gmail.com</a>> wrote:</div>
<br class="x_x_Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">I don't really have an answer for you other than a "hallway comment", that it sounds like a good thing which I would test with a simulator, if I had one. I've been intrigued by (but really not looked much into) <a href="https://slurm.schedmd.com/SLUG23/LANL-Batsim-SLUG23.pdf" originalsrc="https://slurm.schedmd.com/SLUG23/LANL-Batsim-SLUG23.pdf" shash="sJ5Fp5+vwigkHCb3fAoeBilcLwkvDfzthC72qRvDi9kqeF19hk3ckIiPLaCc2+O+mY8HBMJRl+78RlBliau0HLgOLQ0/D4o50pkYrb6Ja8K/ZP5R7jNAk6dg5sSWr9bWoqT065Gvu7jaVkH/BKptXA0Pjc16LKL+RM+RPK4zc6w=" originalsrc="https://slurm.schedmd.com/SLUG23/LANL-Batsim-SLUG23.pdf" shash="aYExLyRKRpdXmoU9v0vtdnYkvKsSVD/sJXpPL/D5qQQ2wJuBrZuA5XjaFqvhkIYGFwKbteO7T8NTCk4a4lVuLBmZ74sjAOfhH96Sdzbec0beKb5796DkWWQ5QV7sk8SwoLkCN7I01nKADBywdXr9Gz4yhr8VMrh2A/UidhYX0ak=" class="">https://slurm.schedmd.com/SLUG23/LANL-Batsim-SLUG23.pdf</a></div>
<br class="">
<div class="x_x_gmail_quote">
<div dir="ltr" class="x_x_gmail_attr">On Fri, Sep 29, 2023 at 10:05 AM Groner, Rob <<a href="mailto:rug262@psu.edu" class="">rug262@psu.edu</a>> wrote:<br class="">
</div>
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_x_msg-2327034242780967385">
<div dir="ltr" class="">
<div class="" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="x_x_gmail_quote">
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_x_msg-2327034242780967385">
<div dir="ltr" class="">
<div class="" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<span class="" style="font-size:12pt">I could obviously let the test run for an hour to verify the lower priority job was never preempted...but that's not really feasible. </span></div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
<style class="">
<!--
.x_x_logotable
{color:navy;
border:0px solid white;
font-family:arial;
font-size:100%;
padding:0px;
border-collapse:collapse}
-->
</style>
<div class="">Why not? Isn't it going to take longer than an hour to wait for responses to this post? Also, you could set up the minimum time to a much smaller value, so it won't take as long to test.</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>