[slurm-users] Verifying preemption WON'T happen

[Davide DelVento]

I don't really have an answer for you other than a "hallway comment", that
it sounds like a good thing which I would test with a simulator, if I had
one. I've been intrigued by (but really not looked much into)
https://slurm.schedmd.com/SLUG23/LANL-Batsim-SLUG23.pdf

On Fri, Sep 29, 2023 at 10:05 AM Groner, Rob <rug262 at psu.edu> wrote:

> On our system, for some partitions, we guarantee that a job can run at
> least an hour before being preempted by a higher priority job.  We use the
> QOS preempt exempt time for this, and it appears to be working.  But of
> course, I want to TEST that it works.
>
> So on a test system, I start a lower priority job on a specific node, wait
> until it starts running, and then I start a higher priority job for the
> same node.  The test should only pass if the higher priority job has an
> OPPORTUNITY to preempt the lower priority job, and doesn't.
>
> Now, I know I can get a preempt eligible time out of scontrol for the
> lower priority job and verify that it's set for an hour (I do check that
> already), but that's not good enough for me.  I could obviously let the
> test run for an hour to verify the lower priority job was never
> preempted...but that's not really feasible.  So instead, I want to verify
> that the higher priority job has had a chance to preempt the lower priority
> job, and it did not.
>
> So far, the way I've been doing that is to check the reported Scheduler in
> the scontrol job output for the higher priority job.  I figure that when
> the scheduler changes to Backfill instead of Main, then the higher priority
> job has been seen by the main scheduler and it passed on the chance to
> preempt the lower priority job.
>
> Is that a good assumption?  Is there any other, or potentially quicker,
> way to verify that the higher priority job will NOT preempt the lower
> priority job?
>
> Rob
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230929/b2eb1475/attachment.htm>