[slurm-users] [External] How can I do to prevent a specific job from being prempted?

[Brian Andrus]

Modify it and raise the priority to something very, very high.

scontrol update job=JOBID priority=9999999

Brian Andrus

On 9/16/2021 8:39 AM, 顏文 wrote:
> Dear users
>
> Thank for the immediate replies.I currently have one important job 
> running. How to prevent the running job from being preempted? That job 
> has been running for two weeks, and I am afraid it will be terminated 
> in the next few days. Can we do anything to secure the job from being 
> cancelled and preempted? Thanks.
>
> regards,
> Peter
>
> Russell Jones <arjones85 at gmail.com <mailto:arjones85 at gmail.com>> 於 
> 2021年9月14日 週二 下午10:52寫道：
>
>     The other option is creating a "special" partition that only this
>     user(s) can submit to, where jobs running in that partition have a
>     higher priority than all the others (if you are using partition
>     priority like we are).
>
>     On Tue, Sep 14, 2021 at 3:26 AM Loris Bennett
>     <loris.bennett at fu-berlin.de <mailto:loris.bennett at fu-berlin.de>>
>     wrote:
>
>         Dear Peter,
>
>         顏文 <peteryen2017 at gmail.com <mailto:peteryen2017 at gmail.com>>
>         writes:
>
>         > Dear Mr. Zillner
>         >
>         > I would like the specific running job not being rescheduled
>         , but also can not be terminated or cancelled in any way. If
>         the job is cancelled, I need to start it over again. Normally
>         this kind of jobs require weeks to
>         > finish. So the time costs it take to restart is quite
>         significant. Can your suggestions also achieve that? Thanks.
>
>         I have not tried this, but could you define a QOS which cannot be
>         preempted by the other QOSs?  That way you can also restrict
>         access to
>         the QOS, since a non-preemptable QOS will probably be more
>         attractive
>         than a preemptable one.  You would then obviously have to
>         explain to the
>         users which QOS they should use.
>
>         Cheers,
>
>         Loris
>
>         > with regards,
>         > Peter
>         >
>         > Florian Zillner <fzillner at lenovo.com
>         <mailto:fzillner at lenovo.com>> 於 2021年9月14日 週二 下午3:40寫道：
>         >
>         >  See the no-requeue option for SBATCH:
>         >
>         >  --no-requeue
>         >  Specifies that the batch job should never be requeued under
>         any circumstances. Setting this option will prevent system
>         administrators from being able to restart the job (for
>         example, after a scheduled
>         >  downtime), recover from a node failure, or be requeued upon
>         preemption by a higher priority job. When a job is requeued,
>         the batch script is initiated from its beginning. Also see the
>         --requeue option. The
>         >  JobRequeue configuration parameter controls the default
>         behavior on the cluster.
>         >
>         > https://slurm.schedmd.com/sbatch.html
>         <https://slurm.schedmd.com/sbatch.html>
>         >
>         >  Get Outlook for Android
>         >
>         --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>         >  From: slurm-users <slurm-users-bounces at lists.schedmd.com
>         <mailto:slurm-users-bounces at lists.schedmd.com>> on behalf of
>         顏文 <peteryen2017 at gmail.com <mailto:peteryen2017 at gmail.com>>
>         >  Sent: Tuesday, September 14, 2021 7:02:13 AM
>         >  To: slurm-users at lists.schedmd.com
>         <mailto:slurm-users at lists.schedmd.com>
>         <slurm-users at lists.schedmd.com
>         <mailto:slurm-users at lists.schedmd.com>>
>         >  Subject: [External] [slurm-users] How can I do to prevent a
>         specific job from being prempted?
>         >
>         >  Dear slurm users,
>         >
>         >  I have some specific jobs that can't be terminated,
>         otherwise they need to be rerun from the beginning. Can we
>         simply apply some settings (either by user or administrator)
>         so that these jobs will not be
>         >  preempted ? Thanks.
>         >
>         >  with regards,
>         >  Peter
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210916/8cc86b62/attachment.htm>