Suspending jobs when preempted
I am new to Slurm, sorry if my question has been asked before, I could only find some older discussions about this. I am currently setting up a job scheduler in our company. Main bottleneck is licenses, so a license scheduler including preemption should be included. I have noticed this in the documentation: "Because licenses are not freed when jobs are suspended, jobs using licenses requested by higher priority jobs will only be prempted when PreemptMode is either REQUEUE or CANCEL and PreemptParameters=reclaim_licenses is set." Which means I cannot set the PreemptMode to SUSPEND, is that correct? Is that something that will ever be implemented? I have created tres licences and implemented a polling mechanism that keeps the slurm tres licenses in sync with the real licenses. Our application automatically releases licenses when jobs get a a TSTP signal. So, all I want is the Slurm scheduler to suspend low prio jobs when needed, send a TSTP signal and let the book-keeping be done by my polling script. Is this in any way possible?
I hear tell that if you use ProctrackType=proctrack/linuxproc you can catch the SIGTSTP and act accordingly. So if that is possible in your setup, that may be your solution. Brian Andrus On 9/10/2026 7:38 AM, Peter via slurm-users wrote:
I am new to Slurm, sorry if my question has been asked before, I could only find some older discussions about this.
I am currently setting up a job scheduler in our company. Main bottleneck is licenses, so a license scheduler including preemption should be included. I have noticed this in the documentation:
"Because licenses are not freed when jobs are suspended, jobs using licenses requested by higher priority jobs will only be prempted when PreemptMode is either REQUEUE or CANCEL and PreemptParameters=reclaim_licenses is set."
Which means I cannot set the PreemptMode to SUSPEND, is that correct? Is that something that will ever be implemented?
I have created tres licences and implemented a polling mechanism that keeps the slurm tres licenses in sync with the real licenses. Our application automatically releases licenses when jobs get a a TSTP signal. So, all I want is the Slurm scheduler to suspend low prio jobs when needed, send a TSTP signal and let the book-keeping be done by my polling script. Is this in any way possible?
Catching the signal is not the issue. The issue is that I would like to use PreemptMode=SUSPEND, not REQUEUE or CANCEL. With SUSPEND, jobs are never actually suspended when they use licenses.
Ok, I think get what you are saying. It does say it won't suspend if PreemptParameters=reclaim_licenses is set. If you don't set that you could then suspend and catch the signal. Once that happens, you can do what you like with the licenses. I guess to answer your direct question, you CAN set PreemptMode to SUSPEND, it just won't work if you also set PreemptParameters=reclaim_licenses. Will that ever be implemented? Probably not, but maybe, if someone who is paying for support requests it. There would be quite a bit of details involved to make it work everywhere. If you are looking for a way to get to your apparent goal (suspend a job, but also free up the license it was using), you could cobble something together that does that. To get to a real solution, you would need to include all of the other pertinent requirements in your plan. It is, of course, possible with all requirements "You cain't get there from here" On 9/11/2026 12:36 AM, Peter via slurm-users wrote:
Catching the signal is not the issue. The issue is that I would like to use PreemptMode=SUSPEND, not REQUEUE or CANCEL. With SUSPEND, jobs are never actually suspended when they use licenses.
participants (2)
-
Brian Andrus -
pie@lem.com