[slurm-users] Execute scripts on suspend and cancel

[Goetz, Patrick G]

Are applications even aware when they've been hit by a SIGSTP?  This 
idea of a license being released under these circumstances just seems 
very unlikely.

On 10/15/19 1:57 PM, Brian Andrus wrote:
> It seems that there are some details that would need addressed.
> 
> A suspend signal is nothing more than sending a SIGSTP (like hitting 
> ctrl-s), so the application is still in memory awaiting SIGCONT
> 
> So what should happen when it continues and there are no more licenses? 
> So the proper place for what you are looking for is in the application 
> itself. If it is given a SIGSTP, it could release the licenses and then 
> check them out again when SIGCONT is received.
> 
> If you are able to tell your app to release/request a license 
> externally, you may want to have a wrapper to do the signal handling 
> until they have it as part of their app.
> 
> Brian Andrus
> 
> 
> On 10/14/2019 4:40 AM, Oytun Peksel wrote:
>>
>> It is quite weird if slurm has no mechanism as described. I have been 
>> digging more into it and someone suggested a workaround using mail 
>> notifications. You use a script instead of the mail application and 
>> catch the event then use use sacct to see what is happening.
>>
>> Two problems with this:
>>
>> ·There is no mail sent with suspended preemption
>>
>> ·If you use requeue instead there will be a mail event and you can 
>> catch it. Sacct will flag it as “preempted” so you know it is 
>> requeued. But then it would change it pending. So you really need to 
>> be quick to catch it. Also there is no distinctive flag for resuming.
>>
>> Anyone has any other method to execute scripts during preemption?
>>
>>
>>
>> *Oytun Peksel*
>>
>> oytun.peksel at semcon.com <mailto:oytun.peksel at semcon.com>
>>
>> 	
>> 	
>>
>> +46739205917
>>
>> 	
>> 	
>>
>> *From:*slurm-users <slurm-users-bounces at lists.schedmd.com> *On Behalf 
>> Of *Oytun Peksel
>> *Sent:* den 11 oktober 2019 09:10
>> *To:* slurm-users at lists.schedmd.com
>> *Subject:* [slurm-users] Execute scripts on suspend and cancel
>>
>> Hi,
>>
>> I was wondering is there an option in Slurm to execute custom scripts 
>> before Suspend signal.  What I need to do is to tell an application to 
>> release it’s licenses before sending the suspend signal during 
>> preemption. I think went through all the documentation but could not 
>> find a mechanism like this.
>>
>> BR
>>
>> /Oytun
>>
>>
>>
>> /When you communicate with us or otherwise interact with Semcon, we 
>> will process personal data that you provide to us or we collect about 
>> you, please read more in our Privacy Policy 
>> <https://semcon.com/data-privacy-policy/>./
>>