[slurm-users] Change ExcNodeList on a running job

[Ransom, Geoffrey M.]

     I'm just curious as to what causes a user to decide that a given node has an issue? 
     If a node is healthy in all respects, why would a user decide not to use the node?

Not enough free TMPDIR space, a GPU starts having memory errors, or a machine with a temporary issue that slurm health checks are not tracking at the time so it can blackhole jobs.

But honestly, this is less about dealing with actual technical problems and more about dealing with keeping users happy as we help port their existing Univa jobs to slurm. We have a user with a run script that will add the local node to the exclude list and requeue itself up to 5 times if it thinks the program it launched is not running correctly because of a machine issue. I could emulate this behavior easily if the running job could update its own ExcNodeList and requeue itself. I can have a job requeue itself (just sleep after the scontrol command as the requeue is not instant) but slurm does not seem to let me update ExcNodeList on a running job.

Thanks for your suggestions.