[slurm-users] How to hold a job until a feature is available?

[Groner, Rob]

I'm trying to setup a system where, when a job from a certain account is submitted, if no nodes are available that have a specific feature, then the job will be paused/held/pending and a node will be dynamically created with that feature.

I can now dynamically bring up the node with the feature, and it shows in the sinfo output as having the feature.  But I can't yet figure out how to intercept the job submission request and put it on hold so that I can bring up the node.

If I don't do anything, then the job just instantly fails because there are no nodes with that feature.

Could I maybe create a "dummy" node that has the feature, but no resources?  So the job would be set to pending for resources and would stay that way until I brought up a new node with the feature and with resources.

I've tried using slurm_job_submit.lua, and I detected the requested feature, and the tried to set the job to hold...but it still error'd out because of "invalid feature specification".

Thanks for the help.

Rob

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220929/405336cc/attachment.htm>