[slurm-users] How to hold a job until a feature is available?

[mercan]

Why don't use a spesific queue instead of the specific feature.The queue 
is an object for waiting resource, it is ready to use for this purpose. 
When required resources are ready to use, the jobs will start.


Regards;


Ahmet M.




29.09.2022 22:27 tarihinde Groner, Rob yazdı:
> I'm trying to setup a system where, when a job from a certain account 
> is submitted, if no nodes are available that have a specific feature, 
> then the job will be paused/held/pending and a node will be 
> dynamically created with that feature.
>
> I can now dynamically bring up the node with the feature, and it shows 
> in the sinfo output as having the feature.  But I can't yet figure out 
> how to intercept the job submission request and put it on hold so that 
> I can bring up the node.
>
> If I don't do anything, then the job just instantly fails because 
> there are no nodes with that feature.
>
> Could I maybe create a "dummy" node that has the feature, but no 
> resources?  So the job would be set to pending for resources and would 
> stay that way until I brought up a new node with the feature and with 
> resources.
>
> I've tried using slurm_job_submit.lua, and I detected the requested 
> feature, and the tried to set the job to hold...but it still error'd 
> out because of "invalid feature specification".
>
> Thanks for the help.
>
> Rob
>