[slurm-users] Automatically migrating jobs to different partitions?

[Chris Samuel]

On Friday, 23 March 2018 8:49:00 AM AEDT Ryan Novosielski wrote:

> If I’m not mistaken, you may submit with multiple partitions specified and
> it will run on the one that makes the most sense.

More importantly it'll run on the one it can first, based on the priority of 
the partitions requested.

You can automate some of this with a submit filter. For instance we have 
overlapping partitions for our skylake nodes, one for GPU jobs that can access 
all 36 cores (and has a higher priority) and one for CPU only jobs that can 
only access 32 cores.  The submit filter checks if they've asked for GPUs and 
sets the partition appropriately and if they've not asked for GPUs then it 
prevents them requesting the GPU partition.

Of course it can't check if the job itself is really going to use GPUs or 
not..

Hope this helps,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC