[slurm-users] Automatically migrating jobs to different partitions?

[Vicker, Darby (JSC-EG311)]

I’m not sure if this is what you are trying to accomplish but we do something similar using features and job constraints to get jobs to run on any set of processor types that are available.  We have 4 generations of processors on one of our clusters and our MPI jobs need them to run on all of one type.  Each node has the appropriate processor feature assigned:

NodeName=r1i[0-3]n[0-15] Feature=wes
NodeName=r2i[0-3]n[0-17] Feature=san
NodeName=r3i[0-3]n[0-17] Feature=has
NodeName=r4i[0-3]n[0-17] Feature=bro

All of those nodes are in the same partition.  Jobs that need it then request any processor type, but all require all the same type like this:

#SBATCH -n 96
#SBATCH --constraint=[wes|san|has|bro]

Read the sbatch man page on constraints to see the options and what those brackets mean.  This works very well – users get whatever processor type is free.  You might be able to adapt this to your situation.

From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of Alexander John Mamach <alex.mamach at northwestern.edu>
Reply-To: Slurm User Community List <slurm-users at lists.schedmd.com>
Date: Thursday, March 22, 2018 at 4:35 PM
To: Slurm Mailing List <slurm-users at lists.schedmd.com>
Subject: [slurm-users] Automatically migrating jobs to different partitions?

Hi all,

I’ve been looking into a way to automatically migrate queued jobs from one partition to another. For example, if someone submits in partition A and must wait for resources, move their job request to partition B and try to run, and if they must still wait, then try partition C, etc?

Thanks,

Alex
Systems Administrator

Processing & Information Platform Services, Cyberinfrastructure
Northwestern University Information Technology (NUIT)

2020 Ridge Ave
Evanston, IL 60208-4311

O: (847) 491-2219
M: (312) 887-1881
www.it.northwestern.edu<http://www.it.northwestern.edu>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20180322/2b00f906/attachment-0001.html>