[slurm-users] one job at a time - how to set?

[Allan Streib]

I'm running an even older slurm than you (it does what I need, I am a
team of one and I have many things to take care of other than chasing
the latest version of every piece of software).

Anyway, did you try Shared=EXCLUSIVE in the partition configuration?

>From the (v14.11.7) slurm.conf man page:

    Shared Controls the ability of the partition to execute more than
    one job at a time on each resource [...]

    EXCLUSIVE Allocates entire nodes to jobs even with select/cons_res
    configured. Jobs that run in partitions with "Shared=EXCLUSIVE" will
    have exclusive access to all allocated nodes. (node, socket or core
    depending upon the value of SelectTypeParameters).

Allan


Rutger Vos <rutger.vos at naturalis.nl> writes:

> I did some tests and this doesn't work with MPI, in the following
> sense: the way we were running parallel jobs was by invoking `sbatch
> -n4 script.sh` where the script.sh would contain an `mpirun` command
> without specifying the number of processors. It would then simply claim
> the ones assigned to it by sbatch. With one CPU as per the slurm.conf,
> mpirun would only ever get one processor, hence no parallelization.
>
> The other test I did was to try OverSubscribe=Exclusive on the
> partition, but the slurmctld failed to start so there is some
> combination of config options that it is unhappy about.
>
> The workable option I can think of right now is to have everyone run
> their jobs with --exclusive as an extra argument.
>
> On Thu, Apr 30, 2020 at 12:22 PM Rutger Vos <rutger.vos at naturalis.nl>
> wrote:
>
>     This seems like the easiest solution, but how would that interact
>     with MPI jobs? Apologies if these are really basic questions, and
>     thank you very much for thinking along.
>    
>     Rutger
>    
>     On Wed, Apr 29, 2020 at 10:25 PM Doug Meyer <dameyer at raytheon.com>
>     wrote:
>
>         Change node definition in slurm.conf for that one node to 1
>         CPU.
>
>         Doug Meyer
>
>         From: slurm-users <slurm-users-bounces at lists.schedmd.com> On
>         Behalf Of Rutger Vos
>         Sent: Wednesday, April 29, 2020 1:20 PM
>         To: Slurm User Community List <slurm-users at lists.schedmd.com>
>         Subject: [External] Re: [slurm-users] one job at a time - how
>         to set?
>
>         Hi Michael,
>
>         thanks very much for your swift reply. So here we would have to
>         convince the users they'd have to specify this when submitting,
>         right? I.e. 'sbatch --exclusive myjob.sh', if I
>         understand correctly. Would there be a way to simply enforce
>         this, i.e. at the slurm.conf level or something?
>
>         Thanks again!
>
>         Rutger
>
>         On Wed, Apr 29, 2020 at 10:06 PM Renfro, Michael <
>         Renfro at tntech.edu> wrote:
>        
>             That’s a *really* old version, but 
>             https://slurm.schedmd.com/archive/slurm-15.08.13/sbatch.html
>              indicates there’s an exclusive flag you can set.
>            
>                 On Apr 29, 2020, at 1:54 PM, Rutger Vos <
>                 rutger.vos at naturalis.nl> wrote:
>                
>                                            .                           
>                
>                 Hi,
>
>                 for a smallish machine that has been having degraded
>                 performance we want to implement a policy where
>                 only one job (submitted with sbatch) is allowed to run
>                 and any others submitted after it are supposed to wait
>                 in line.
>
>                 I assumed this was straightforward but I can't seem to
>                 figure it out. Can I set that up in slurm.conf or in
>                 some other way? Thank you very much for your help. BTW
>                 we are running slurm 15.08.7 if that is at all
>                 relevant.
>
>                 Best wishes,
>
>                 Dr. Rutger A. Vos
>                
>                 Researcher / Bioinformatician
>                
>                 [logo-new]
>
>                 [+31717519600 - +31627085806 ]
>                
>                 [rutger.vos at naturalis.nl - www.naturalis.nl]
>                
>                 [Darwinweg 2, 2333 CR Leiden]
>                
>                 [Postbus 9517, 2300 RA Leiden]
>
>                 [schildpad]
>
>         --
>
>         Met vriendelijke groet,
>
>         Dr. Rutger A. Vos
>        
>         Researcher / Bioinformatician
>        
>         [logo-new]
>
>         [+31717519600 - +31627085806 ]
>        
>         [rutger.vos at naturalis.nl - www.naturalis.nl]
>        
>         [Darwinweg 2, 2333 CR Leiden]
>        
>         [Postbus 9517, 2300 RA Leiden]
>
>         [schildpad]
>
>     --
>    
>     Met vriendelijke groet,
>    
>     Dr. Rutger A. Vos
>     Researcher / Bioinformatician
>     [logo-new]
>
>     [+31717519600 - +31627085806 ]
>     [rutger.vos at naturalis.nl - www.naturalis.nl]
>     [Darwinweg 2, 2333 CR Leiden]
>     [Postbus 9517, 2300 RA Leiden]
>    
>     [schildpad]
>
> --
>
> Met vriendelijke groet,
>
> Dr. Rutger A. Vos
> Researcher / Bioinformatician
> [logo-new]
>
> [+31717519600 - +31627085806 ]
> [rutger.vos at naturalis.nl - www.naturalis.nl]
> [Darwinweg 2, 2333 CR Leiden]
> [Postbus 9517, 2300 RA Leiden]
>
> [schildpad]
>