[slurm-users] one job at a time - how to set?

[Allan Streib]

Well looking at the current slurm.conf it appears that the name was
changed, and "Shared" is now called "OverSubscribe" in more modern slurm
versions. So you might look deeper at what config options are in
conflict, since with the EXCLUSIVE mode I get one node per job here.

Allan

Allan Streib <astreib at indiana.edu> writes:

> I'm running an even older slurm than you (it does what I need, I am a
> team of one and I have many things to take care of other than chasing
> the latest version of every piece of software).
>
> Anyway, did you try Shared=EXCLUSIVE in the partition configuration?
>
> From the (v14.11.7) slurm.conf man page:
>
>     Shared Controls the ability of the partition to execute more than
>     one job at a time on each resource [...]
>
>     EXCLUSIVE Allocates entire nodes to jobs even with select/cons_res
>     configured. Jobs that run in partitions with "Shared=EXCLUSIVE" will
>     have exclusive access to all allocated nodes. (node, socket or core
>     depending upon the value of SelectTypeParameters).
>
> Allan
>
>
> Rutger Vos <rutger.vos at naturalis.nl> writes:
>
>> I did some tests and this doesn't work with MPI, in the following
>> sense: the way we were running parallel jobs was by invoking `sbatch
>> -n4 script.sh` where the script.sh would contain an `mpirun` command
>> without specifying the number of processors. It would then simply claim
>> the ones assigned to it by sbatch. With one CPU as per the slurm.conf,
>> mpirun would only ever get one processor, hence no parallelization.
>>
>> The other test I did was to try OverSubscribe=Exclusive on the
>> partition, but the slurmctld failed to start so there is some
>> combination of config options that it is unhappy about.
>>
>> The workable option I can think of right now is to have everyone run
>> their jobs with --exclusive as an extra argument.
>>
>> On Thu, Apr 30, 2020 at 12:22 PM Rutger Vos <rutger.vos at naturalis.nl>
>> wrote:
>>
>>     This seems like the easiest solution, but how would that interact
>>     with MPI jobs? Apologies if these are really basic questions, and
>>     thank you very much for thinking along.
>>    
>>     Rutger
>>    
>>     On Wed, Apr 29, 2020 at 10:25 PM Doug Meyer <dameyer at raytheon.com>
>>     wrote:
>>
>>         Change node definition in slurm.conf for that one node to 1
>>         CPU.
>>
>>         Doug Meyer
>>
>>         From: slurm-users <slurm-users-bounces at lists.schedmd.com> On
>>         Behalf Of Rutger Vos
>>         Sent: Wednesday, April 29, 2020 1:20 PM
>>         To: Slurm User Community List <slurm-users at lists.schedmd.com>
>>         Subject: [External] Re: [slurm-users] one job at a time - how
>>         to set?
>>
>>         Hi Michael,
>>
>>         thanks very much for your swift reply. So here we would have to
>>         convince the users they'd have to specify this when submitting,
>>         right? I.e. 'sbatch --exclusive myjob.sh', if I
>>         understand correctly. Would there be a way to simply enforce
>>         this, i.e. at the slurm.conf level or something?
>>
>>         Thanks again!
>>
>>         Rutger
>>
>>         On Wed, Apr 29, 2020 at 10:06 PM Renfro, Michael <
>>         Renfro at tntech.edu> wrote:
>>        
>>             That’s a *really* old version, but 
>>             https://slurm.schedmd.com/archive/slurm-15.08.13/sbatch.html
>>              indicates there’s an exclusive flag you can set.
>>            
>>                 On Apr 29, 2020, at 1:54 PM, Rutger Vos <
>>                 rutger.vos at naturalis.nl> wrote:
>>                
>>                                            .                           
>>                
>>                 Hi,
>>
>>                 for a smallish machine that has been having degraded
>>                 performance we want to implement a policy where
>>                 only one job (submitted with sbatch) is allowed to run
>>                 and any others submitted after it are supposed to wait
>>                 in line.
>>
>>                 I assumed this was straightforward but I can't seem to
>>                 figure it out. Can I set that up in slurm.conf or in
>>                 some other way? Thank you very much for your help. BTW
>>                 we are running slurm 15.08.7 if that is at all
>>                 relevant.
>>
>>                 Best wishes,
>>
>>                 Dr. Rutger A. Vos
>>                
>>                 Researcher / Bioinformatician
>>                
>>                 [logo-new]
>>
>>                 [+31717519600 - +31627085806 ]
>>                
>>                 [rutger.vos at naturalis.nl - www.naturalis.nl]
>>                
>>                 [Darwinweg 2, 2333 CR Leiden]
>>                
>>                 [Postbus 9517, 2300 RA Leiden]
>>
>>                 [schildpad]
>>
>>         --
>>
>>         Met vriendelijke groet,
>>
>>         Dr. Rutger A. Vos
>>        
>>         Researcher / Bioinformatician
>>        
>>         [logo-new]
>>
>>         [+31717519600 - +31627085806 ]
>>        
>>         [rutger.vos at naturalis.nl - www.naturalis.nl]
>>        
>>         [Darwinweg 2, 2333 CR Leiden]
>>        
>>         [Postbus 9517, 2300 RA Leiden]
>>
>>         [schildpad]
>>
>>     --
>>    
>>     Met vriendelijke groet,
>>    
>>     Dr. Rutger A. Vos
>>     Researcher / Bioinformatician
>>     [logo-new]
>>
>>     [+31717519600 - +31627085806 ]
>>     [rutger.vos at naturalis.nl - www.naturalis.nl]
>>     [Darwinweg 2, 2333 CR Leiden]
>>     [Postbus 9517, 2300 RA Leiden]
>>    
>>     [schildpad]
>>
>> --
>>
>> Met vriendelijke groet,
>>
>> Dr. Rutger A. Vos
>> Researcher / Bioinformatician
>> [logo-new]
>>
>> [+31717519600 - +31627085806 ]
>> [rutger.vos at naturalis.nl - www.naturalis.nl]
>> [Darwinweg 2, 2333 CR Leiden]
>> [Postbus 9517, 2300 RA Leiden]
>>
>> [schildpad]
>>