<div dir="ltr"><div dir="ltr"><div>Hi,</div><div><br></div><div>If you only want 1 job at a time, the best option is to configure the SelectType plugin to select/linear (<a href="https://slurm.schedmd.com/slurm.conf.html#OPT_select/cons_res">https://slurm.schedmd.com/slurm.conf.html#OPT_select/cons_res</a>). That will force slurm to allocate full nodes.</div><div><br></div><div>Regards,<br></div><div>Carlos<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 30, 2020 at 7:01 PM Allan Streib <<a href="mailto:astreib@indiana.edu">astreib@indiana.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Well looking at the current slurm.conf it appears that the name was<br>
changed, and "Shared" is now called "OverSubscribe" in more modern slurm<br>
versions. So you might look deeper at what config options are in<br>
conflict, since with the EXCLUSIVE mode I get one node per job here.<br>
<br>
Allan<br>
<br>
Allan Streib <<a href="mailto:astreib@indiana.edu" target="_blank">astreib@indiana.edu</a>> writes:<br>
<br>
> I'm running an even older slurm than you (it does what I need, I am a<br>
> team of one and I have many things to take care of other than chasing<br>
> the latest version of every piece of software).<br>
><br>
> Anyway, did you try Shared=EXCLUSIVE in the partition configuration?<br>
><br>
> From the (v14.11.7) slurm.conf man page:<br>
><br>
> Shared Controls the ability of the partition to execute more than<br>
> one job at a time on each resource [...]<br>
><br>
> EXCLUSIVE Allocates entire nodes to jobs even with select/cons_res<br>
> configured. Jobs that run in partitions with "Shared=EXCLUSIVE" will<br>
> have exclusive access to all allocated nodes. (node, socket or core<br>
> depending upon the value of SelectTypeParameters).<br>
><br>
> Allan<br>
><br>
><br>
> Rutger Vos <<a href="mailto:rutger.vos@naturalis.nl" target="_blank">rutger.vos@naturalis.nl</a>> writes:<br>
><br>
>> I did some tests and this doesn't work with MPI, in the following<br>
>> sense: the way we were running parallel jobs was by invoking `sbatch<br>
>> -n4 script.sh` where the script.sh would contain an `mpirun` command<br>
>> without specifying the number of processors. It would then simply claim<br>
>> the ones assigned to it by sbatch. With one CPU as per the slurm.conf,<br>
>> mpirun would only ever get one processor, hence no parallelization.<br>
>><br>
>> The other test I did was to try OverSubscribe=Exclusive on the<br>
>> partition, but the slurmctld failed to start so there is some<br>
>> combination of config options that it is unhappy about.<br>
>><br>
>> The workable option I can think of right now is to have everyone run<br>
>> their jobs with --exclusive as an extra argument.<br>
>><br>
>> On Thu, Apr 30, 2020 at 12:22 PM Rutger Vos <<a href="mailto:rutger.vos@naturalis.nl" target="_blank">rutger.vos@naturalis.nl</a>><br>
>> wrote:<br>
>><br>
>> This seems like the easiest solution, but how would that interact<br>
>> with MPI jobs? Apologies if these are really basic questions, and<br>
>> thank you very much for thinking along.<br>
>> <br>
>> Rutger<br>
>> <br>
>> On Wed, Apr 29, 2020 at 10:25 PM Doug Meyer <<a href="mailto:dameyer@raytheon.com" target="_blank">dameyer@raytheon.com</a>><br>
>> wrote:<br>
>><br>
>> Change node definition in slurm.conf for that one node to 1<br>
>> CPU.<br>
>><br>
>> Doug Meyer<br>
>><br>
>> From: slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> On<br>
>> Behalf Of Rutger Vos<br>
>> Sent: Wednesday, April 29, 2020 1:20 PM<br>
>> To: Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
>> Subject: [External] Re: [slurm-users] one job at a time - how<br>
>> to set?<br>
>><br>
>> Hi Michael,<br>
>><br>
>> thanks very much for your swift reply. So here we would have to<br>
>> convince the users they'd have to specify this when submitting,<br>
>> right? I.e. 'sbatch --exclusive myjob.sh', if I<br>
>> understand correctly. Would there be a way to simply enforce<br>
>> this, i.e. at the slurm.conf level or something?<br>
>><br>
>> Thanks again!<br>
>><br>
>> Rutger<br>
>><br>
>> On Wed, Apr 29, 2020 at 10:06 PM Renfro, Michael <<br>
>> <a href="mailto:Renfro@tntech.edu" target="_blank">Renfro@tntech.edu</a>> wrote:<br>
>> <br>
>> That’s a *really* old version, but <br>
>> <a href="https://slurm.schedmd.com/archive/slurm-15.08.13/sbatch.html" rel="noreferrer" target="_blank">https://slurm.schedmd.com/archive/slurm-15.08.13/sbatch.html</a><br>
>> indicates there’s an exclusive flag you can set.<br>
>> <br>
>> On Apr 29, 2020, at 1:54 PM, Rutger Vos <<br>
>> <a href="mailto:rutger.vos@naturalis.nl" target="_blank">rutger.vos@naturalis.nl</a>> wrote:<br>
>> <br>
>> . <br>
>> <br>
>> Hi,<br>
>><br>
>> for a smallish machine that has been having degraded<br>
>> performance we want to implement a policy where<br>
>> only one job (submitted with sbatch) is allowed to run<br>
>> and any others submitted after it are supposed to wait<br>
>> in line.<br>
>><br>
>> I assumed this was straightforward but I can't seem to<br>
>> figure it out. Can I set that up in slurm.conf or in<br>
>> some other way? Thank you very much for your help. BTW<br>
>> we are running slurm 15.08.7 if that is at all<br>
>> relevant.<br>
>><br>
>> Best wishes,<br>
>><br>
>> Dr. Rutger A. Vos<br>
>> <br>
>> Researcher / Bioinformatician<br>
>> <br>
>> [logo-new]<br>
>><br>
>> [+31717519600 - +31627085806 ]<br>
>> <br>
>> [<a href="mailto:rutger.vos@naturalis.nl" target="_blank">rutger.vos@naturalis.nl</a> - <a href="http://www.naturalis.nl" rel="noreferrer" target="_blank">www.naturalis.nl</a>]<br>
>> <br>
>> [Darwinweg 2, 2333 CR Leiden]<br>
>> <br>
>> [Postbus 9517, 2300 RA Leiden]<br>
>><br>
>> [schildpad]<br>
>><br>
>> --<br>
>><br>
>> Met vriendelijke groet,<br>
>><br>
>> Dr. Rutger A. Vos<br>
>> <br>
>> Researcher / Bioinformatician<br>
>> <br>
>> [logo-new]<br>
>><br>
>> [+31717519600 - +31627085806 ]<br>
>> <br>
>> [<a href="mailto:rutger.vos@naturalis.nl" target="_blank">rutger.vos@naturalis.nl</a> - <a href="http://www.naturalis.nl" rel="noreferrer" target="_blank">www.naturalis.nl</a>]<br>
>> <br>
>> [Darwinweg 2, 2333 CR Leiden]<br>
>> <br>
>> [Postbus 9517, 2300 RA Leiden]<br>
>><br>
>> [schildpad]<br>
>><br>
>> --<br>
>> <br>
>> Met vriendelijke groet,<br>
>> <br>
>> Dr. Rutger A. Vos<br>
>> Researcher / Bioinformatician<br>
>> [logo-new]<br>
>><br>
>> [+31717519600 - +31627085806 ]<br>
>> [<a href="mailto:rutger.vos@naturalis.nl" target="_blank">rutger.vos@naturalis.nl</a> - <a href="http://www.naturalis.nl" rel="noreferrer" target="_blank">www.naturalis.nl</a>]<br>
>> [Darwinweg 2, 2333 CR Leiden]<br>
>> [Postbus 9517, 2300 RA Leiden]<br>
>> <br>
>> [schildpad]<br>
>><br>
>> --<br>
>><br>
>> Met vriendelijke groet,<br>
>><br>
>> Dr. Rutger A. Vos<br>
>> Researcher / Bioinformatician<br>
>> [logo-new]<br>
>><br>
>> [+31717519600 - +31627085806 ]<br>
>> [<a href="mailto:rutger.vos@naturalis.nl" target="_blank">rutger.vos@naturalis.nl</a> - <a href="http://www.naturalis.nl" rel="noreferrer" target="_blank">www.naturalis.nl</a>]<br>
>> [Darwinweg 2, 2333 CR Leiden]<br>
>> [Postbus 9517, 2300 RA Leiden]<br>
>><br>
>> [schildpad]<br>
>><br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">--<br>Carles Fenoy<br></div></div>