[slurm-users] one job at a time - how to set?

[Rutger Vos]

I did some tests and this doesn't work with MPI, in the following sense:
the way we were running parallel jobs was by invoking `sbatch -n4
script.sh` where the script.sh would contain an `mpirun` command without
specifying the number of processors. It would then simply claim the ones
assigned to it by sbatch. With one CPU as per the slurm.conf, mpirun would
only ever get one processor, hence no parallelization.

The other test I did was to try OverSubscribe=Exclusive on the partition,
but the slurmctld failed to start so there is some combination of config
options that it is unhappy about.

The workable option I can think of right now is to have everyone run their
jobs with --exclusive as an extra argument.

On Thu, Apr 30, 2020 at 12:22 PM Rutger Vos <rutger.vos at naturalis.nl> wrote:

> This seems like the easiest solution, but how would that interact with MPI
> jobs? Apologies if these are really basic questions, and thank you very
> much for thinking along.
>
> Rutger
>
> On Wed, Apr 29, 2020 at 10:25 PM Doug Meyer <dameyer at raytheon.com> wrote:
>
>> Change node definition in slurm.conf for that one node to 1 CPU.
>>
>>
>>
>> *Doug Meyer*
>>
>>
>>
>> *From:* slurm-users <slurm-users-bounces at lists.schedmd.com> *On Behalf
>> Of *Rutger Vos
>> *Sent:* Wednesday, April 29, 2020 1:20 PM
>> *To:* Slurm User Community List <slurm-users at lists.schedmd.com>
>> *Subject:* [External] Re: [slurm-users] one job at a time - how to set?
>>
>>
>>
>> Hi Michael,
>>
>>
>>
>> thanks very much for your swift reply. So here we would have to convince
>> the users they'd have to specify this when submitting, right? I.e. 'sbatch
>> --exclusive myjob.sh', if I understand correctly. Would there be a way to
>> simply enforce this, i.e. at the slurm.conf level or something?
>>
>>
>>
>> Thanks again!
>>
>>
>>
>> Rutger
>>
>>
>>
>> On Wed, Apr 29, 2020 at 10:06 PM Renfro, Michael <Renfro at tntech.edu>
>> wrote:
>>
>> That’s a *really* old version, but
>> https://slurm.schedmd.com/archive/slurm-15.08.13/sbatch.html indicates
>> there’s an exclusive flag you can set.
>>
>> On Apr 29, 2020, at 1:54 PM, Rutger Vos <rutger.vos at naturalis.nl> wrote:
>>
>> *.*
>>
>> Hi,
>>
>>
>>
>> for a smallish machine that has been having degraded performance we want
>> to implement a policy where only one job (submitted with sbatch) is allowed
>> to run and any others submitted after it are supposed to wait in line.
>>
>>
>>
>> I assumed this was straightforward but I can't seem to figure it out. Can
>> I set that up in slurm.conf or in some other way? Thank you very much for
>> your help. BTW we are running slurm 15.08.7 if that is at all relevant.
>>
>>
>>
>> Best wishes,
>>
>>
>>
>> Dr. Rutger A. Vos
>>
>> Researcher / Bioinformatician
>>
>>
>>
>>
>>
>>
>>
>> +31717519600 - +31627085806
>>
>> rutger.vos at naturalis.nl - www.naturalis.nl
>>
>> Darwinweg 2, 2333 CR Leiden
>>
>> Postbus 9517, 2300 RA Leiden
>>
>>
>>
>> <https://www.naturalis.nl/lang-leve>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>> Met vriendelijke groet,
>>
>>
>>
>> Dr. Rutger A. Vos
>>
>> Researcher / Bioinformatician
>>
>>
>>
>>
>>
>>
>>
>> +31717519600 - +31627085806
>>
>> rutger.vos at naturalis.nl - www.naturalis.nl
>>
>> Darwinweg 2, 2333 CR Leiden
>>
>> Postbus 9517, 2300 RA Leiden
>>
>>
>>
>> <https://www.naturalis.nl/lang-leve>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
>
> Met vriendelijke groet,
>
> Dr. Rutger A. Vos
> Researcher / Bioinformatician
>
>
>
>
>
>
> +31717519600 - +31627085806
> rutger.vos at naturalis.nl - www.naturalis.nl
> Darwinweg 2, 2333 CR Leiden
> Postbus 9517, 2300 RA Leiden
>
> <https://www.naturalis.nl/over-ons> <https://www.naturalis.nl/lang-leve>
>
>
>
>
>
>
>
>
>
>

-- 

Met vriendelijke groet,

Dr. Rutger A. Vos
Researcher / Bioinformatician






+31717519600 - +31627085806
rutger.vos at naturalis.nl - www.naturalis.nl
Darwinweg 2, 2333 CR Leiden
Postbus 9517, 2300 RA Leiden

<https://www.naturalis.nl/over-ons> <https://www.naturalis.nl/lang-leve>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20200430/f5ef68f9/attachment.htm>