[slurm-users] Meaning of --cpus-per-task and --mem-per-cpu when SMT processors are used

[William Brown]

What Marcus reports is quite correct.   It can be confusing, and Slurm uses
'CPU' I think as a non-specific term to mean 'the smallest assignable
compute object'.   With SMT enabled that is the thread, and with it
disabled it is the core.

We were told by the company that installed the cluster at my last employer
that they always turned off SMT as that was best practice, which they had
numbers to prove.  The problem with SMT on most processors is that there
are shared resources between the threads, and the context switching is bad
for straight line performance.  For compute clusters, the performance is
usually what is wanted more than the threads.

So if you disable SMT, it gets a little easier as CPU=Core.

William

On Wed, 4 Mar 2020 at 11:47, Marcus Wagner <wagner at itc.rwth-aachen.de>
wrote:

> Hi Loris,
>
> CPU is the smallest schedulable unit, in case of SMT its threads.
>
> At the moment we have HT disabled on our systems, therefore CPU is equal
> to the cores for us. But with HT enabled, CPU is double that large (at
> least form slurm 18.08).
>
>
> Best
> Marcus
>
> On 3/4/20 10:33 AM, Loris Bennett wrote:
> > Hi Alexander,
> >
> > Alexander Grund <alexander.grund at tu-dresden.de> writes:
> >
> >> Hi,
> >>
> >> we have a Power9 partition with 44 processors having 4 cores each
> >> totaling 176.
> >>
> >> `scontrol show node <node>` shows "CoresPerSocket=22" and "CPUTot=176"
> >> which confuses me. Especially as `whypending` reports e.g. "172 cores
> >> free: 1"
> > What's 'whypending'?
> >
> >> So what are "CPUs" and what are "Cores" to SLURM? Why does it mix up
> those 2?
> > This is just historical as far as I can tell.  I think 'CPU' almost
> > always means 'core'.
> >
> >> Most importantly: Does this mean `--cpus-per-task` can be as high as
> >> 176 on this node and `--mem-per-cpu` can be up to the reported
> >> "RealMemory"/176?
> > Yes.
> >
> > Cheers,
> >
> > Loris
> >
>
> --
> Marcus Wagner, Dipl.-Inf.
>
> IT Center
> Abteilung: Systeme und Betrieb
> RWTH Aachen University
> Seffenter Weg 23
> 52074 Aachen
> Tel: +49 241 80-24383
> Fax: +49 241 80-624383
> wagner at itc.rwth-aachen.de
> www.itc.rwth-aachen.de
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20200304/75d83936/attachment.htm>