[slurm-users] How to allocate SMT cores

[Eli V]

On Fri, Dec 7, 2018 at 7:53 AM Maik Schmidt <maik.schmidt at tu-dresden.de> wrote:
>
> I have found --hint=multithread, but this only works with task/affinity.
> We use task/cgroup. Are there any downsides to activating both task
> plugins at the same time?
>
> Best, Maik
>
> Am 07.12.18 um 13:33 schrieb Maik Schmidt:
> > Hi all,
> >
> > we recently got ourselves some Power9 nodes with 4-way SMT. However,
> > other than using --exclusive I cannot seem to find a possibility to
> > make SLURM allocate all SMT threads for me. There simply does not seem
> > to exist a parameter for that. One might think that --threads-per-core
> > would be right, but this only limits the job to run on nodes that have
> > ThreadsPerCore=x set, it does not actually allocate the CPUs.
> >
> > Also, I cannot use more than -c 44 even though the node has
> > CPUTot=176, so it's not possible to allocate all (virtual) cores.

I'd check your slurmd.log, it might already be doing it seeing as 44 *
4 == 176. I think slurm used to auto assign all threads on a core to a
job, and noted it in slurmd.log.(or maybe slurmctld.log) I think that
has changed though, since I've recently upgraded to 18.03 from 17.02
and no longer see this. sbatch -n 64 -N 1 and sbatch --cpus-per-task
64 -N 1 both ran for me reserving all 64 threads on my test 32 core
host. I use SelectTypeParameters=CR_Core_Memory,CR_LLN

> >
> > So, how can this be done?
> >
> > Here are some relevant SLURM config settings:
> >
> > NodeName=taurusml[1-22] Feature=IB Gres=gpu:6 Procs=176 Sockets=2
> > CoresPerSocket=22 ThreadsPerCore=4 RealMemory=450000 Weight=128
> >
> > TaskPlugin              = task/cgroup
> > TaskPluginParam         = cpusets,autobind=threads
> >
> > Still on SLURM 17.02.11 if that is relevant.
> >
> > Thanks in advance.
> >
>
> --
> Maik Schmidt
> HPC Services
>
> Technische Universität Dresden
> Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH)
> Willers-Bau A116
> D-01062 Dresden
> Telefon: +49 351 463-32836
>
>