[slurm-users] TaskPlugin task/affinity and task/cgroup

[Max Quast]

Dear Slurm users, 

i would like to use the taskplugin cgroups on my system. I have configured
cgroups as recommended by slurm:

 

cgroup.conf:

CgroupAutomount=yes

CgroupMountpoint=/sys/fs/cgroup

TaskAffinity=no

ConstrainCores=yes

 

slurm.conf:

...

TaskPlugin=task/affinity,task/cgroup

#TaskPluginParam=Sched

...

# COMPUTE NODES

GresTypes=gpu

NodeName=lsm[216-217] Gres=gpu:tesla:1 CPUs=64 RealMemory=192073 Sockets=2
CoresPerSocket=16 ThreadsPerCore=2 State=UNKNOWN

PartitionName=admin Nodes=lsm[216-217] Default=YES MaxTime=INFINITE State=UP

 

It works, but the performance is much worse than before with
TaskPlugin=task/none.

 

What I noticed:

With TaskPlugin=task/none the tasks often change the execution unit
(thread). With affinity/cgroups the tasks remain on their assigned thread
from start to finish. 

 

As an example:

With srun -n9 -N2 ./prog and TaskPlugin=task/affinity,task/cgroup, the tasks
are divided among the threads: 

node 1: 1,17,33,49,34

Node 2: 1,17,33,49

Execution time: ~210 sec

 

With srun -n9 -N2 ./prog and TaskPlugin=task/none the taks change the
threads during execution.

execution time: ~140 sec (like manual with mpirun)

 

What are proper parameters for task affinity and cgroups?

 

Thanks for any help :)

-max

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20201020/539f4d8b/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5014 bytes
Desc: not available
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20201020/539f4d8b/attachment.bin>