<div dir="ltr"><br><div class="gmail_quote"><div dir="ltr"><div><div>Dear everyone,<br><br></div>Greetings!!!!<br><br></div>Answer to my post:<br><br>Actually slurmctld uses best-fit approach on the available resources on each node. It does not obey our specified cpu map mask to assign task to the logical CPUs.<br><br>I have added/Modified code to fulfil my requirement. Here is my experiment result.<br><br><b>$ srun -n 4 --cpu_bind=verbose,map_cpu:0,<wbr>1,8,9 --distribution=block:block --mem=1024 sleep 10</b><br>cpu_bind=MAP - clusterhost1, task 0 0 [3334]: mask 0x1 set<br>cpu_bind=MAP - clusterhost1, task 1 1 [3335]: mask 0x2 set<br>cpu_bind=MAP - clusterhost1, task 2 2 [3336]: mask 0x100 set<br>cpu_bind=MAP - clusterhost1, task 3 3 [3337]: mask 0x200 set<br><br><b>$ srun -n 16 --cpu_bind=verbose,map_cpu:0,<wbr>1,2,3,4,5,6,7,16,17,18,19,20,<wbr>21,22,23 --distribution=block:block --mem=1024 sleep 10</b><br>cpu_bind=MAP - clusterhost1, task 0 0 [3084]: mask 0x1 set<br>cpu_bind=MAP - clusterhost1, task 1 1 [3085]: mask 0x2 set<br>cpu_bind=MAP - clusterhost1, task 2 2 [3086]: mask 0x4 set<br>cpu_bind=MAP - clusterhost1, task 3 3 [3087]: mask 0x8 set<br>cpu_bind=MAP - clusterhost1, task 12 12 [3097]: mask 0x100000 set<br>cpu_bind=MAP - clusterhost1, task 15 15 [3100]: mask 0x800000 set<br>cpu_bind=MAP - clusterhost1, task 9 9 [3094]: mask 0x20000 set<br>cpu_bind=MAP - clusterhost1, task 11 11 [3096]: mask 0x80000 set<br>cpu_bind=MAP - clusterhost1, task 4 4 [3088]: mask 0x10 set<br>cpu_bind=MAP - clusterhost1, task 10 10 [3095]: mask 0x40000 set<br>cpu_bind=MAP - clusterhost1, task 13 13 [3098]: mask 0x200000 set<br>cpu_bind=MAP - clusterhost1, task 5 5 [3089]: mask 0x20 set<br>cpu_bind=MAP - clusterhost1, task 7 7 [3091]: mask 0x80 set<br>cpu_bind=MAP - clusterhost1, task 14 14 [3099]: mask 0x400000 set<br>cpu_bind=MAP - clusterhost1, task 8 8 [3093]: mask 0x10000 set<br>cpu_bind=MAP - clusterhost1, task 6 6 [3090]: mask 0x40 set<br><br><b>$ srun -n 16 --cpu_bind=verbose,map_cpu:8,<wbr>9,10,11,12,13,14,15,24,25,26,<wbr>27,28,29,30,31 --distribution=block:block --mem=1024 sleep 10</b><br>cpu_bind=MAP - clusterhost1, task 2 2 [3157]: mask 0x400 set<br>cpu_bind=MAP - clusterhost1, task 0 0 [3155]: mask 0x100 set<br>cpu_bind=MAP - clusterhost1, task 1 1 [3156]: mask 0x200 set<br>cpu_bind=MAP - clusterhost1, task 3 3 [3158]: mask 0x800 set<br>cpu_bind=MAP - clusterhost1, task 4 4 [3159]: mask 0x1000 set<br>cpu_bind=MAP - clusterhost1, task 5 5 [3160]: mask 0x2000 set<br>cpu_bind=MAP - clusterhost1, task 6 6 [3161]: mask 0x4000 set<br>cpu_bind=MAP - clusterhost1, task 14 14 [3169]: mask 0x40000000 set<br>cpu_bind=MAP - clusterhost1, task 7 7 [3162]: mask 0x8000 set<br>cpu_bind=MAP - clusterhost1, task 13 13 [3168]: mask 0x20000000 set<br>cpu_bind=MAP - clusterhost1, task 12 12 [3167]: mask 0x10000000 set<br>cpu_bind=MAP - clusterhost1, task 8 8 [3163]: mask 0x1000000 set<br>cpu_bind=MAP - clusterhost1, task 9 9 [3164]: mask 0x2000000 set<br>cpu_bind=MAP - clusterhost1, task 15 15 [3170]: mask 0x80000000 set<br>cpu_bind=MAP - clusterhost1, task 10 10 [3165]: mask 0x4000000 set<br>cpu_bind=MAP - clusterhost1, task 11 11 [3166]: mask 0x8000000 set<br><br><b>$ srun -n 16 --cpu_bind=verbose --mem=1024 sleep 10<br></b>cpu_bind=MASK - clusterhost1, task 2 2 [3207]: mask 0x2 set<br>cpu_bind=MASK - clusterhost1, task 15 15 [3220]: mask 0x800000 set<br>cpu_bind=MASK - clusterhost1, task 3 3 [3208]: mask 0x20000 set<br>cpu_bind=MASK - clusterhost1, task 4 4 [3209]: mask 0x4 set<br>cpu_bind=MASK - clusterhost1, task 10 10 [3215]: mask 0x20 set<br>cpu_bind=MASK - clusterhost1, task 11 11 [3216]: mask 0x200000 set<br>cpu_bind=MASK - clusterhost1, task 12 12 [3217]: mask 0x40 set<br>cpu_bind=MASK - clusterhost1, task 13 13 [3218]: mask 0x400000 set<br>cpu_bind=MASK - clusterhost1, task 14 14 [3219]: mask 0x80 set<br>cpu_bind=MASK - clusterhost1, task 1 1 [3206]: mask 0x10000 set<br>cpu_bind=MASK - clusterhost1, task 5 5 [3210]: mask 0x40000 set<br>cpu_bind=MASK - clusterhost1, task 0 0 [3205]: mask 0x1 set<br>cpu_bind=MASK - clusterhost1, task 6 6 [3211]: mask 0x8 set<br>cpu_bind=MASK - clusterhost1, task 7 7 [3212]: mask 0x80000 set<br>cpu_bind=MASK - clusterhost1, task 9 9 [3214]: mask 0x100000 set<br>cpu_bind=MASK - clusterhost1, task 8 8 [3213]: mask 0x10 set<br><br><b>$ srun -n 4 --cpu_bind=verbose --mem=1024 sleep 10</b><br>cpu_bind=MASK - clusterhost1, task 3 3 [3266]: mask 0x20000 set<br>cpu_bind=MASK - clusterhost1, task 2 2 [3265]: mask 0x2 set<br>cpu_bind=MASK - clusterhost1, task 0 0 [3263]: mask 0x1 set<br>cpu_bind=MASK - clusterhost1, task 1 1 [3264]: mask 0x10000 set<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 27, 2017 at 1:23 PM, Animesh Kuity <span dir="ltr"><<a href="mailto:animesh2kuity@gmail.com" target="_blank">animesh2kuity@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi everyone,<br><br>My objective: I want to assign few tasks to the logical CPUs belong to a particular socket(e.g., say socket 0) and at other time, I want to assign another set of tasks to the logical CPUs belongs to another socket (e.g., say socket 0). In summary, I want to achieve task affinity to a particular logical CPU<br><br>slurm version used: slurm 16.05.10-2 <br><br>slurm.conf to achieve task affinity:<br><br>SelectType=select/cons_res<br>SelectTypeParameters=CR_Core<br>TaskPlugin=task/affinity<br>TaskPluginParam=sched<br><br>Node used: Xeon processor; two sockets each having 8 cores with 2 threads/core<br> <br>Processor layout(/proc/cpuinfo): <br>processor physical id core id<br>0,16 0 0<br>1,17 0 1<br>2,18 0 2<br>3,19 0 3<br>4,20 0 4<br>5,21 0 5<br>6,22 0 6<br>7,23 0 7<br>8,24 1 0<br>9,25 1 1<br>10,26 1 2<br>11,27 1 3<br>12,28 1 4<br>13,29 1 5<br>14,30 1 6<br>15,31 1 7<br><br>Question: <span style="background-color:rgb(255,255,255)"><b><span style="color:rgb(255,0,0)">I am unable to assign all the tasks to the particular logical CPUs belong to socket 0/ Socket 1</span> </b></span><br><br>The tasks are always assigning to the sockets 0 first irrespective of the specified map_cpu before going to socket 1<br><br><b>My observation:</b><br><br><b>$ srun -n 8 --cpu_bind=verbose,map_cpu:0,1<wbr>,2,3,16,17,18,19 --distribution=block:block --mem=1024 sleep 100 &</b><br>[1] 14665<br>cpu_bind=MASK - clusterhost1, task 0 0 [14697]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 1 1 [14698]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 4 4 [14701]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 2 2 [14699]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 3 3 [14700]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 5 5 [14702]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 6 6 [14703]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 7 7 [14704]: mask 0xf000f set<br><b>$ srun bash -c "cat /proc/self/status | grep Cpus_allowed_list"</b><br>Cpus_allowed_list: 4,20<br><br><br><b>$ srun -n 8 --cpu_bind=verbose,map_cpu:0,1<wbr>,2,3,4,5,6,7 --distribution=block:block --mem=1024 sleep 100 &</b><br>[1] 14814<br>cpu_bind=MASK - clusterhost1, task 1 1 [14847]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 2 2 [14848]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 3 3 [14849]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 0 0 [14846]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 5 5 [14851]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 6 6 [14852]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 4 4 [14850]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 7 7 [14853]: mask 0xf000f set<br><b>$ srun bash -c "cat /proc/self/status | grep Cpus_allowed_list"</b><br>Cpus_allowed_list: 4,20<br><br><b>$ srun -n 20 --cpu_bind=verbose,map_cpu:0,1<wbr>,2,3,4,5,6,7,9,10,11,12,13,14,<wbr>15,16,17,18,19 --distribution=block:block --mem=1024 sleep 100 &</b><br>[1] 15688<br>cpu_bind=MASK - clusterhost1, task 1 1 [15721]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 2 2 [15722]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 4 4 [15724]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 5 5 [15725]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 7 7 [15727]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 0 0 [15720]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 6 6 [15726]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 3 3 [15723]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 10 10 [15730]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 8 8 [15728]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 9 9 [15729]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 11 11 [15731]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 12 12 [15732]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 14 14 [15734]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 13 13 [15733]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 15 15 [15735]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 16 16 [15736]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 17 17 [15737]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 18 18 [15738]: mask 0x3ff03ff set<br>cpu_bind=MASK - clusterhost1, task 19 19 [15739]: mask 0x3ff03ff set<br><b>$ srun bash -c "cat /proc/self/status | grep Cpus_allowed_list"</b><br>Cpus_allowed_list: 10,26<br><br><b>$ srun -n 8 --cpu_bind=verbose,map_cpu:8,9<wbr>,10,11,24,25,26,27 --distribution=block:block --mem=1024 sleep 100 &</b><br>[1] 16816<br>cpu_bind=MASK - clusterhost1, task 1 1 [16850]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 4 4 [16853]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 3 3 [16852]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 2 2 [16851]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 0 0 [16849]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 6 6 [16855]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 5 5 [16854]: mask 0xf000f set<br>cpu_bind=MASK - clusterhost1, task 7 7 [16856]: mask 0xf000f set<br><br><b>$ srun bash -c "cat /proc/self/status | grep Cpus_allowed_list"</b><br>Cpus_allowed_list: 4,20<br><br><b>$ srun --nodes=1 --ntasks=32 --cpu_bind=cores,verbose --label cat /proc/self/status | grep Cpus_allowed_list</b><br>00: cpu_bind=MASK - clusterhost1, task 0 0 [13955]: mask 0x10001 set<br>01: cpu_bind=MASK - clusterhost1, task 1 1 [13956]: mask 0x20002 set<br>04: cpu_bind=MASK - clusterhost1, task 4 4 [13959]: mask 0x100010 set<br>05: cpu_bind=MASK - clusterhost1, task 5 5 [13960]: mask 0x200020 set<br>06: cpu_bind=MASK - clusterhost1, task 6 6 [13961]: mask 0x400040 set<br>03: cpu_bind=MASK - clusterhost1, task 3 3 [13958]: mask 0x80008 set<br>02: cpu_bind=MASK - clusterhost1, task 2 2 [13957]: mask 0x40004 set<br>09: cpu_bind=MASK - clusterhost1, task 9 9 [13964]: mask 0x2000200 set<br>07: cpu_bind=MASK - clusterhost1, task 7 7 [13962]: mask 0x800080 set<br>10: cpu_bind=MASK - clusterhost1, task 10 10 [13965]: mask 0x4000400 set<br>11: cpu_bind=MASK - clusterhost1, task 11 11 [13966]: mask 0x8000800 set<br>14: cpu_bind=MASK - clusterhost1, task 14 14 [13969]: mask 0x40004000 set<br>15: cpu_bind=MASK - clusterhost1, task 15 15 [13970]: mask 0x80008000 set<br>12: cpu_bind=MASK - clusterhost1, task 12 12 [13967]: mask 0x10001000 set<br>13: cpu_bind=MASK - clusterhost1, task 13 13 [13968]: mask 0x20002000 set<br>08: cpu_bind=MASK - clusterhost1, task 8 8 [13963]: mask 0x1000100 set<br>17: cpu_bind=MASK - clusterhost1, task 17 17 [13972]: mask 0x20002 set<br>16: cpu_bind=MASK - clusterhost1, task 16 16 [13971]: mask 0x10001 set<br>20: cpu_bind=MASK - clusterhost1, task 20 20 [13975]: mask 0x100010 set<br>19: cpu_bind=MASK - clusterhost1, task 19 19 [13974]: mask 0x80008 set<br>18: cpu_bind=MASK - clusterhost1, task 18 18 [13973]: mask 0x40004 set<br>22: cpu_bind=MASK - clusterhost1, task 22 22 [13977]: mask 0x400040 set<br>21: cpu_bind=MASK - clusterhost1, task 21 21 [13976]: mask 0x200020 set<br>24: cpu_bind=MASK - clusterhost1, task 24 24 [13979]: mask 0x1000100 set<br>25: cpu_bind=MASK - clusterhost1, task 25 25 [13980]: mask 0x2000200 set<br>23: cpu_bind=MASK - clusterhost1, task 23 23 [13978]: mask 0x800080 set<br>26: cpu_bind=MASK - clusterhost1, task 26 26 [13981]: mask 0x4000400 set<br>30: cpu_bind=MASK - clusterhost1, task 30 30 [13985]: mask 0x40004000 set<br>31: cpu_bind=MASK - clusterhost1, task 31 31 [13986]: mask 0x80008000 set<br>28: cpu_bind=MASK - clusterhost1, task 28 28 [13983]: mask 0x10001000 set<br>29: cpu_bind=MASK - clusterhost1, task 29 29 [13984]: mask 0x20002000 set<br>27: cpu_bind=MASK - clusterhost1, task 27 27 [13982]: mask 0x8000800 set<br>03: Cpus_allowed_list: 3,19<br>04: Cpus_allowed_list: 4,20<br>01: Cpus_allowed_list: 1,17<br>06: Cpus_allowed_list: 6,22<br>00: Cpus_allowed_list: 0,16<br>02: Cpus_allowed_list: 2,18<br>05: Cpus_allowed_list: 5,21<br>09: Cpus_allowed_list: 9,25<br>10: Cpus_allowed_list: 10,26<br>14: Cpus_allowed_list: 14,30<br>11: Cpus_allowed_list: 11,27<br>15: Cpus_allowed_list: 15,31<br>12: Cpus_allowed_list: 12,28<br>13: Cpus_allowed_list: 13,29<br>17: Cpus_allowed_list: 1,17<br>07: Cpus_allowed_list: 7,23<br>16: Cpus_allowed_list: 0,16<br>08: Cpus_allowed_list: 8,24<br>20: Cpus_allowed_list: 4,20<br>19: Cpus_allowed_list: 3,19<br>18: Cpus_allowed_list: 2,18<br>21: Cpus_allowed_list: 5,21<br>22: Cpus_allowed_list: 6,22<br>24: Cpus_allowed_list: 8,24<br>23: Cpus_allowed_list: 7,23<br>26: Cpus_allowed_list: 10,26<br>30: Cpus_allowed_list: 14,30<br>31: Cpus_allowed_list: 15,31<br>25: Cpus_allowed_list: 9,25<br>28: Cpus_allowed_list: 12,28<br>29: Cpus_allowed_list: 13,29<br>27: Cpus_allowed_list: 11,27<br><br><br></div><b><span style="color:rgb(255,0,0)">Kindly help me to assign all the tasks to either socket.</span></b><br clear="all"><div><br></div><div>Any kind of help will be appreciated.<br><br></div><div>Thanks in advance.<span class="m_6136385988941237212HOEnZb"><font color="#888888"><br><br></font></span></div><span class="m_6136385988941237212HOEnZb"><font color="#888888"><div>-- <br><div class="m_6136385988941237212m_-3300999133302058460gmail_signature"><div dir="ltr"><div>Thanks & Regards,<br>Animesh Kuity,<br></div><div>Research Scholar,<br>Computer Science department,<br>IIT Roorkee<br></div></div></div>
</div></font></span></div>
</blockquote></div><br><br clear="all"><br></div></div></div></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Thanks & Regards,<br>Animesh Kuity,<br></div><div>Research Scholar,<br>Computer Science department,<br>IIT Roorkee<br></div></div></div>
</div>