<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof">
<span class="x_elementToProof x_ContentPasted0 ContentPasted0" style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Dear all,</span>
<div class="x_elementToProof x_ContentPasted0 ContentPasted0" style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Our compute nodes have 128 CPU cores with 8 nvidia GPU cards. <br class="ContentPasted0">
</div>
<span class="x_elementToProof x_ContentPasted0 ContentPasted0" style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> I set 8 numa node like this
</span>.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">
[<span style="color: rgb(102, 102, 102); font-size: 9pt;">root@g0025 ~]# numactl --hardware</span>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">available: 8 nodes (0-7)</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 0 size: 64130 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 0 free: 62086 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 1 size: 64492 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 1 free: 62306 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 2 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 2 size: 64508 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 2 free: 62487 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 3 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 3 size: 64496 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 3 free: 62443 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 4 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 4 size: 64508 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 4 free: 62283 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 5 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 5 size: 64508 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 5 free: 61074 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 6 cpus: 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 6 size: 64508 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 6 free: 62284 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 7 cpus: 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 7 size: 64507 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 7 free: 62354 MB</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node distances:</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;">node 0 1 2 3 4 5 6 7</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;"> 0: 10 12 12 12 32 32 32 32</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;"> 1: 12 10 12 12 32 32 32 32</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;"> 2: 12 12 10 12 32 32 32 32</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;"> 3: 12 12 12 10 32 32 32 32</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;"> 4: 32 32 32 32 10 12 12 12</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;"> 5: 32 32 32 32 12 10 12 12</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;"> 6: 32 32 32 32 12 12 10 12</span></div>
<div class="ContentPasted1"><span style="color: rgb(102, 102, 102); font-size: 9pt;"> 7: 32 32 32 32 12 12 12 10</span></div>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted3">
<span style="font-size: 9pt; color: rgb(102, 102, 102);">[root@g0025 ~]# nvidia-smi topo -m</span>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);"> GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);">GPU0 X SYS SYS SYS SYS SYS SYS SYS 48-63 3</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);">GPU1 SYS X SYS SYS SYS SYS SYS SYS 32-47 2</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);">GPU2 SYS SYS X SYS SYS SYS SYS SYS 16-31 1</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);">GPU3 SYS SYS SYS X SYS SYS SYS SYS 0-15 0</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);">GPU4 SYS SYS SYS SYS X SYS SYS SYS 112-127 7</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);">GPU5 SYS SYS SYS SYS SYS X SYS SYS 96-111 6</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);">GPU6 SYS SYS SYS SYS SYS SYS X SYS 80-95 5</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);">GPU7 SYS SYS SYS SYS SYS SYS SYS X 64-79 4</span></div>
<div><br class="ContentPasted3">
</div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);">Legend:</span></div>
<div><br class="ContentPasted3">
</div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);"> X = Self</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);"> SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);"> NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);"> PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);"> PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);"> PIX = Connection traversing at most a single PCIe bridge</span></div>
<div class="ContentPasted3"><span style="font-size: 9pt; color: rgb(102, 102, 102);"> NV# = Connection traversing a bonded set of # NVLinks</span></div>
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">
<span class="x_elementToProof x_ContentPasted0 ContentPasted2" style="font-size:12pt;color:rgb(0, 0, 0) !important">Users can submit job one GPU with 6~16 CPU cores .</span><br class="ContentPasted2">
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">
<span class="x_elementToProof x_ContentPasted0 ContentPasted2" style="font-size:12pt;color:rgb(0, 0, 0) !important">And I set gres.conf like this .</span><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted4">
<span style="color: rgb(102, 102, 102); font-size: 9pt;">[root@g0038 ~]# cat /etc/slurm/gres.conf</span>
<div class="ContentPasted4"><span style="color: rgb(102, 102, 102); font-size: 9pt;">Name=gpu File=/dev/nvidia0 COREs=0-15</span></div>
<div class="ContentPasted4"><span style="color: rgb(102, 102, 102); font-size: 9pt;">Name=gpu File=/dev/nvidia1 COREs=16-31</span></div>
<div class="ContentPasted4"><span style="color: rgb(102, 102, 102); font-size: 9pt;">Name=gpu File=/dev/nvidia2 COREs=32-47</span></div>
<div class="ContentPasted4"><span style="color: rgb(102, 102, 102); font-size: 9pt;">Name=gpu File=/dev/nvidia3 COREs=48-63</span></div>
<div class="ContentPasted4"><span style="color: rgb(102, 102, 102); font-size: 9pt;">Name=gpu File=/dev/nvidia4 COREs=64-79</span></div>
<div class="ContentPasted4"><span style="color: rgb(102, 102, 102); font-size: 9pt;">Name=gpu File=/dev/nvidia5 COREs=80-95</span></div>
<div class="ContentPasted4"><span style="color: rgb(102, 102, 102); font-size: 9pt;">Name=gpu File=/dev/nvidia6 COREs=96-111</span></div>
<div class="ContentPasted4"><span style="color: rgb(102, 102, 102); font-size: 9pt;">Name=gpu File=/dev/nvidia7 COREs=112-127</span></div>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted4">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted4">
<span class="x_elementToProof x_ContentPasted0 x_ContentPasted4 ContentPasted5" style="font-size:12pt;color:rgb(0, 0, 0) !important"><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted4">
<span class="x_elementToProof x_ContentPasted0 x_ContentPasted4 ContentPasted5" style="font-size:12pt;color:rgb(0, 0, 0) !important">How to realize this function .</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted4">
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted4 ContentPasted5" style="font-size:12pt;color:rgb(0, 0, 0) !important">
For example ,</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted4 x_ContentPasted5 ContentPasted5" style="font-size:12pt;color:rgb(0, 0, 0) !important">
A job requires 6 CPUs with 1 GPU .And it runs on gpu ID 0 , CPU ID 0-5 .</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted4 x_ContentPasted5 x_ContentPasted6 ContentPasted5" style="font-size:12pt;color:rgb(0, 0, 0) !important">
The second job requires 8 CPUs with 1 GPU . If it runs on gpu ID 1 ,we hope the CPU ID is 16-23.</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted4 x_ContentPasted5 x_ContentPasted6 x_ContentPasted7 x_ContentPasted8 ContentPasted5" style="font-size:12pt;color:rgb(0, 0, 0) !important">
The third job requires 6 CPUs with 1 GPU . If it runs on gpu ID 2 ,we hope the CPU ID is 32-37.</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted4 x_ContentPasted5 x_ContentPasted6 x_ContentPasted7 x_ContentPasted8 x_ContentPasted9 ContentPasted5" style="font-size:12pt;color:rgb(0, 0, 0) !important">
The next job requires 12 CPUs with 2 GPU . If it runs on gpu ID 3-4 ,we hope the CPU ID is 48-53,64-69 .
<br class="ContentPasted5">
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-size:12pt;color:rgb(0, 0, 0) !important">
<br class="ContentPasted5">
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-size:12pt;color:rgb(0, 0, 0) !important">
<br class="ContentPasted5">
</div>
<span class="x_elementToProof x_ContentPasted0 x_ContentPasted10 ContentPasted5" style="font-size:12pt;color:rgb(0, 0, 0) !important">Can we implement this function ?</span><br>
</div>
</body>
</html>