<html><head><style>p{margin-top:0px;margin-bottom:0px;}</style></head><body><div style="font-size:10pt; font-family:Gulim, sans-serif;"><p>Hi</p><p>I'm using slurm with GRES(4 GPU).</p><p> </p><p>I wanna allocate jobs uniformly through GRES(especially GPU). But, It does not work when I use Docker.</p><p> </p><p>For example,</p><p> </p><p>If i run this command for 4 times with different tty, I could get what i want to get.</p><p> </p><p>As you can see, All Bus-Ids are different.</p><p> </p><table class="__se_tbl" border="0" cellpadding="0" cellspacing="1" _se2_tbl_template="1" style="background-color: rgb(199, 199, 199);"><tbody>
<tr><td style="padding: 3px 4px 2px; background-color: rgb(255, 255, 255); width: 1559px; height: 19px;"><p style="color: rgb(102, 102, 102);">#1</p><p style="color: rgb(102, 102, 102);"> </p><p style="color: rgb(102, 102, 102);">$ srun --gres=gpu:1 --gres-flags=enforce-binding --cpus-per-task=8 --mem=20G --pty bash<br>$ nvidia-smi</p><p style=""><font color="#666666">Wed Jan 2 01:02:00 2019 </font></p><p style=""><font color="#666666">+-----------------------------------------------------------------------------+</font></p><p style=""><font color="#666666">| NVIDIA-SMI 410.79 Driver Version: 410.79 CUDA Version: 10.0 |</font></p><p style=""><font color="#666666">|-------------------------------+----------------------+----------------------+</font></p><p style=""><font color="#666666">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</font></p><p style=""><font color="#666666">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</font></p><p style=""><font color="#666666">|===============================+======================+======================|</font></p><p style=""><font color="#666666">| 0 Tesla P100-PCIE... On | <b><span style="color: rgb(255, 0, 0);">00000000:14:00.0</span></b> Off | 0 |</font></p><p style=""><font color="#666666">| N/A 30C P0 27W / 250W | 0MiB / 16280MiB | 0% Default |</font></p><p style=""><font color="#666666">+-------------------------------+----------------------+----------------------+</font></p><p style=""><font color="#666666"> </font></p><p style=""><font color="#666666">+-----------------------------------------------------------------------------+</font></p><p style=""><font color="#666666">| Processes: GPU Memory |</font></p><p style=""><font color="#666666">| GPU PID Type Process name Usage |</font></p><p style=""><font color="#666666">|=============================================================================|</font></p><p style=""><font color="#666666">| No running processes found |</font></p><p style=""><font color="#666666">+-----------------------------------------------------------------------------+</font></p><p style=""><font color="#666666"><br></font></p><p style=""><span style="color: rgb(102, 102, 102); font-size: 13.3333px;">#2</span><font color="#666666"><br></font></p><p style=""><span style="color: rgb(102, 102, 102); font-size: 13.3333px;"> </span></p><p style=""><span style="color: rgb(102, 102, 102); font-size: 13.3333px;">$ srun --gres=gpu:1 --gres-flags=enforce-binding --cpus-per-task=8 --mem=20G --pty bash</span><font color="#666666"><br></font></p><p style=""><font color="#666666">$ nvidia-smi </font></p><p><font color="#666666">Wed Jan 2 01:02:39 2019 </font></p><p><font color="#666666">+-----------------------------------------------------------------------------+</font></p><p><font color="#666666">| NVIDIA-SMI 410.79 Driver Version: 410.79 CUDA Version: 10.0 |</font></p><p><font color="#666666">|-------------------------------+----------------------+----------------------+</font></p><p><font color="#666666">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</font></p><p><font color="#666666">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</font></p><p><font color="#666666">|===============================+======================+======================|</font></p><p><font color="#666666">| 0 Tesla P100-PCIE... On | <b><span style="color: rgb(255, 0, 0);">00000000:15:00.0</span></b> Off | 0 |</font></p><p><font color="#666666">| N/A 32C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |</font></p><p><font color="#666666">+-------------------------------+----------------------+----------------------+</font></p><p><font color="#666666"> </font></p><p><font color="#666666">+-----------------------------------------------------------------------------+</font></p><p><font color="#666666">| Processes: GPU Memory |</font></p><p><font color="#666666">| GPU PID Type Process name Usage |</font></p><p><font color="#666666">|=============================================================================|</font></p><p><font color="#666666">| No running processes found |</font></p><p><font color="#666666">+-----------------------------------------------------------------------------+</font></p><p><font color="#666666"><br></font></p><p><font color="#666666">#3</font></p><p><font color="#666666"><br></font></p><p style="font-size: 13.3333px;"><span style="color: rgb(102, 102, 102); font-size: 13.3333px;">$ srun --gres=gpu:1 --gres-flags=enforce-binding --cpus-per-task=8 --mem=20G --pty bash</span><font color="#666666"><br></font></p><p style="font-size: 13.3333px;"><font color="#666666">$ nvidia-smi</font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">Wed Jan 2 00:36:22 2019 </span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-----------------------------------------------------------------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| NVIDIA-SMI 410.79 Driver Version: 410.79 CUDA Version: 10.0 |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">|-------------------------------+----------------------+----------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">|===============================+======================+======================|</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| 0 Tesla P100-PCIE... On | <span style="color: rgb(255, 0, 0);"><b>00000000:39:00.0</b></span> Off | 0 |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| N/A 30C P0 27W / 250W | 0MiB / 16280MiB | 0% Default |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-------------------------------+----------------------+----------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;"> </span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-----------------------------------------------------------------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| Processes: GPU Memory |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| GPU PID Type Process name Usage |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">|=============================================================================|</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| No running processes found |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-----------------------------------------------------------------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;"><br></span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">#4</span></font></p><p style=""> </p><p style="font-size: 13.3333px;"><span style="color: rgb(102, 102, 102); font-size: 13.3333px;">$ srun --gres=gpu:1 --gres-flags=enforce-binding --cpus-per-task=8 --mem=20G --pty bash</span><font color="#666666"><br></font></p><p style="font-size: 13.3333px;"><font color="#666666">$ nvidia-smi</font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">Wed Jan 2 01:03:50 2019 </span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-----------------------------------------------------------------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| NVIDIA-SMI 410.79 Driver Version: 410.79 CUDA Version: 10.0 |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">|-------------------------------+----------------------+----------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">|===============================+======================+======================|</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| 0 Tesla P100-PCIE... On | <span style="color: rgb(255, 0, 0);"><b>00000000:3A:00.0</b></span> Off | 0 |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| N/A 29C P0 27W / 250W | 0MiB / 16280MiB | 0% Default |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-------------------------------+----------------------+----------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;"> </span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-----------------------------------------------------------------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| Processes: GPU Memory |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| GPU PID Type Process name Usage |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">|=============================================================================|</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| No running processes found |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-----------------------------------------------------------------------------+</span></font></p></td>
</tr>
</tbody>
</table><p> </p><p>Also scontrol show Command is OK.</p><p> </p><p>All of GRES_IDXs are different.</p><p> </p><table class="__se_tbl" border="0" cellpadding="0" cellspacing="1" _se2_tbl_template="1" style="background-color: rgb(199, 199, 199);"><tbody>
<tr><td width="1569" style="padding: 3px 4px 2px; background-color: rgb(255, 255, 255);"><p style=""><font color="#666666">$ scontrol show job=472 --details</font></p><p style=""><font color="#666666">JobId=472 JobName=bash</font></p><p style=""><font color="#666666"> UserId=root(0) GroupId=root(0) MCS_label=N/A</font></p><p style=""><font color="#666666"> Priority=4294901759 Nice=0 Account=(null) QOS=(null)</font></p><p style=""><font color="#666666"> JobState=RUNNING Reason=None Dependency=(null)</font></p><p style=""><font color="#666666"> Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0</font></p><p style=""><font color="#666666"> DerivedExitCode=0:0</font></p><p style=""><font color="#666666"> RunTime=00:29:12 TimeLimit=UNLIMITED TimeMin=N/A</font></p><p style=""><font color="#666666"> SubmitTime=2019-01-02T00:35:37 EligibleTime=2019-01-02T00:35:37</font></p><p style=""><font color="#666666"> StartTime=2019-01-02T00:35:37 EndTime=Unknown Deadline=N/A</font></p><p style=""><font color="#666666"> PreemptTime=None SuspendTime=None SecsPreSuspend=0</font></p><p style=""><font color="#666666"> Partition=all AllocNode:Sid=...:30423</font></p><p style=""><font color="#666666"> ReqNodeList=(null) ExcNodeList=(null)</font></p><p style=""><font color="#666666"> NodeList=...</font></p><p style=""><font color="#666666"> BatchHost=...</font></p><p style=""><font color="#666666"> NumNodes=1 NumCPUs=8 NumTasks=1 CPUs/Task=8 ReqB:S:C:T=0:0:*:*</font></p><p style=""><font color="#666666"> TRES=cpu=8,mem=20G,node=1</font></p><p style=""><font color="#666666"> Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*</font></p><p style=""><font color="#666666"> Nodes=... CPU_IDs=0-7 Mem=20480 <span style="color: rgb(255, 0, 0);"><b>GRES_IDX=gpu(IDX:0)</b></span></font></p><p style=""><font color="#666666"> MinCPUsNode=8 MinMemoryNode=20G MinTmpDiskNode=0</font></p><p style=""><font color="#666666"> Features=(null) DelayBoot=00:00:00</font></p><p style=""><font color="#666666"> Gres=gpu:1 Reservation=(null)</font></p><p style=""><font color="#666666"> OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)</font></p><p style=""><font color="#666666"> Command=bash</font></p><p style=""><font color="#666666"> WorkDir=/etc/slurm</font></p><p style=""><font color="#666666"> Power=</font></p><p style=""><font color="#666666"> GresEnforceBind=Yes</font></p><p style=""><font color="#666666"><br></font></p><p style=""><font color="#666666">$ scontrol show job=473 --details</font></p><p><font color="#666666">JobId=473 JobName=bash</font></p><p><font color="#666666"> UserId=root(0) GroupId=root(0) MCS_label=N/A</font></p><p><font color="#666666"> Priority=4294901758 Nice=0 Account=(null) QOS=(null)</font></p><p><font color="#666666"> JobState=RUNNING Reason=None Dependency=(null)</font></p><p><font color="#666666"> Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0</font></p><p><font color="#666666"> DerivedExitCode=0:0</font></p><p><font color="#666666"> RunTime=00:30:10 TimeLimit=UNLIMITED TimeMin=N/A</font></p><p><font color="#666666"> SubmitTime=2019-01-02T00:36:14 EligibleTime=2019-01-02T00:36:14</font></p><p><font color="#666666"> StartTime=2019-01-02T00:36:14 EndTime=Unknown Deadline=N/A</font></p><p><font color="#666666"> PreemptTime=None SuspendTime=None SecsPreSuspend=0</font></p><p><font color="#666666"> Partition=all AllocNode:Sid=...:31738</font></p><p><font color="#666666"> ReqNodeList=(null) ExcNodeList=(null)</font></p><p><font color="#666666"> NodeList=...</font></p><p><font color="#666666"> BatchHost=...</font></p><p><font color="#666666"> NumNodes=1 NumCPUs=8 NumTasks=1 CPUs/Task=8 ReqB:S:C:T=0:0:*:*</font></p><p><font color="#666666"> TRES=cpu=8,mem=20G,node=1</font></p><p><font color="#666666"> Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*</font></p><p><font color="#666666"> Nodes=... CPU_IDs=8-15 Mem=20480 <b><span style="color: rgb(255, 0, 0);">GRES_IDX=gpu(IDX:1)</span></b></font></p><p><font color="#666666"> MinCPUsNode=8 MinMemoryNode=20G MinTmpDiskNode=0</font></p><p><font color="#666666"> Features=(null) DelayBoot=00:00:00</font></p><p><font color="#666666"> Gres=gpu:1 Reservation=(null)</font></p><p><font color="#666666"> OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)</font></p><p><font color="#666666"> Command=bash</font></p><p><font color="#666666"> WorkDir=/root</font></p><p><font color="#666666"> Power=</font></p><p><font color="#666666"> GresEnforceBind=Yes</font></p><p><font color="#666666"><br></font></p><p><font color="#666666">...</font></p></td>
</tr>
</tbody>
</table><p> </p><p>But the problem is here.</p><p> </p><p>when I use Docker, Slurm GRES is not working.</p><p> </p><table class="__se_tbl" border="0" cellpadding="0" cellspacing="1" _se2_tbl_template="1" style="background-color: rgb(199, 199, 199);"><tbody>
<tr><td width="1569" style="padding: 3px 4px 2px; background-color: rgb(255, 255, 255);"><p style="color: rgb(102, 102, 102);"><span style="font-size: 13.3333px;">$ srun --gres=gpu:1 --gres-flags=enforce-binding --cpus-per-task=8 --mem=20G --pty bash</span></p><p style="color: rgb(102, 102, 102); font-size: 13.3333px;">$ nvidia-smi</p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">Wed Jan 2 01:02:00 2019 </font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">+-----------------------------------------------------------------------------+</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">| NVIDIA-SMI 410.79 Driver Version: 410.79 CUDA Version: 10.0 |</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">|-------------------------------+----------------------+----------------------+</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">|===============================+======================+======================|</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">| 0 Tesla P100-PCIE... On | <b><span style="color: rgb(255, 0, 0);">00000000:14:00.0</span></b> Off | 0 |</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">| N/A 30C P0 27W / 250W | 0MiB / 16280MiB | 0% Default |</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">+-------------------------------+----------------------+----------------------+</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666"> </font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">+-----------------------------------------------------------------------------+</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">| Processes: GPU Memory |</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">| GPU PID Type Process name Usage |</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">|=============================================================================|</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">| No running processes found |</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666">+-----------------------------------------------------------------------------+</font></p><p style="color: rgb(0, 0, 0); font-size: 13.3333px;"><font color="#666666"><br></font></p><p style=""><font color="#666666" style="color: rgb(0, 0, 0); font-size: 13.3333px;">$ </font><font color="#666666"><span style="font-size: 13.3333px;">docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">Wed Jan 2 01:10:35 2019 </span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-----------------------------------------------------------------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| NVIDIA-SMI 410.79 Driver Version: 410.79 CUDA Version: 10.0 |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">|-------------------------------+----------------------+----------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">|===============================+======================+======================|</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| 0 Tesla P100-PCIE... On |<span style="color: rgb(255, 0, 0);"><b> 00000000:14:00.0 </b><span style="color: rgb(255, 0, 0);">Off</span></span> | 0 |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| N/A 30C P0 27W / 250W | 0MiB / 16280MiB | 0% Default |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-------------------------------+----------------------+----------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| 1 Tesla P100-PCIE... On | <b><span style="color: rgb(255, 0, 0);">00000000:15:00.0</span></b> Off | 0 |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| N/A 32C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-------------------------------+----------------------+----------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| 2 Tesla P100-PCIE... On | <b><span style="color: rgb(255, 0, 0);">00000000:39:00.0</span></b> Off | 0 |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| N/A 30C P0 27W / 250W | 0MiB / 16280MiB | 0% Default |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-------------------------------+----------------------+----------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| 3 Tesla P100-PCIE... On | <b><span style="color: rgb(255, 0, 0);">00000000:3A:00.0</span></b> Off | 0 |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| N/A 28C P0 27W / 250W | 0MiB / 16280MiB | 0% Default |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-------------------------------+----------------------+----------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;"> </span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-----------------------------------------------------------------------------+</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| Processes: GPU Memory |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| GPU PID Type Process name Usage |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">|=============================================================================|</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">| No running processes found |</span></font></p><p style=""><font color="#666666"><span style="font-size: 13.3333px;">+-----------------------------------------------------------------------------+</span></font></p></td>
</tr>
</tbody>
</table><p> </p><p>Here is my configs(slurm.conf, gres.conf)</p><p> </p><table class="__se_tbl" border="0" cellpadding="0" cellspacing="1" _se2_tbl_template="1" style="background-color: rgb(199, 199, 199);"><tbody>
<tr><td width="1584" style="padding: 3px 4px 2px; background-color: rgb(255, 255, 255);"><p style=""><font color="#666666">ControlMachine=...</font></p><p style=""><font color="#666666">ControlAddr=...</font></p><p style=""><font color="#666666"><br></font></p><p style=""><font color="#666666">MailProg=/bin/mail</font></p><p style=""><font color="#666666">MpiDefault=none</font></p><p style=""><font color="#666666">ReturnToService=2</font></p><p style=""><font color="#666666">SlurmctldPidFile=/var/run/slurmctld.pid</font></p><p style=""><font color="#666666">SlurmdPidFile=/var/run/slurmd.pid</font></p><p style=""><font color="#666666">SlurmdSpoolDir=/var/spool/slurmd</font></p><p style=""><font color="#666666">SlurmdUser=root</font></p><p style=""><font color="#666666">StateSaveLocation=/var/spool</font></p><p style=""><font color="#666666">SwitchType=switch/none</font></p><p style=""><font color="#666666">ProctrackType=proctrack/cgroup</font></p><p style=""><font color="#666666">TaskPlugin=task/cgroup</font></p><p style=""><font color="#666666">AuthType=auth/munge</font></p><p style=""><font color="#666666"><br></font></p><p style=""><font color="#666666">FastSchedule=1</font></p><p style=""><font color="#666666">SchedulerType=sched/backfill</font></p><p style=""><font color="#666666">SelectType=select/cons_res</font></p><p style=""><font color="#666666">SelectTypeParameters=CR_Core_Memory</font></p><p style=""><font color="#666666"><br></font></p><p style=""><font color="#666666">GresTypes=gpu</font></p><p style=""><font color="#666666"><br></font></p><p style=""><font color="#666666">AccountingStorageType=accounting_storage/filetxt</font></p><p style=""><font color="#666666">JobCompType=jobcomp/filetxt</font></p><p style=""><font color="#666666">JobAcctGatherType=jobacct_gather/cgroup</font></p><p style=""><font color="#666666">ClusterName=...</font></p><p style=""><font color="#666666"><br></font></p><p style=""><font color="#666666">SlurmctldDebug=7</font></p><p style=""><font color="#666666">SlurmctldLogFile=/var/log/slurmctld.log</font></p><p style=""><font color="#666666">SlurmdDebug=7</font></p><p style=""><font color="#666666">SlurmdLogFile=/var/log/slurmd.log</font></p><p style=""><font color="#666666"><br></font></p><p style=""><font color="#666666"># COMPUTE NODES</font></p><p style=""><font color="#666666">NodeName=... NodeHostName=... Gres=gpu:4 CPUs=32 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=128432 State=UNKNOWN</font></p><p style=""><font color="#666666">PartitionName=all Nodes=... Default=YES MaxTime=INFINITE State=UP</font></p></td>
</tr>
</tbody>
</table><p> </p><table class="__se_tbl" border="0" cellpadding="0" cellspacing="1" _se2_tbl_template="1" style="background-color: rgb(199, 199, 199);"><tbody>
<tr><td width="1569" style="padding: 3px 4px 2px; background-color: rgb(255, 255, 255);"><p style=""><font color="#666666">Name=gpu Type=tesla File=/dev/nvidia0 CPUs=0-7</font></p><p style=""><font color="#666666">Name=gpu Type=tesla File=/dev/nvidia1 CPUs=8-15</font></p><p style=""><font color="#666666">Name=gpu Type=tesla File=/dev/nvidia2 CPUs=16-23</font></p><p style=""><font color="#666666">Name=gpu Type=tesla File=/dev/nvidia3 CPUs=24-31</font></p></td>
</tr>
</tbody>
</table><p> </p><p>what's the problem?</p><p> </p></div></body></html><table style='display:none'><tr><td><img src="https://mail.naver.com/readReceipt/notify/?img=3XRTWX%2B0Wr2spzeTpr%2B5hAbdaAt9pxt9azpoFoEdFAp4a6K%2FMrU9K6JCazuwFxigMX%2B0MogwFZl5WLl5pNiC740ThoRqWHkZWL99brkZbdIqMrC0pz9v74FTWt%3D%3D.gif" border="0"/></td></tr></table>