<html><head><style>p{margin-top:0px;margin-bottom:0px;}</style></head><body><div style="font-size:10pt; font-family:Gulim, sans-serif;"><p>I agree with Chris's opinion.</p><p> </p><p>I could find out the reason.</p><p> </p><p>As Chris said, the problem is cgroup.</p><p> </p><p>when I request a job to slurm that using 1 gres:gpu, slurm assign the job to the node who can have enough resource.</p><p> </p><p>when slurm assign a job to the node, slurm gives resource information to node after make a cgroup environment.</p><p> </p><p>But, the problem is that Docker uses their own cgroup config.</p><p> </p><p>That's why I could get right information through slurm-side not Docker-side.</p><p> </p><p>Here is my workaround code for get right information in the Docker-side.</p><p> </p><pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;"><br></pre><pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;"><br></pre><pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">scontrol show job=$SLURM_JOBID --details | grep GRES_IDX | awk -F "IDX:" '{print $2}' | awk -F ")" '{print $1}'</pre><pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;"><br></pre><pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;"><br></pre><pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;"><br></pre><pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">scontrol show with --details option can get GRES_IDX.</pre><pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;"><br></pre><pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">So, I've used this information in my application.</pre><pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;"><br></pre><pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">Please refer to this command if someone is suffering this.</pre><p> </p><p style="font-size:10pt;font-family:sans-serif;padding:0 0 0 10pt"><span>-----Original Message-----</span><br><b>From:</b> "Chris Samuel"<chris@csamuel.org> <br><b>To:</b> <slurm-users@lists.schedmd.com>; <br><b>Cc:</b> <br><b>Sent:</b> 2019-01-07 (월) 11:59:09<br><b>Subject:</b> Re: [slurm-users] gres with docker problem<br> </p>On 4/1/19 5:48 am, Marcin Stolarek wrote:<br><br>> I think that the main reason is the lack of access to some /dev "files" <br>> in your docker container. For singularity nvidia plugin is required, <br>> maybe there is something similar for docker...<br><br>That's unlikely, the problem isn't that nvidia-smi isn't working in <br>Docker because of a lack of device files, the problem is that it's <br>seeing all 4 GPUs and thus is no longer being controlled by the device <br>cgroup that Slurm is creating.<br><br>-- <br> Chris Samuel : <a href="http://www.csamuel.org/">http://www.csamuel.org/</a> : Melbourne, VIC<br><br></div></body></html><table style='display:none'><tr><td><img src="https://mail.naver.com/readReceipt/notify/?img=3XFTWX%2B0Wr2spzeTpr%2B5hA3CaAKlFztZKx0vM6UmpxElpovmFzMXaAMlMxkSMou%2FtzFXp6UmKAt5W4d5W4pZMLlGWq%2Fsbrw9b4dR%2BBF0bNFgWz0q%2BHK5brFcp63RpVloWrd%3D.gif" border="0"/></td></tr></table>