<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
But that means, the docker container runs outside the cgroup of the
slurm job. Thus there exists no restriction to the container, so it
can use all resources!<br>
<br>
If e.g. one badly configured job requests one GPU, but uses all
four, the following jobs on the node will all crash, because they
cannot find a free gpu. What about restriction to cores, can the
container use all cores? What about memory cgroups, can every
container then use all the memory of the host? <br>
<br>
If this is the case, in my opinion docker cannot be used on shared
systems but only on exclusive nodes.<br>
<br>
<br>
Best<br>
Marcus<br>
<br>
<div class="moz-cite-prefix">On 01/07/2019 05:26 AM, 허웅 wrote:<br>
</div>
<blockquote type="cite"
cite="mid:4a8394d219dab0e59f904f7869a5bb18@cweb004.nm.nfra.io">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style>p{margin-top:0px;margin-bottom:0px;}</style>
<div style="font-size:10pt; font-family:Gulim, sans-serif;">
<p>I agree with Chris's opinion.</p>
<p> </p>
<p>I could find out the reason.</p>
<p> </p>
<p>As Chris said, the problem is cgroup.</p>
<p> </p>
<p>when I request a job to slurm that using 1 gres:gpu, slurm
assign the job to the node who can have enough resource.</p>
<p> </p>
<p>when slurm assign a job to the node, slurm gives resource
information to node after make a cgroup environment.</p>
<p> </p>
<p>But, the problem is that Docker uses their own cgroup config.</p>
<p> </p>
<p>That's why I could get right information through slurm-side
not Docker-side.</p>
<p> </p>
<p>Here is my workaround code for get right information in the
Docker-side.</p>
<p> </p>
<pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">
</pre>
<pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">
</pre>
<pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">scontrol show job=$SLURM_JOBID --details | grep GRES_IDX | awk -F "IDX:" '{print $2}' | awk -F ")" '{print $1}'</pre>
<pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">
</pre>
<pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">
</pre>
<pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">
</pre>
<pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">scontrol show with --details option can get GRES_IDX.</pre>
<pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">
</pre>
<pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">So, I've used this information in my application.</pre>
<pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">
</pre>
<pre style="margin-top: 0px; margin-bottom: 0px; tab-size: 4; white-space: pre-wrap; color: rgb(51, 51, 51); font-size: 14px;">Please refer to this command if someone is suffering this.</pre>
<p> </p>
<p style="font-size:10pt;font-family:sans-serif;padding:0 0 0
10pt"><span>-----Original Message-----</span><br>
<b>From:</b> "Chris Samuel"<a class="moz-txt-link-rfc2396E" href="mailto:chris@csamuel.org"><chris@csamuel.org></a> <br>
<b>To:</b> <a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a>; <br>
<b>Cc:</b> <br>
<b>Sent:</b> 2019-01-07 (월) 11:59:09<br>
<b>Subject:</b> Re: [slurm-users] gres with docker problem<br>
</p>
On 4/1/19 5:48 am, Marcin Stolarek wrote:<br>
<br>
> I think that the main reason is the lack of access to some
/dev "files" <br>
> in your docker container. For singularity nvidia plugin is
required, <br>
> maybe there is something similar for docker...<br>
<br>
That's unlikely, the problem isn't that nvidia-smi isn't working
in <br>
Docker because of a lack of device files, the problem is that
it's <br>
seeing all 4 GPUs and thus is no longer being controlled by the
device <br>
cgroup that Slurm is creating.<br>
<br>
-- <br>
Chris Samuel : <a href="http://www.csamuel.org/"
moz-do-not-send="true">http://www.csamuel.org/</a> :
Melbourne, VIC<br>
<br>
</div>
<table style="display:none">
<tbody>
<tr>
<td><img
src="https://mail.naver.com/readReceipt/notify/?img=3XFTWX%2B0Wr2spzeTpr%2B5hA3CaAKlFztZKx0vM6UmpxElpovmFzMXaAMlMxkSMou%2FtzFXp6UmKAt5W4d5W4pZMLlGWq%2Fsbrw9b4dR%2BBF0bNFgWz0q%2BHK5brFcp63RpVloWrd%3D.gif"
moz-do-not-send="true" border="0"></td>
</tr>
</tbody>
</table>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de">www.itc.rwth-aachen.de</a>
</pre>
</body>
</html>