<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>After doing some Googling</div><div><a href="https://jvns.ca/blog/2017/02/17/mystery-swap/">https://jvns.ca/blog/2017/02/17/mystery-swap/</a>  <span style="font:inherit;text-align:left;color:rgb(0,0,0);text-transform:none;text-indent:0px;letter-spacing:normal;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;font-size-adjust:none;font-stretch:inherit;float:none;background-color:rgb(255,255,255)">Swapping is weird and
confusing  (Amen to that!)</span></div><div><a href="https://jvns.ca/blog/2016/12/03/how-much-memory-is-my-process-using-/">https://jvns.ca/blog/2016/12/03/how-much-memory-is-my-process-using-/</a>  (interesting article)<br></div><div><br></div><div>From the Docker documentation, below.</div><div>Bill - this is what you are seeing. Twice as much swap as real memory usage.  So that seems to be a kernel default behaviour.</div><div>As you say the slurm logs say swap is set to 0 though.</div><div><br></div><div><a href="https://docs.docker.com/config/containers/resource_constraints/#--memory-swap-details">https://docs.docker.com/config/containers/resource_constraints/#--memory-swap-details</a><br></div><div><br></div><div><span style="font:400 14px/24px "Open Sans",sans-serif;text-align:left;color:rgb(51,68,76);text-transform:none;text-indent:0px;letter-spacing:normal;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;font-size-adjust:none;font-stretch:normal;float:none;background-color:rgb(255,255,255)">f </span><code class="gmail-highlighter-rouge" style="margin:15px 0px;padding:3px 7px;border-radius:2px;text-align:left;color:rgb(12,81,118);text-transform:none;text-indent:0px;letter-spacing:normal;font-size:90%;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;white-space:nowrap;overflow-x:scroll;box-sizing:border-box;background-color:rgba(12,81,118,0.1)">--memory-swap</code><span style="font:400 14px/24px "Open Sans",sans-serif;text-align:left;color:rgb(51,68,76);text-transform:none;text-indent:0px;letter-spacing:normal;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;font-size-adjust:none;font-stretch:normal;float:none;background-color:rgb(255,255,255)"> is unset, and </span><code class="gmail-highlighter-rouge" style="margin:15px 0px;padding:3px 7px;border-radius:2px;text-align:left;color:rgb(12,81,118);text-transform:none;text-indent:0px;letter-spacing:normal;font-size:90%;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;white-space:nowrap;overflow-x:scroll;box-sizing:border-box;background-color:rgba(12,81,118,0.1)">--memory</code><span style="font:400 14px/24px "Open Sans",sans-serif;text-align:left;color:rgb(51,68,76);text-transform:none;text-indent:0px;letter-spacing:normal;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;font-size-adjust:none;font-stretch:normal;float:none;background-color:rgb(255,255,255)"> is set, the container can use
twice as much swap as the </span><code class="gmail-highlighter-rouge" style="margin:15px 0px;padding:3px 7px;border-radius:2px;text-align:left;color:rgb(12,81,118);text-transform:none;text-indent:0px;letter-spacing:normal;font-size:90%;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;white-space:nowrap;overflow-x:scroll;box-sizing:border-box;background-color:rgba(12,81,118,0.1)">--memory</code><span style="font:400 14px/24px "Open Sans",sans-serif;text-align:left;color:rgb(51,68,76);text-transform:none;text-indent:0px;letter-spacing:normal;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;font-size-adjust:none;font-stretch:normal;float:none;background-color:rgb(255,255,255)"> setting, if the host container has swap
memory configured. For instance, if </span><code class="gmail-highlighter-rouge" style="margin:15px 0px;padding:3px 7px;border-radius:2px;text-align:left;color:rgb(12,81,118);text-transform:none;text-indent:0px;letter-spacing:normal;font-size:90%;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;white-space:nowrap;overflow-x:scroll;box-sizing:border-box;background-color:rgba(12,81,118,0.1)">--memory="300m"</code><span style="font:400 14px/24px "Open Sans",sans-serif;text-align:left;color:rgb(51,68,76);text-transform:none;text-indent:0px;letter-spacing:normal;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;font-size-adjust:none;font-stretch:normal;float:none;background-color:rgb(255,255,255)"> and </span><code class="gmail-highlighter-rouge" style="margin:15px 0px;padding:3px 7px;border-radius:2px;text-align:left;color:rgb(12,81,118);text-transform:none;text-indent:0px;letter-spacing:normal;font-size:90%;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;white-space:nowrap;overflow-x:scroll;box-sizing:border-box;background-color:rgba(12,81,118,0.1)">--memory-swap</code><span style="font:400 14px/24px "Open Sans",sans-serif;text-align:left;color:rgb(51,68,76);text-transform:none;text-indent:0px;letter-spacing:normal;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;font-size-adjust:none;font-stretch:normal;float:none;background-color:rgb(255,255,255)"> is
not set, the container can use 300m of memory and 600m of swap.</span><b></b><i></i><u></u><strike></strike><br></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Sat, 20 Oct 2018 at 03:39, Chris Samuel <<a href="mailto:chris@csamuel.org">chris@csamuel.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Tuesday, 16 October 2018 2:47:34 PM AEDT Bill Broadley wrote:<br>
<br>
> AllowedSwapSpace=0<br>
> <br>
> So I expect jobs to not use swap.  Turns out if I run a 3GB ram process with<br>
> sbatch --mem=1000 I just get a process that uses 1GB ram and 2GB of swap.<br>
<br>
That's intended.  The manual page says:<br>
<br>
       AllowedSwapSpace=<number><br>
              Constrain  the  job cgroup swap space to this percentage of the<br>
              allocated memory.  The default value is 0, which means that<br>
              RAM+Swap will  be  limited  to  AllowedRAMSpace. <br>
<br>
You probably want this as well:<br>
<br>
       MemorySwappiness=<number><br>
              Configure the kernel's priority for swapping out anonymous pages<br>
              (such as program data) verses file cache pages for the job<br>
              cgroup.  Valid  values are  between  0 and 100, inclusive. A<br>
              value of 0 prevents the kernel from swapping out program data.<br>
<br>
cheers!<br>
Chris<br>
-- <br>
 Chris Samuel  :  <a href="http://www.csamuel.org/" rel="noreferrer" target="_blank">http://www.csamuel.org/</a>  :  Melbourne, VIC<br>
<br>
<br>
<br>
<br>
</blockquote></div>