<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Exactly. The easiest way is just to underreport the amount of
memory in slurm. That way slurm will take care of it natively.
We do this here as well even though we have disks in order to make
sure the OS has memory left to run.<br>
</p>
<p>-Paul Edmon-<br>
</p>
<div class="moz-cite-prefix">On 3/14/19 8:36 AM, Doug Meyer wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJvTnXKAioCqwHEMqnZuSSBS4VfjtfPJY0qQLEHapAMs4qGgrw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>We also run diskless. In the slurm.conf we round down on
memory so slurm does not have the total budget to work with
and use a default memory per job value reflecting declared
memory/# of threads per node. If users don't declarememory
limit we are fine. If they declare more we are fine too.
Mostly. We had to turn off memory enforcement as the job
memory usage is very uneven during runtime but with the above
we have seldom had problems.</div>
<div><br>
</div>
<div>Doug<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Mar 14, 2019 at 3:57
AM david baker <<a href="mailto:djbaker12@gmail.com"
moz-do-not-send="true">djbaker12@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Hello Paul,
<div><br>
</div>
<div>Thank you for your advice. That all makes sense.
We're running diskless compute nodes and so the usable
memory is less than the total memory. So I have added a
memory check to my job_submit.lua -- see below. I think
that all makes sense.</div>
<div><br>
</div>
<div>Best regards,</div>
<div>David</div>
<div><br>
</div>
<div>
<div>-- Check memory/node is valid</div>
<div> if job_desc.min_mem_per_cpu ==
9223372036854775808 then</div>
<div> job_desc.min_mem_per_cpu = 4300</div>
<div> end</div>
<div><br>
</div>
<div> memory = job_desc.min_mem_per_cpu *
job_desc.min_cpus</div>
<div><br>
</div>
<div> if memory > 172000 then</div>
<div> slurm.log_user("You cannot request more than
172000 Mbytes per node")</div>
<div> slurm.log_user("memory is: %u",memory)</div>
<div> return slurm.ERROR</div>
<div> end</div>
</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Mar 12, 2019 at
4:48 PM Paul Edmon <<a
href="mailto:pedmon@cfa.harvard.edu" target="_blank"
moz-do-not-send="true">pedmon@cfa.harvard.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Slurm should automatically block or reject jobs that
can't run on that partition in terms of memory usage
for a single node. So you shouldn't need to do
anything. If you need something less than the max
memory per node then you will need to enforce some
limits. We do this via a jobsubmit lua script. That
would be my recommended method.<br>
</p>
<p><br>
</p>
<p>-Paul Edmon-</p>
<p><br>
</p>
<div
class="gmail-m_-7672031880103951612gmail-m_-7371107165191797214moz-cite-prefix">On
3/12/19 12:31 PM, David Baker wrote:<br>
</div>
<blockquote type="cite">
<div
id="gmail-m_-7672031880103951612gmail-m_-7371107165191797214divtagdefaultwrapper"
style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif"
dir="ltr">
<p style="margin-top:0px;margin-bottom:0px"><span>Hello,</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span><br>
</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span>I
have set up a serial queue to run small jobs in
the cluster. Actually, I route jobs to this
queue using the job_submit.lua script. Any 1
node job using up to 20 cpus is routed to this
queue, unless a user submits their job with an
exclusive flag. </span></p>
<p style="margin-top:0px;margin-bottom:0px"><span><br>
</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span>The
partition is shared and so I defined memory to
be a resource. I've set default memory/cpu to be
4300 Mbytes. There are 40 cpus installed in the
nodes and the usable memory is circa 17200
Mbytes -- hence my default mem/cpu.<span></span></span></p>
<p style="margin-top:0px;margin-bottom:0px"><span><br>
</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span>The
compute nodes are defined with <span>RealMemory=190000,
by the way.</span></span></p>
<p style="margin-top:0px;margin-bottom:0px"><span><span><br>
</span></span></p>
<p style="margin-top:0px;margin-bottom:0px"><span><span>I
am curious to understand how I can impose a
memory limit on the jobs that are submitted to
this partition. It doesn't make any sense to
request more than the total usable memory on
the nodes. So could anyone please advise me
how to ensure that users cannot request more
than the usable memory on the nodes.</span></span></p>
<p style="margin-top:0px;margin-bottom:0px"><span><span><br>
</span></span></p>
<p style="margin-top:0px;margin-bottom:0px"><span><span>Best
regards,</span></span></p>
<p style="margin-top:0px;margin-bottom:0px"><span><span>David</span></span></p>
<p style="margin-top:0px;margin-bottom:0px"><span><br>
</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span>PartitionName=serial
nodes=red[460-464] Shared=Yes MaxCPUsPerNode=40
DefaultTime=02:00:00 MaxTime=60:00:00 QOS=serial
SelectTypeParameters=CR_Core_Memory <b>DefMemPerCPU=4300</b> State=UP
AllowGroups=jfAccessToIridis5
PriorityJobFactor=10 PreemptMode=off</span><br>
</p>
<p style="margin-top:0px;margin-bottom:0px"><span><br>
</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span><br>
</span></p>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</body>
</html>