<html style="direction: ltr;">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style id="bidiui-paragraph-margins" type="text/css">body p { margin-bottom: 0cm; margin-top: 0pt; } </style>
</head>
<body bidimailui-charset-is-forced="true" style="direction: ltr;">
<p>Hello Cristóbal,</p>
<p><br>
</p>
<p>I think you might have a slight misunderstanding of how Slurm
works, which can cause this difference in expectation.</p>
<p><br>
</p>
<p>The MaxMemPerNode is there to allow the scheduler to plan job
placement according to resources. It does not enforce limitations
during job execution, only placement with the assumption that the
job will not use more than the resources it requested.</p>
<p><br>
</p>
<p>One option to limit the job during execution is through cgroups,
another might be using <b>JobAcctGatherParams/</b><b>OverMemoryKill
</b>but I would suspect cgroups would indeed be the better option
for your use case, and see from the slurm.conf man page:</p>
<p><br>
</p>
<p>
<blockquote type="cite">
<p>Kill processes that are being detected to use more memory
than requested by
steps every time accounting information is gathered by the
JobAcctGather plugin.
This parameter should be used with caution because a job
exceeding its memory
allocation may affect other processes and/or machine health.
</p>
<p>
<b>NOTE</b>: If available, it is recommended to limit memory
by enabling
task/cgroup as a TaskPlugin and making use of
ConstrainRAMSpace=yes in the
cgroup.conf instead of using this JobAcctGather mechanism for
memory
enforcement. Using JobAcctGather is polling based and there is
a
delay before a job is killed, which could lead to system Out
of Memory events.
</p>
<p>
<b>NOTE</b>: When using <b>OverMemoryKill</b>, if the
combined memory used by
all the processes in a step exceeds the memory limit, the
entire step will be
killed/cancelled by the JobAcctGather plugin.
This differs from the behavior when using <b>ConstrainRAMSpace</b>,
where
processes in the step will be killed, but the step will be
left active,
possibly with other processes left running.</p>
</blockquote>
<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 12/01/2023 03:47:53, Cristóbal
Navarro wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CACg_LAXfPdvj=hSqaPFcTbHxJLfxrThzOvJizGbhErcJyMiK5w@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Hi Slurm community,</div>
<div>Recently we found a small problem triggered by one of our
jobs. We have a <b>MaxMemPerNode</b>=<b>532000</b> setting in
our compute node in slurm.conf file, however we found out that
a job that started with mem=65536, and after hours of
execution it was able to grow its memory usage during
execution up to ~650GB. We expected that <b>MaxMemPerNode</b>
would stop any job exceeding the limit of 532000, did we miss
something in the slurm.conf file? We were trying to avoid
going into setting QOS for each group of users.<br>
</div>
<div>any help is welcome<br>
</div>
<div><br>
</div>
<div>Here is the node definition in the conf file</div>
<div><span style="font-family:monospace">## Nodes list<br>
## use native GPUs<br>
NodeName=nodeGPU01 SocketsPerBoard=8 CoresPerSocket=16
ThreadsPerCore=1 RealMemory=1024000 MemSpecLimit=65556
State=UNKNOWN Gres=gpu:A100:8 Feature=gpu</span></div>
<div><span style="font-family:monospace"><br>
</span></div>
<div><span style="font-family:monospace"><br>
</span></div>
<div><span style="font-family:monospace"><font
face="arial,sans-serif">And here is the full slurm.conf
file</font><br>
</span></div>
<div><span style="font-family:monospace"># node health check<br>
HealthCheckProgram=/usr/sbin/nhc<br>
HealthCheckInterval=300<br>
<br>
## Timeouts<br>
SlurmctldTimeout=600<br>
SlurmdTimeout=600<br>
<br>
GresTypes=gpu<br>
AccountingStorageTRES=gres/gpu<br>
DebugFlags=CPU_Bind,gres<br>
<br>
## We don't want a node to go back in pool without sys admin
acknowledgement<br>
ReturnToService=0<br>
<br>
## Basic scheduling<br>
SelectType=select/cons_tres<br>
SelectTypeParameters=CR_Core_Memory,CR_ONE_TASK_PER_CORE<br>
SchedulerType=sched/backfill<br>
<br>
## Accounting <br>
AccountingStorageType=accounting_storage/slurmdbd<br>
AccountingStoreJobComment=YES<br>
AccountingStorageHost=10.10.0.1<br>
AccountingStorageEnforce=limits<br>
<br>
JobAcctGatherFrequency=30<br>
JobAcctGatherType=jobacct_gather/linux<br>
<br>
TaskPlugin=task/cgroup<br>
ProctrackType=proctrack/cgroup<br>
<br>
## scripts<br>
Epilog=/etc/slurm/epilog<br>
Prolog=/etc/slurm/prolog<br>
PrologFlags=Alloc<br>
<br>
## MPI<br>
MpiDefault=pmi2<br>
<br>
## Nodes list<br>
## use native GPUs<br>
NodeName=nodeGPU01 SocketsPerBoard=8 CoresPerSocket=16
ThreadsPerCore=1 RealMemory=1024000 MemSpecLimit=65556
State=UNKNOWN Gres=gpu:A100:8 Feature=gpu<br>
<br>
## Partitions list<br>
PartitionName=gpu OverSubscribe=No MaxCPUsPerNode=64
DefMemPerNode=65556 DefCpuPerGPU=8 DefMemPerGPU=65556
MaxMemPerNode=532000 MaxTime=3-12:00:00 State=UP
Nodes=nodeGPU01 Default=YES <br>
PartitionName=cpu OverSubscribe=No MaxCPUsPerNode=64
DefMemPerNode=16384 MaxMemPerNode=420000 MaxTime=3-12:00:00
State=UP Nodes=nodeGPU01</span> <br>
</div>
<div><br clear="all">
<br>
-- <br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>Cristóbal A. Navarro</div>
</div>
</div>
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Regards,
Daniel Letai
+972 (0)505 870 456</pre>
</body>
</html>