<div dir="ltr">Hello,<div><br></div><div>We are testing Slurm19.05 on Linux RHEL7.5+ with the intent to migrate from it toTorque/Moab in the near future.</div><div><br></div><div>One of the things our users are used to is that when their jobs exceed the amount of memory they requested, the job is terminated by the scheduler.   We realize the Slurm prefers to use cgroups to contain rather than kill the jobs but initially we need to have the kill option in place to transition our users.</div><div><br></div><div>So, looking at the documentation, it appears that in 19.05, the following needs to be set to accomplish this:</div><div><br></div><div>





<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">JobAcctGatherParams <span class="gmail-Apple-converted-space">    </span>= OverMemoryKill</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;font-family:Menlo"><font color="#000000">Other possibly relevant settings we made:</font></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;font-family:Menlo">





</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">





</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">JobAcctGatherType <span class="gmail-Apple-converted-space">      </span>= jobacct_gather/linux</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">ProctrackType <span class="gmail-Apple-converted-space">          </span>= proctrack/linuxproc</span></p><div><br></div><div>We have avoided configuring any cgroup parameters for the time being.</div><div><br></div><div>Unfortunately, when we submit a job with the following:</div><div>





<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#SBATCH --nodes=1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#SBATCH --ntasks-per-node=1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#SBATCH --mem=1GBB</span></p></div><div><br></div><div>We see RSS ofthe  job steadily increase beyond the 1GB limit and it is never killed.    Interestingly enough, the proc information shows the ulimit (hard and soft) for the process set to around 1GB.</div><div><br></div><div>We have tried various settings without any success.   Can anyone point out what we are doing wrong?</div><div><br></div><div>Thanks,</div><div><br></div><div>Mike</div><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="font-size:12.8000001907349px"><div dir="ltr"><div><span style="font-family:'times new roman',serif"><b><i>J. Michael Mosley</i></b><br>University Research Computing<br>The University of North Carolina at Charlotte<br>9201 University City Blvd<br>Charlotte, NC  28223<br><u>704.687.7065 </u>    <u> j<i><a href="mailto:mmosley@uncc.edu" target="_blank">mmosley@uncc.edu</a></i></u></span></div></div></div></div></div></div></div>