<div dir="ltr">One other thing I noticed is that the contents of the *_job_table has entries in tres_alloc and tres_req that seem to match types in the tres_table, but there are no mem entries.<div>For example, tres_table=</div><div>+---------------+---------+------+----------------+------+<br>| creation_time | deleted | id | type | name |<br>+---------------+---------+------+----------------+------+<br>| 1559250721 | 0 | 1 | cpu | |<br>| 1559250721 | 0 | 2 | mem | |<br>| 1559250721 | 0 | 3 | energy | |<br>| 1559250721 | 0 | 4 | node | |<br>| 1559250721 | 0 | 5 | billing | |<br>| 1559250721 | 0 | 6 | fs | disk |<br>| 1559250721 | 0 | 7 | vmem | |<br>| 1559250721 | 0 | 8 | pages | |<br>| 1559250721 | 1 | 1000 | dynamic_offset | |<br>+---------------+---------+------+----------------+------+<br></div><div><br></div><div>But none of the jobs poplate a value for 2 (mem): </div><div>+--------+-------------+------------------------------------+<br>| id_job | tres_req | tres_alloc |<br>+--------+-------------+------------------------------------+<br>| 19779 | 1=1,4=1,5=1 | 1=4,4=1,5=4 |<br>| 19780 | 1=1,4=1,5=1 | 1=4,4=1,5=4 |<br>| 19781 | 1=1,4=1,5=1 | 1=4,3=18446744073709551614,4=1,5=4 |<br>| 19782 | 1=1,4=1,5=1 | 1=16,4=1,5=16 |<br>| 19783 | 1=1,4=1,5=1 | 1=16,4=1,5=16 |<br>| 19784 | 1=1,4=1,5=1 | 1=4,3=18446744073709551614,4=1,5=4 |<br>| 19785 | 1=1,4=1,5=1 | 1=16,4=1,5=16 |<br>| 19786 | 1=1,4=1,5=1 | 1=16,4=1,5=16 |<br>| 19787 | 1=1,4=1,5=1 | 1=4,3=18446744073709551614,4=1,5=4 |<br>| 19788 | 1=1,4=1,5=1 | 1=4,3=18446744073709551614,4=1,5=4 |<br>+--------+-------------+------------------------------------+<br></div><div><br></div><div>Brian Andrus</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 16, 2019 at 2:58 PM Brian Andrus <<a href="mailto:toomuchit@gmail.com">toomuchit@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I have<br>JobAcctGatherType = jobacct_gather/linux<br><div><br></div><div>Brian</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 16, 2019 at 12:40 PM Antony Cleave <<a href="mailto:antony.cleave@gmail.com" target="_blank">antony.cleave@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto">Just a quick thought.<br></div><div dir="auto"><br></div><div dir="auto">What is your slurm.conf setting for this?</div><div dir="auto"><br></div><div dir="auto"><b style="margin:0px;padding:0px;border:0px;font-size:18px;line-height:inherit;font-family:"source sans pro",helvetica,arial,sans-serif;vertical-align:baseline;color:rgb(70,84,92);background-color:rgb(255,255,255)">JobAcctGatherType</b><span style="color:rgb(70,84,92);font-family:"source sans pro",helvetica,arial,sans-serif;font-size:18px;background-color:rgb(255,255,255)"> is operating system dependent and controls what mechanism is used to collect accounting information. Supported values are </span><i style="margin:0px;padding:0px;border:0px;font-size:18px;line-height:inherit;font-family:"source sans pro",helvetica,arial,sans-serif;vertical-align:baseline;color:rgb(70,84,92);background-color:rgb(255,255,255)">jobacct_gather/linux</i><span style="color:rgb(70,84,92);font-family:"source sans pro",helvetica,arial,sans-serif;font-size:18px;background-color:rgb(255,255,255)"> (recommended), </span><i style="margin:0px;padding:0px;border:0px;font-size:18px;line-height:inherit;font-family:"source sans pro",helvetica,arial,sans-serif;vertical-align:baseline;color:rgb(70,84,92);background-color:rgb(255,255,255)">jobacct_gather/cgroup</i><span style="color:rgb(70,84,92);font-family:"source sans pro",helvetica,arial,sans-serif;font-size:18px;background-color:rgb(255,255,255)"> and </span><i style="margin:0px;padding:0px;border:0px;font-size:18px;line-height:inherit;font-family:"source sans pro",helvetica,arial,sans-serif;vertical-align:baseline;color:rgb(70,84,92);background-color:rgb(255,255,255)">jobacct_gather/none</i><span style="color:rgb(70,84,92);font-family:"source sans pro",helvetica,arial,sans-serif;font-size:18px;background-color:rgb(255,255,255)"> (no information collected).</span><br></div><div dir="auto"><span style="color:rgb(70,84,92);font-family:"source sans pro",helvetica,arial,sans-serif;font-size:18px;background-color:rgb(255,255,255)"><br></span></div><div dir="auto"><span style="color:rgb(70,84,92);font-family:"source sans pro",helvetica,arial,sans-serif;font-size:18px;background-color:rgb(255,255,255)">Antony </span></div><div dir="auto"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 16 Sep 2019, 14:07 Brian Andrus, <<a href="mailto:toomuchit@gmail.com" rel="noreferrer" target="_blank">toomuchit@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Yep, the maxrss field is always blank.<br>
<br>
I just checked on a different cluster and have the same result. Jobs <br>
that completed last week even have nothing in that field.<br>
<br>
So how can I troubleshoot this? Is there a way to log the sql queries <br>
made by slurmdbd?<br>
<br>
Brian<br>
<br>
On 9/15/2019 10:29 PM, Christopher Samuel wrote:<br>
> On 9/15/19 4:17 PM, Brian Andrus wrote:<br>
><br>
>> Are steps required to capture Max RSS?<br>
><br>
> No, you should see a MaxRSS reported for the batch step, for instance:<br>
><br>
> $ sacct -j $JOBID -o jobid,jobname,maxrss<br>
><br>
> All the best,<br>
> Chris<br>
<br>
</blockquote></div>
</blockquote></div>
</blockquote></div>