<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type"/>
<style type="text/css"> P {margin-top: 0;margin-bottom: 0;}
<!-- p {margin-top: 0; margin-bottom: 0;}
-->
</style></head><body style="">
<div>
Hi Sean,
</div>
<div>
After changing those values yesterday, it worked today after the data got rebuild, i can see the sreport working now.
</div>
<div>
Thank you all so much for your help in solving this issue.
</div>
<div>
</div>
<div>
<br/>On October 20, 2021 at 6:38 PM mshubham <mshubham@cdac.in> wrote:
</div>
<div style="position: relative;">
<blockquote style="margin-left: 0px; padding-left: 10px; border-left: solid 1px blue;" type="cite">
<div>
Hi Sean,
</div>
<div>
Yes, i have changed those values to that time. After restarting slurmdbd, timestamp changed again to today's date but sreport still not showing data for a year.
</div>
<div>
<br/>On October 19, 2021 at 10:55 AM Sean Crosby <scrosby@unimelb.edu.au> wrote:
</div>
<div style="position: relative;">
<blockquote style="margin-left: 0px; padding-left: 10px; border-left: solid 1px blue;" type="cite">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: #000000;">
Did you change those values to a timestamp before May 2020 like I suggested? If you do that, then Slurm will regenerate the data needed for sreport to work properly.
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: #000000;">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: #000000;">
Sean
</div>
<div id="appendonsend">
</div>
<hr/>
<div id="divRplyFwdMsg" dir="ltr">
<span style="font-size: 11pt; color: #000000; font-family: Calibri, sans-serif;"><strong>From:</strong> mshubham <mshubham@cdac.in><br/> <strong>Sent:</strong> Tuesday, 19 October 2021 16:20<br/> <strong>To:</strong> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com>; Sean Crosby <scrosby@unimelb.edu.au><br/> <strong>Subject:</strong> Re: [EXT] Re: [slurm-users] Missing data in sreport for a time period in slurm</span>
<div>
</div>
</div>
<div>
<div class="x_mc-ip-hide">
<div style="color: #000000; font-size: 12px; text-align: left; font-family: Helvetica,Arial,sans-serif;">
<table style="width: 100%; float: left;" cellpadding="0" border="0" cellspacing="0"><tbody><tr><td style="color: red;"><strong>External email: </strong>Please exercise caution</td></tr></tbody></table>
</div>
<hr/>
</div>
<div>
Dear All,
</div>
<div>
By checking the value of last ran table, hourly rollup shows today's date.
</div>
<div>
+---------------+--------------+----------------+-------------
<br/> | hourly_rollup | daily_rollup | monthly_rollup |
<br/> +---------------+--------------+----------------+-------------
<br/> | 1634617800 | 1634581800 | 1634581785 |
<br/> +---------------+--------------+----------------+-------------
</div>
<div>
As we can see the data from last 1 month in sreport but not the data from may 2020 to sept 21, but the same can be visible through sacct.
</div>
<div>
</div>
<div>
<br/> On October 19, 2021 at 12:27 AM Sean Crosby <scrosby@unimelb.edu.au> wrote:
</div>
<div>
<blockquote style="margin-left: 0px; padding-left: 10px; border-left: solid 1px blue;" type="cite">
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
sreport keeps a track of when it has done the last rollup calculations in the database.
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
Open MySQL for your Slurm accounting database, do
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
select * from slurm_acct_db.clustername_last_ran_table;
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
where slurm_acct_db is your accounting database name (slurm_acct_db is default), and clustername is the name of your cluster in Slurm.
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
The numbers are UNIX timestamps. e.g for mine
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
</div>
<blockquote style="margin-top: 0; margin-bottom: 0;">
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
select * from thespian_slurm_acct_db.thespian_last_ran_table;
<div>
+---------------+--------------+----------------+
</div>
<div>
| hourly_rollup | daily_rollup | monthly_rollup |
</div>
<div>
+---------------+--------------+----------------+
</div>
<div>
| 1634580000 | 1634562000 | 1633420706 |
</div>
<div>
+---------------+--------------+----------------+
</div>
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
1 row in set (0.00 sec)
</div>
</blockquote>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
1634580000 is Mon Oct 18 2021 18:00:00 GMT+0000
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
If you set those values to something before 26 May 2020, you can then either wait for the rollups to happen again, or you can trigger them to start by restarting slurmdbd. Once they start, you can monitor their progress by running that SQL query again. Once they have caught up again, sreport should be back to normal.
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
</div>
<div style="font-family: Calibri,Arial,Helvetica,sans-serif; font-size: 12pt; color: #000000;">
Sean
</div>
<div id="x_appendonsend">
</div>
<hr/>
<div id="x_divRplyFwdMsg" dir="ltr">
<span style="font-size: 11pt; color: #000000; font-family: Calibri,sans-serif;"><strong>From:</strong> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of mshubham <mshubham@cdac.in><br/> <strong>Sent:</strong> Tuesday, 19 October 2021 00:22<br/> <strong>To:</strong> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br/> <strong>Subject:</strong> [EXT] Re: [slurm-users] Missing data in sreport for a time period in slurm</span>
<div>
</div>
</div>
<div>
<div class="x_x_mc-ip-hide">
<div style="color: #000000; font-size: 12px; text-align: left; font-family: Helvetica,Arial,sans-serif;">
<table style="width: 100%; float: left;" cellpadding="0" border="0" cellspacing="0"><tbody><tr><td style="color: red;"><strong>External email: </strong>Please exercise caution</td></tr></tbody></table>
</div>
<hr/>
</div>
<div>
Dear all,
</div>
<div>
As i recall, there were some runaway jobs one month ago, and i used this command to fix it, and from that date only(from 15 Sept) , previous data from 26 May 2020 to Sept 14 2021
</div>
<div>
has been gone in sreport.
</div>
<div>
I hope there is no co-relation between using this command and data missing in sreport.
</div>
<div>
</div>
<div>
I have backup from that day of mysql, if you can suggest some way to integrate that data into slurm table in mysql.
</div>
<div>
Or is there any other way so that i can reinitialize the slurm utillization tables from initial.
</div>
<div>
</div>
<div>
<br/> On October 18, 2021 at 5:38 PM mshubham <mshubham@cdac.in> wrote:
</div>
<div>
<blockquote style="margin-left: 0px; padding-left: 10px; border-left: solid 1px blue;" type="cite">
<div>
<div class="x_x_DTXlsb">
Dear all,
</div>
<div class="x_x_DTXlsb">
"sacctmgr show runawayjobs" doesnot show any problematic jobs.
</div>
<div class="x_x_DTXlsb">
</div>
<div class="x_x_DTXlsb">
On 10/18/21 12:41 PM, mshubham wrote:
<br/> > Dear all,
<br/> > I am facing a issue in slurm(v19.05.1) in which data from 26 May 2020 to
<br/> > Sept 14 2021 is missing in sreport but the same data is present through
<br/> > sacct command, It which was working fine few days ago.
<br/> > Right now, we have to get data utilization from sacct command for each user.
<br/> > It would be really helpful if this issue get resolved.
</div>
</div>
<div>
</div>
<div>
Does "sacctmgr show runawayjobs" reveal any problematic jobs?
</div>
<div>
</div>
<div>
Thanks and Regards,
<br/> Shubham
</div>
</blockquote>
<br/>
</div>
<div id="x_x_ox-signature">
Thanks and Regards,
<br/> Shubham Mehta
<br/> HPC Technology
<br/> CDAC Pune
</div>
<br/> ------------------------------------------------------------------------------------------------------------
<br/> [ C-DAC is on Social-Media too. Kindly follow us at:
<br/> Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
<br/>
<br/> This e-mail is for the sole use of the intended recipient(s) and may
<br/> contain confidential and privileged information. If you are not the
<br/> intended recipient, please contact the sender by reply e-mail and destroy
<br/> all copies and the original message. Any unauthorized review, use,
<br/> disclosure, dissemination, forwarding, printing or copying of this email
<br/> is strictly prohibited and appropriate legal action will be taken.
<br/> ------------------------------------------------------------------------------------------------------------
</div>
</blockquote>
<br/>
</div>
<div id="x_ox-signature">
Thanks and Regards,
<br/> Shubham Mehta
<br/> HPC Technology
<br/> CDAC Pune
</div>
<br/> ------------------------------------------------------------------------------------------------------------
<br/> [ C-DAC is on Social-Media too. Kindly follow us at:
<br/> Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
<br/>
<br/> This e-mail is for the sole use of the intended recipient(s) and may
<br/> contain confidential and privileged information. If you are not the
<br/> intended recipient, please contact the sender by reply e-mail and destroy
<br/> all copies and the original message. Any unauthorized review, use,
<br/> disclosure, dissemination, forwarding, printing or copying of this email
<br/> is strictly prohibited and appropriate legal action will be taken.
<br/> ------------------------------------------------------------------------------------------------------------
</div>
</blockquote>
<br/>
</div>
<div>
Thanks and Regards,
<br/>Shubham Mehta
<br/>HPC Technology
<br/>CDAC Pune
</div>
</blockquote>
<br/>
</div>
<div id="ox-signature">
Thanks and Regards,
<br/>Shubham Mehta
<br/>HPC Technology
<br/>CDAC Pune
</div>
<br />
<br /> For assimilation and dissemination of knowledge, visit <a href="https://cakes.cdac.in">cakes.cdac.in</a>
<br />
<br />------------------------------------------------------------------------------------------------------------
<br />[ C-DAC is on Social-Media too. Kindly follow us at:
<br />Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
<br />
<br />This e-mail is for the sole use of the intended recipient(s) and may
<br />contain confidential and privileged information. If you are not the
<br />intended recipient, please contact the sender by reply e-mail and destroy
<br />all copies and the original message. Any unauthorized review, use,
<br />disclosure, dissemination, forwarding, printing or copying of this email
<br />is strictly prohibited and appropriate legal action will be taken.
<br />------------------------------------------------------------------------------------------------------------
</body></html>