<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type"/>
</head><body style="">
<div>
Dear All,
</div>
<div>
I am facing an issue in SLURM(20.11.8), in which sreport cluster utilization is 100%, and when I run sreport cluster userutilizationbyaccount, Some user utilisation is greater than 100%, three users including root showing utilisation over 250%, making overall utilisation 500% (though user has not submitted any job in past one week)
</div>
<div>
</div>
<div>
It was showing some runaway jobs, but we cleared it, then again, it was showing same runaway jobs, and we cleared it again. (both manually/through command)
</div>
<div>
</div>
<div>
Before that, we had encountered an issue in the past in which, in our
<span style="font-size: 9pt;"> cluster with primary and backup slurm controller, we kept a common mount point for the "StateSaveLocation" /var/share/slurm/ctld. Then we observed a strange </span>behaviour
<span style="font-size: 9pt;"> that "</span>
<span style="font-size: 9pt;">If the mount point is present and the service is restarted on the primary controller then it replaces all the statesavelocation files."</span>
<p>This resulted in cancellation of all the jobs (running, pending state), reservations and assigns the JobID from 1 for newly submitted jobs. <span style="font-size: 9pt;">If the SateSaveLocation is kept on local file system instead of shared mount point then everything works fine even after restarting the slurmctld service.</span></p>
<p>After that issue, utilisation is higher than expected, though it has not impacted any real job utilisation.</p>
<p>Also, we have removed those user's account in SLURM, yet it is still showing their utilisation</p>
<p>Please help in resolving this issue.</p>
</div>
<div>
</div>
<div id="ox-signature">
Thanks and Regards,
<br/>Shubham Mehta
<br/>HPC Technology
<br/>CDAC Pune
</div>
<br />------------------------------------------------------------------------------------------------------------
<br />[ C-DAC is on Social-Media too. Kindly follow us at:
<br />Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
<br />
<br />This e-mail is for the sole use of the intended recipient(s) and may
<br />contain confidential and privileged information. If you are not the
<br />intended recipient, please contact the sender by reply e-mail and destroy
<br />all copies and the original message. Any unauthorized review, use,
<br />disclosure, dissemination, forwarding, printing or copying of this email
<br />is strictly prohibited and appropriate legal action will be taken.
<br />------------------------------------------------------------------------------------------------------------
</body></html>