<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Bill,<o:p></o:p></p>
<p class="MsoNormal">Your best bet is probably /var/log/slurmctld on the server that is acting as active controller.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
<p class="MsoNormal">Roberto P. Monti<span style="font-size:12.0pt"><o:p></o:p></span></p>
<p class="MsoNormal">DevOps Engineer I<o:p></o:p></p>
<p class="MsoNormal">roberto.monti@jax.org<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The Jackson Laboratory<o:p></o:p></p>
<p class="MsoNormal">United States | China | Japan<o:p></o:p></p>
<p class="MsoNormal">www.jax.org<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Bill Benedetto<br>
<b>Sent:</b> Wednesday, July 5, 2023 1:21 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com<br>
<b>Subject:</b> [EXTERNAL][slurm-users] Slurm node history / log ?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Good day.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Is there some command that I can use in Slurm to see a node’s history?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Not the job history, but the state history.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Something like:<o:p></o:p></p>
<p class="MsoNormal">Jul 5 13:11:01 node01 taken offline by slurmctld because node01 not responding<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And/Or:<o:p></o:p></p>
<p class="MsoNormal">Jul 5 13:11:01 node01 taken offline by USER1 state=DRAIN reason=”System acting up, going to reboot”<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And/Or:<o:p></o:p></p>
<p class="MsoNormal">Jul 5 13:11:01 node01 online by USER1<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My goal/idea is to see if a node has been having problems according to Slurm itself.<o:p></o:p></p>
<p class="MsoNormal">Or if someone DOWNed a node for some reason.<o:p></o:p></p>
<p class="MsoNormal">Or to see if a node was down and just returned to service recently.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Does anything like that already exist in Slurm?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">- Bill<o:p></o:p></p>
<p class="MsoNormal">+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+<o:p></o:p></p>
<p class="MsoNormal">Bill Benedetto <a href="mailto:bbenedetto@goodyear.com">
bbenedetto@goodyear.com</a> The Goodyear Tire & Rubber Co.<o:p></o:p></p>
<p class="MsoNormal">I don't speak for Goodyear and they don't speak for me. We're both happy.<o:p></o:p></p>
</div>
---<br><br>The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.</body>
</html>