<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Good Morning (at least for those on the West coast of the US)
<div class=""><br class="">
</div>
<div class="">My nodes are no longer “down”</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><b class="">eric@radoncmaster</b>:<b class="">~</b>$ sinfo</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">PARTITION AVAIL TIMELIMIT NODES STATE NODELIST</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">debug* up infinite 4 idle radonc[01-04]</span></div>
</div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
I think the NTP configuration did the trick</div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
<blockquote type="cite" class="">So one possibility there is that the clocks are out of step between the nodes. <br class="">
Usually that's configured via NTP to have a common reference source.<br class="">
</blockquote>
</div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">after ntp configuration i had to reboot the nodes, </span>restarted and enabled ntp.</div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
restarted slurmd on all execute nodes and slurmctl on headnote/master</div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
then i ran:</div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
<div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">scontrol update nodename=radonc[01-04] state=UNDRAIN </span></div>
<div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">scontrol update nodename=radonc[01-04] state=IDLE</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">All seem good for now </span></div>
</div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">Thank you everyone for your help. I learned a lot through everyone’s comments, tips and advice.</span></div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">I look forward my post-docs to run their jobs. I am certain that i will have more questions by then.</span></div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">Again, I greatly appreciate everyone’s help.</span></div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">Cheers,</span></div>
<div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">Eric</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<br class="">
</div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="text-align: -webkit-auto; orphans: 2; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="orphans: auto; widows: auto;" class=""><span style="text-align: -webkit-auto; background-color: rgba(255, 255, 255, 0);" class="">_____________________________________________________________________________________________________</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><br class="">
</span></div>
<span style="background-color: rgba(255, 255, 255, 0);" class=""><b class="">
<div style="orphans: auto; widows: auto;" class=""><b style="text-align: -webkit-auto;" class="">Eric F. Alemany</b></div>
</b>
<div style="orphans: auto; widows: auto;" class=""><i style="text-align: -webkit-auto;" class="">System Administrator for Research</i></div>
</span>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><br class="">
</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="text-align: -webkit-auto; background-color: rgba(255, 255, 255, 0);" class="">Division of Radiation & Cancer Biology</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="text-align: -webkit-auto; background-color: rgba(255, 255, 255, 0);" class="">Department of Radiation Oncology</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><br class="">
</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="text-align: -webkit-auto; background-color: rgba(255, 255, 255, 0);" class="">Stanford University School of Medicine</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="text-align: -webkit-auto; background-color: rgba(255, 255, 255, 0);" class="">Stanford, California 94305</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><br class="">
</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><font style="text-align: -webkit-auto;" class="">Tel:</font><a href="tel:1-650-498-7969" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="1" style="text-align: -webkit-auto;" class="">1-650-498-7969</a><font style="text-align: -webkit-auto;" class="">
No Texting</font></span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><font style="text-align: -webkit-auto;" class="">Fax:</font><a href="tel:1-650-723-7382" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="2" style="text-align: -webkit-auto;" class="">1-650-723-7382</a></span></div>
<div style="orphans: auto; widows: auto;" class=""><br class="">
</div>
</div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<font style="background-color: rgba(255, 255, 255, 0);" class=""></font></div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On May 7, 2018, at 5:30 PM, Chris Samuel <<a href="mailto:chris@csamuel.org" class="">chris@csamuel.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">On Tuesday, 8 May 2018 9:40:53 AM AEST Eric F. Alemany wrote:<br class="">
<br class="">
<blockquote type="cite" class="">I followed the link as well as the instruction on “Securing the<br class="">
installation” and “Testing the installation”<br class="">
</blockquote>
<br class="">
Great.<br class="">
<br class="">
<blockquote type="cite" class="">The only thing that i am not able to do is: Check if a credential can be<br class="">
remotely decoded<br class="">
</blockquote>
<br class="">
So one possibility there is that the clocks are out of step between the nodes. <br class="">
Usually that's configured via NTP to have a common reference source.<br class="">
<br class="">
That's pretty standard as if you're running an HPC system with a distributed <br class="">
filesystem like GPFS or Lustre then you need the clocks in lockstep for it to <br class="">
function properly.<br class="">
<br class="">
Good luck!<br class="">
Chris<br class="">
-- <br class="">
Chris Samuel : <a href="http://www.csamuel.org/" class="">http://www.csamuel.org/</a> : Melbourne, VIC<br class="">
<br class="">
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>