<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1434980524;
mso-list-template-ids:-1642315234;}
@list l0:level1
{mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level4
{mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level7
{mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body lang="en-CH" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Thanks for the quick reply.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="mso-fareast-language:EN-US">check if munge is working properly<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">root@ecpsinf01:~# munge -n | ssh ecpsc10 unmunge<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Warning: the ECDSA host key for 'ecpsc10' differs from the key for the IP address '128.178.242.136'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Offending key for IP in /root/.ssh/known_hosts:5<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Matching host key in /root/.ssh/known_hosts:28<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Are you sure you want to continue connecting (yes/no)? yes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">STATUS: Success (0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">ENCODE_HOST: ecpsc10 (127.0.1.1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">ENCODE_TIME: 2021-11-16 16:57:56 +0100 (1637078276)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">DECODE_TIME: 2021-11-16 16:58:10 +0100 (1637078290)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">TTL: 300<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">CIPHER: aes128 (4)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">MAC: sha256 (5)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">ZIP: none (0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">UID: root (0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">GID: root (0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">LENGTH: 0<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="mso-fareast-language:EN-US">Check if SE linux is enforced<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="mso-fareast-language:EN-US">controller node<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">root@ecpsinf01:~# getenforce<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">-bash: getenforce: command not found<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">root@ecpsinf01:~# sestatus<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">-bash: sestatus: command not found<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="mso-fareast-language:EN-US">compute node<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">root@ecpsc10:~# getenforce<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Command 'getenforce' not found, but can be installed with:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">apt install selinux-utils<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">root@ecpsc10:~# sestatus<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Command 'sestatus' not found, but can be installed with:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">apt install policycoreutils<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="mso-fareast-language:EN-US">Check slurm log file<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.646] debug: Log file re-opened<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.666] Message aggregation disabled<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.666] topology NONE plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.666] route default plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.667] CPU frequency setting not configured for this node<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.667] debug: Resource spec: No specialized cores configured by default on this node<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.667] debug: Resource spec: Reserved system memory limit not configured for this node<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.667] debug: Reading cgroup.conf file /etc/slurm/cgroup.conf<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.667] debug: Ignoring obsolete CgroupReleaseAgentDir option.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.669] debug: Reading cgroup.conf file /etc/slurm/cgroup.conf<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.670] debug: Ignoring obsolete CgroupReleaseAgentDir option.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.670] debug: task/cgroup: now constraining jobs allocated cores<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.670] debug: task/cgroup/memory: total:112428M allowed:100%(enforced), swap:0%(permissive), max:100%(112428M) max+swap:100%(224856M) min:30M kmem:100%(112428M enforced) min:30M
swappiness:0(unset)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.670] debug: task/cgroup: now constraining jobs allocated memory<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.670] debug: task/cgroup: now constraining jobs allocated devices<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.670] debug: task/cgroup: loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.671] debug: Munge authentication plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.671] debug: spank: opening plugin stack /etc/slurm/plugstack.conf<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.671] Munge cryptographic signature plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.673] slurmd version 17.11.12 started<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.673] debug: Job accounting gather cgroup plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.674] debug: job_container none plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.674] debug: switch NONE plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.674] slurmd started on Tue, 16 Nov 2021 16:19:54 +0100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.675] CPUs=16 Boards=1 Sockets=2 Cores=8 Threads=1 Memory=112428 TmpDisk=224253 Uptime=1911799 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.675] debug: AcctGatherEnergy NONE plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.675] debug: AcctGatherProfile NONE plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.675] debug: AcctGatherInterconnect NONE plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">[2021-11-16T16:19:54.676] debug: AcctGatherFilesystem NONE plugin loaded<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="mso-fareast-language:EN-US">check if firewalld is enable<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">No<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Hadrian Djohari <hxd58@case.edu><br>
<b>Reply to: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Tuesday, 16 November 2021 at 16:56<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] Unable to start slurmd service<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">There can be few possibilities: <o:p></o:p></p>
<div>
<ol start="1" type="1">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Check if munge is working properly. From the scheduler master run "munge -n | ssh ecpsc10 unmunge"<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Check if selinux is enforced<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Check if firewalld or similar firewall is enabled<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Check the logs /var/log/slurm/slurmctld.log or slurmd.log on the compute node<o:p></o:p></li></ol>
<div>
<p class="MsoNormal">Best,<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Nov 16, 2021 at 10:12 AM Jaep Emmanuel <<a href="mailto:emmanuel.jaep@epfl.ch">emmanuel.jaep@epfl.ch</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="FR-CH">Hi,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="FR-CH"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">It might be a newbie question since I'm new to slurm.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">I'm trying to restart the slurmd service on one of our Ubuntu box.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">The slurmd.service is defined by:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">[Unit]</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">Description=Slurm node daemon</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">After=network.target munge.service</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">ConditionPathExists=/etc/slurm/slurm.conf</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">[Service]</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">Type=forking</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">EnvironmentFile=-/etc/sysconfig/slurmd</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">ExecStart=/usr/sbin/slurmd -d /usr/sbin/slurmstepd $SLURMD_OPTIONS</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">ExecReload=/bin/kill -HUP $MAINPID</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">PIDFile=/var/run/slurmd.pid</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">KillMode=process</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">LimitNOFILE=51200</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">LimitMEMLOCK=infinity</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">LimitSTACK=infinity</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">[Install]</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">WantedBy=multi-user.target</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">The service start without issue (systemctl start slurmd.service).</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">However, when checking the status of the service, I get a couple of error messages, but nothing alarming:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">~# systemctl status slurmd.service</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">โ slurmd.service - Slurm node daemon</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> Loaded: loaded (/etc/systemd/system/slurmd.service; enabled; vendor preset: enabled)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> Active: active (running) since Tue 2021-11-16 15:58:01 CET; 50s ago</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> Process: 2713019 ExecStart=/usr/sbin/slurmd -d /usr/sbin/slurmstepd $SLURMD_OPTIONS (code=exited, status=0/SUCCESS)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> Main PID: 2713021 (slurmd)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> Tasks: 1 (limit: 134845)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> Memory: 1.9M</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> CGroup: /system.slice/slurmd.service</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> โโ2713021 /usr/sbin/slurmd -d /usr/sbin/slurmstepd</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">Nov 16 15:58:01 ecpsc10 systemd[1]: Starting Slurm node daemon...</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">Nov 16 15:58:01 ecpsc10 systemd[1]: slurmd.service: Can't open PID file /run/slurmd.pid (yet?) after start: Operation not pe></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">Nov 16 15:58:01 ecpsc10 systemd[1]: Started Slurm node daemon.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Unfortunately, the node is still seen as down when a issue a 'sinfo':</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">root@ecpsc10:~# sinfo</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">PARTITION AVAIL TIMELIMIT NODES STATE NODELIST</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">Compute up infinite 2 idle ecpsc[11-12]</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US" style="font-family:Symbol">รจ</span><span lang="EN-US">Compute up infinite 1 down ecpsc10</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">FastCompute* up infinite 2 idle ecpsf[10-11]</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">When I get the details on this node, I get the following details:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">root@ecpsc10:~# scontrol show node ecpsc10</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">NodeName=ecpsc10 Arch=x86_64 CoresPerSocket=8</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> CPUAlloc=0 CPUErr=0 CPUTot=16 CPULoad=0.00</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> AvailableFeatures=(null)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> ActiveFeatures=(null)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> Gres=(null)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> NodeAddr=ecpsc10 NodeHostName=ecpsc10 Version=17.11</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> OS=Linux 5.8.0-43-generic #49~20.04.1-Ubuntu SMP Fri Feb 5 09:57:56 UTC 2021</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> RealMemory=40195 AllocMem=0 FreeMem=4585 Sockets=2 Boards=1</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> State=DOWN ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> </span><span lang="FR-CH">Partitions=Compute</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="FR-CH"> BootTime=2021-10-25T14:16:35 SlurmdStartTime=2021-11-16T15:58:01</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="FR-CH"> </span><span lang="EN-US">CfgTRES=cpu=16,mem=40195M,billing=16</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> AllocTRES=</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> CapWatts=n/a</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US"> Reason=Node unexpectedly rebooted [slurm@2021-11-16T14:41:04]</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">From the reason, I get that the daemon won't reload because the machine was rebooted.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">However, the /etc/slurm/slurm.conf looks like:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">root@ecpsc10:~# cat /etc/slurm/slurm.conf | grep -i returntoservice</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span lang="EN-US">ReturnToService=2</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">So I'm quite puzzled on the reason why the node will not go back online.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Any help will be greatly appreciated.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Best,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Emmanuel</span><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hadrian Djohari<br>
Manager of Research Computing Services, [U]Tech<br>
Case Western Reserve University<br>
(W): 216-368-0395<br>
(M): 216-798-7490 <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>