<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
The nodes are being removed as they aren't resolving in DNS anymore; are you using a dynamic system where only active hosts' names resolve?</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
<br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
Xand</div>
<div id="ms-outlook-mobile-signature" dir="auto">
<div dir="auto"><br>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Joe Teumer <joe.teumer@gmail.com><br>
<b>Sent:</b> Tuesday, October 25, 2022 7:42:16 PM<br>
<b>To:</b> slurm-users@schedmd.com <slurm-users@schedmd.com><br>
<b>Subject:</b> [slurm-users] slurmctld removing offline nodes</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>We noticed that the slurm controller will remove nodes that it cannot reach.</div>
<div>How can this be disabled?</div>
<div>We would like to see the nodes marked down/drain instead of the controller removing the nodes from sinfo.</div>
<div><br>
</div>
<div>/var/log/slurm/slurmctld.log<br>
</div>
<div>[2022-10-25T13:10:01.500] debug: Log file re-opened<br>
</div>
<div>[2022-10-25T13:10:01.589] error: get_addr_info: getaddrinfo() failed: Temporary failure in name resolution</div>
<div>[2022-10-25T13:10:01.589] error: slurm_set_addr: Unable to resolve "spg-ethx-f4ce"</div>
<div>[2022-10-25T13:10:01.589] error: slurm_get_port: Address family '0' not supported</div>
<div>[2022-10-25T13:10:01.589] error: _set_slurmd_addr: failure on spg-ethx-f4ce</div>
<div><br>
</div>
<div>
<div>cat /etc/slurm/slurm.conf | grep -i f4ce</div>
<div>NodeName=spg-ethx-f4ce ...<br>
</div>
</div>
<div>PartitionName=debug spg-ethx-f4ce ...<br>
</div>
<div><br>
</div>
<div>No output in sinfo:</div>
<div>sinfo -N | grep f4ce<br>
</div>
<div>sinfo -R | grep f4ce<br>
</div>
<div><br>
</div>
<div>
<div>slurmd -V</div>
<div>slurm 21.08.0</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>