<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
A quick test to see if it's a configuration error is to set config_overrides in your slurm.conf and see if the node then responds to scontrol update. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Brian Andrus <toomuchit@gmail.com><br>
<b>Sent:</b> Thursday, May 25, 2023 10:54 AM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Nodes stuck in drain state</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">That output of slurmd -C is your answer.<br>
<br>
Slurmd only sees 6GB of memory and you are claiming it has 10GB.<br>
<br>
I would run some memtests, look at meminfo on the node, etc.<br>
<br>
Maybe even check that the type/size of memory in there is what you think <br>
it is.<br>
<br>
Brian Andrus<br>
<br>
On 5/25/2023 7:30 AM, Roger Mason wrote:<br>
> Ole Holm Nielsen <Ole.H.Nielsen@fysik.dtu.dk> writes:<br>
><br>
>> 1. Is slurmd running on the node?<br>
> Yes.<br>
><br>
>> 2. What's the output of "slurmd -C" on the node?<br>
> NodeName=node012 CPUs=4 Boards=1 SocketsPerBoard=2 CoresPerSocket=2<br>
> ThreadsPerCore=1 RealMemory=6097<br>
><br>
>> 3. Define State=UP in slurm.conf in stead of UNKNOWN<br>
> Will do.<br>
><br>
>> 4. Why have you configured TmpDisk=0? It should be the size of the<br>
>> /tmp filesystem.<br>
> I have not configured TmpDisk. This the entry in slurm.conf for that<br>
> node:<br>
> NodeName=node012 CPUs=4 Boards=1 SocketsPerBoard=2 CoresPerSocket=2<br>
> ThreadsPerCore=1 RealMemory=10193 State=UNKNOWN<br>
><br>
> But I do notice that slurmd -C now says there is less memory than<br>
> configured.<br>
><br>
> Thanks again.<br>
><br>
> Roger<br>
><br>
<br>
</div>
</span></font></div>
</body>
</html>