<div dir="ltr">Hi Caleb<div><br></div><div>I noticed the same thing. If you configure a host with more memory than it really has slurm will think that the host has something wrong with it and put it in drain status. At least that is my theory. The vendor can likely give you a better more detailed answer. </div><div><br></div><div><br></div><div>-jfk</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 2, 2018 at 6:23 PM, Caleb Smith <span dir="ltr"><<a href="mailto:caleb@calebsmith.net" target="_blank">caleb@calebsmith.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<div><br></div><div>Out of curiosity, what causes that? It'd be good to know for the future -- I ran into the same issue and just edited the memory down and it works fine now, but I'd like to know why/what causes that error. I'm assuming low resources, ie memory or CPU or whatever. Mind clarifying?<br><div dir="auto"><br><div class="gmail_quote"><div dir="ltr">On Wed, May 2, 2018, 7:11 PM John Kelly <<a href="mailto:john.kelly@broadcom.com" target="_blank">john.kelly@broadcom.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi matt</div><div><br></div><div>scontrol update nodename=odin state=resume</div><div>scontrol update nodename=odin state=idle</div><div><br></div><div>-jfk</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 2, 2018 at 5:28 PM, Matt Hohmeister <span dir="ltr"><<a href="mailto:hohmeister@psy.fsu.edu" target="_blank">hohmeister@psy.fsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_-5076401645372159752m_-1171031600902622751m_-3212876675101077922WordSection1">
<p class="MsoNormal">I have a two-node cluster: the server/compute node is a Dell PowerEdge R730; the compute node, a Dell PowerEdge R630. On both of these nodes,
<span style="font-family:Consolas">slurmd -C</span> gives me the exact same line:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-family:Consolas">[me@odin slurm]$ slurmd -C<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">NodeName=odin CPUs=40 Boards=1 SocketsPerBoard=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=128655<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">[me@thor slurm]$ slurmd -C<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">NodeName=thor CPUs=40 Boards=1 SocketsPerBoard=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=128655<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">So I edited my slurm.conf appropriately:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-family:Consolas">NodeName=odin CPUs=40 Boards=1 SocketsPerBoard=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=128655<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">NodeName=thor CPUs=40 Boards=1 SocketsPerBoard=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=128655<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">…and it looks good, except for the drain on my server/compute node:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-family:Consolas">[me@odin slurm]$ sinfo<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">debug* up infinite 1 drain odin<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">debug* up infinite 1 idle thor
<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">…for the following reason:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-family:Consolas">[me@odin slurm]$ sinfo -R<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">REASON USER TIMESTAMP NODELIST<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Low socket*core*thre slurm 2018-05-02T11:55:38 odin<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Any ideas?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Matt Hohmeister<u></u><u></u></p>
<p class="MsoNormal">Systems and Network Administrator<u></u><u></u></p>
<p class="MsoNormal">Department of Psychology<u></u><u></u></p>
<p class="MsoNormal">Florida State University<u></u><u></u></p>
<p class="MsoNormal">PO Box 3064301<u></u><u></u></p>
<p class="MsoNormal">Tallahassee, FL 32306-4301<u></u><u></u></p>
<p class="MsoNormal">Phone: +1 850 645 1902<u></u><u></u></p>
<p class="MsoNormal">Fax: +1 850 644 7739<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</blockquote></div><br></div>
</blockquote></div></div></div>
</blockquote></div><br></div>