Hi all,<div><br></div><div>Out of curiosity, what causes that? It'd be good to know for the future -- I ran into the same issue and just edited the memory down and it works fine now, but I'd like to know why/what causes that error. I'm assuming low resources, ie memory or CPU or whatever. Mind clarifying?<br><div dir="auto"><br><div class="gmail_quote"><div dir="ltr">On Wed, May 2, 2018, 7:11 PM John Kelly <<a href="mailto:john.kelly@broadcom.com">john.kelly@broadcom.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi matt</div><div><br></div><div>scontrol update nodename=odin state=resume</div><div>scontrol update nodename=odin state=idle</div><div><br></div><div>-jfk</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 2, 2018 at 5:28 PM, Matt Hohmeister <span dir="ltr"><<a href="mailto:hohmeister@psy.fsu.edu" target="_blank">hohmeister@psy.fsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_-1171031600902622751m_-3212876675101077922WordSection1">
<p class="MsoNormal">I have a two-node cluster: the server/compute node is a Dell PowerEdge R730; the compute node, a Dell PowerEdge R630. On both of these nodes,
<span style="font-family:Consolas">slurmd -C</span> gives me the exact same line:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-family:Consolas">[me@odin slurm]$ slurmd -C<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">NodeName=odin CPUs=40 Boards=1 SocketsPerBoard=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=128655<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">[me@thor slurm]$ slurmd -C<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">NodeName=thor CPUs=40 Boards=1 SocketsPerBoard=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=128655<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">So I edited my slurm.conf appropriately:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-family:Consolas">NodeName=odin CPUs=40 Boards=1 SocketsPerBoard=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=128655<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">NodeName=thor CPUs=40 Boards=1 SocketsPerBoard=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=128655<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">…and it looks good, except for the drain on my server/compute node:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-family:Consolas">[me@odin slurm]$ sinfo<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">debug* up infinite 1 drain odin<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">debug* up infinite 1 idle thor
<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">…for the following reason:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-family:Consolas">[me@odin slurm]$ sinfo -R<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">REASON USER TIMESTAMP NODELIST<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Low socket*core*thre slurm 2018-05-02T11:55:38 odin<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Any ideas?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Matt Hohmeister<u></u><u></u></p>
<p class="MsoNormal">Systems and Network Administrator<u></u><u></u></p>
<p class="MsoNormal">Department of Psychology<u></u><u></u></p>
<p class="MsoNormal">Florida State University<u></u><u></u></p>
<p class="MsoNormal">PO Box 3064301<u></u><u></u></p>
<p class="MsoNormal">Tallahassee, FL 32306-4301<u></u><u></u></p>
<p class="MsoNormal">Phone: +1 850 645 1902<u></u><u></u></p>
<p class="MsoNormal">Fax: +1 850 644 7739<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</blockquote></div><br></div>
</blockquote></div></div></div>