<div dir="auto"><div><div style="font-family:sans-serif;font-size:12.8px" dir="auto"><p>Hi,</p><p><br></p><p>Just tried running that command, but it only shows nodes that are up and running, doesn’t tell me about any nodes that are down and turned off, as an example please see below. There is a job running that should be using the 100 nodes but only 52 are allocated (plus 2 down* (that I know about and don’t care about in this case)) where are the stats and details on why the 40ish other nodes are not being used? (nothing in the masters log file either)<u></u><u></u></p><p><u></u> <u></u></p><p>btuser@bt_slurm_login001 ~ % tail /etc/slurm/slurm.conf<u></u><u></u></p><p>NodeName=ip-10-0-8-[2-100] CPUs=16 RealMemory=27648 Sockets=1 CoresPerSocket=16 ThreadsPerCore=1 State=CLOUD<u></u><u></u></p><p>NodeName=bt_slurm_login00[1-10] State=DOWN # these are the login nodes<u></u><u></u></p><p>PartitionName=backtest Nodes=ip-10-0-8-[2-100] Default=YES MaxTime=300 Oversubscribe=NO State=UP Priority=1 PreemptMode=requeue <u></u><u></u></p><p><u></u> <u></u></p><p>btuser@bt_slurm_login001 ~ % sinfo -p backtest<u></u><u></u></p><p>PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<u></u><u></u></p><p>backtest* up 5:00:00 2 down* ip-10-0-8-[29-30]<u></u><u></u></p><p>backtest* up 5:00:00 52 alloc ip-10-0-8-[4-17,19-24,26-28,31-59]<u></u><u></u></p><p><u></u> <u></u></p><p>btuser@bt_slurm_login001 ~ %<u></u><u></u></p><p>btuser@bt_slurm_login001 ~ % sinfo -p backtest -Rl -O reason:35,user,timestamp,statelong,nodelist <u></u><u></u></p><p>Wed Jun 19 01:24:59 2019<u></u><u></u></p><p>REASON USER TIMESTAMP STATE NODELIST<u></u><u></u></p><p>Not responding root 2019-06-04T04:09:31 down* ip-10-0-8-[29-30]<u></u><u></u></p><p><u></u> <u></u></p></div><br><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue., 18 Jun. 2019, 9:32 pm Sam Gallop (NBI), <<a href="mailto:sam.gallop@nbi.ac.uk">sam.gallop@nbi.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Nathan,<br>
<br>
The command I use to get the reason for failed nodes is ... 'sinfo -Ral'. If you need to extend the width of the output then ... 'sinfo -Ral -O reason:35,user,timestamp,statelong,nodelist'.<br>
<br>
Using the timestamp of the failure look in the slurmd or slurmctld logs.<br>
<br>
---<br>
Sam Gallop<br>
<br>
-----Original Message-----<br>
From: slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank" rel="noreferrer">slurm-users-bounces@lists.schedmd.com</a>> On Behalf Of nathan norton<br>
Sent: 18 June 2019 09:33<br>
To: <a href="mailto:slurm-users@lists.schedmd.com" target="_blank" rel="noreferrer">slurm-users@lists.schedmd.com</a><br>
Subject: [slurm-users] status of cloud nodes<br>
<br>
Hi all,<br>
<br>
I am using slurm with a cloud provider it is all working a treat.<br>
<br>
lets say i have 100 nodes all working fine and able to be scheduled, everything works fine.<br>
<br>
$ srun -N100 hostname<br>
<br>
works fine.<br>
<br>
For some unknown reason after machines shut down for example over the weekend if no jobs get scheduled for an hour. The next time a job runs<br>
<br>
$srun -N90 hostname<br>
<br>
fails with:<br>
<br>
"srun: Required node not available (down, drained or reserved)"<br>
<br>
"srun: job JOBID queued and waiting for resources"<br>
<br>
This is weird as no other jobs are running and i should be able to start up the nodes as requested.<br>
<br>
<br>
Being 'cloud' type nodes if i run<br>
<br>
$scontrol show node<br>
<br>
only the up and working nodes are displayed and not the failed nodes. <br>
how do i get the failed nodes information?<br>
<br>
if i stop all nodes and run below i can then start up all nodes again<br>
<br>
scontrol update NodeName=node-1-100 State=DOWN Reason="undraining"<br>
scontrol update NodeName=node-1-100 State=RESUME<br>
scontrol: show node node<br>
<br>
<br>
So that fixes it, but i want to figure out why nodes get into this state <br>
and how can i monitor it ? is there a command to get the status of CLOUD <br>
nodes?<br>
<br>
any help appreciated<br>
<br>
Thanks<br>
<br>
Nathan.<br>
<br>
<br>
<br>
</blockquote></div></div></div>