<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Thanks Brian indeed we did have it set in bytes. I set it to the MB value. Hoping this takes care of the situation.<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jul 8, 2019, at 4:02 PM, Brian Andrus <<a href="mailto:toomuchit@gmail.com" class="">toomuchit@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Your problem here is that the configuration for the nodes in
question have an incorrect amount of memory set for them. Looks
like you have it set in bytes instead of megabytes</p><p class="">In your slurm.conf you should look at the RealMemory setting:</p><div class=""><br class="webkit-block-placeholder"></div><dt style="box-sizing: border-box; color: rgb(70, 84, 92);
font-family: "Source Sans Pro", Helvetica, Arial,
sans-serif; font-size: 20px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-style: initial; text-decoration-color:
initial;" class=""><b style="box-sizing: border-box; margin: 0px;
padding: 0px; border: 0px; font-style: inherit; font-variant:
inherit; font-weight: bold; font-stretch: inherit; font-size:
inherit; line-height: inherit; font-family: inherit;
vertical-align: baseline;" class="">RealMemory</b></dt>
<dd style="box-sizing: border-box; color: rgb(70, 84, 92);
font-family: "Source Sans Pro", Helvetica, Arial,
sans-serif; font-size: 20px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255); text-decoration-style: initial; text-decoration-color:
initial;" class="">Size of real memory on the node in megabytes (e.g.
"2048"). The default value is 1.<span class=""> </span></dd><div class=""><br class="webkit-block-placeholder"></div><p class="">I would suggest RealMemory=191879 , where I suspect you have
RealMemory=196489092</p><p class="">Brian Andrus<br class="">
</p>
<div class="moz-cite-prefix">On 7/8/2019 11:59 AM, Robert Kudyba
wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:D788E26B-4C5D-4059-8EE6-0DB7219DE1A6@fordham.edu" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
<div class="">I’m new to Slurm and we have a 3 node + head node
cluster running Centos 7 and Bright Cluster 8.1. Their support
sent me here as they say Slurm is configured optimally to allow
multiple tasks to run. However at times a job will hold up new
jobs. Are there any other logs I can look at and/or settings to
change to prevent this or alert me when this is happening? Here
are some tests and commands that I hope will illuminate where I
may be going wrong. The slurn.conf file has these options set:</div>
<div class="">SelectType=select/cons_res</div>
<div class="">SelectTypeParameters=CR_CPU<br class="">
SchedulerTimeSlice=60<br class="">
<br class="">
</div>
<div class="">I also see <span style="font-family: Menlo;
font-size: 11px; background-color: rgb(255, 255, 255);" class="">/var/log/slurmctld is loaded with errors like these:</span></div>
<div class="">[2019-07-03T02:21:30.913] error:
_slurm_rpc_node_registration node=node003: Invalid argument<br class="">
[2019-07-03T02:54:50.655] error: Node node002 has low
real_memory size (191879 < 196489092)<br class="">
[2019-07-03T02:54:50.655] error: _slurm_rpc_node_registration
node=node002: Invalid argument<br class="">
[2019-07-03T02:54:50.655] error: Node node001 has low
real_memory size (191883 < 196489092)<br class="">
[2019-07-03T02:54:50.655] error: _slurm_rpc_node_registration
node=node001: Invalid argument<br class="">
[2019-07-03T02:54:50.655] error: Node node003 has low
real_memory size (191879 < 196489092)<br class="">
[2019-07-03T02:54:50.655] error: _slurm_rpc_node_registration
node=node003: Invalid argument<br class="">
[2019-07-03T03:28:10.293] error: Node node002 has low
real_memory size (191879 < 196489092)<br class="">
[2019-07-03T03:28:10.293] error: _slurm_rpc_node_registration
node=node002: Invalid argument<br class="">
[2019-07-03T03:28:10.293] error: Node node003 has low
real_memory size (191879 < 196489092)</div>
<div class=""><br class="">
</div>
<div class=""><span style="caret-color: rgb(102, 102, 102); color:
rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica;
font-size: 14px; background-color: rgb(255, 255, 255);" class="">squeue</span><br style="caret-color: rgb(102, 102,
102); color: rgb(102, 102, 102); font-family: Verdana, Arial,
Helvetica; font-size: 14px;" class="">
<font class="" face="Courier"><span style="caret-color: rgb(102,
102, 102); color: rgb(102, 102, 102); font-size: 14px;
background-color: rgb(255, 255, 255);" class="">JOBID
PARTITION NAME <span class="Apple-tab-span" style="white-space:pre"> </span> USER
ST TIME NODES </span><span style="caret-color: rgb(102,
102, 102); color: rgb(102, 102, 102); font-size: 14px;
background-color: rgb(255, 255, 255);" class="">NODELIST(REASON)</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-size: 14px; background-color: rgb(255, 255,
255);" class="">352 defq <span class="Apple-tab-span" style="white-space:pre"> </span>TensorFl </span>myuser<span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-size: 14px; background-color: rgb(255, 255,
255);" class=""> PD 0:00 3 </span><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-size: 14px; background-color: rgb(255, 255,
255);" class="">(Resources)</span></font><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
</div>
<div class=""><span style="caret-color: rgb(102, 102, 102); color:
rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica;
font-size: 14px; background-color: rgb(255, 255, 255);" class=""><br class="">
</span></div>
<div class="">
<div class=""> scontrol show jobid -dd 352</div>
<div class="">JobId=352 JobName=TensorFlowGPUTest</div>
<div class="">UserId=myuser(1001) GroupId=myuser(1001)
MCS_label=N/A</div>
<div class="">Priority=4294901741 Nice=0 Account=(null)
QOS=normal</div>
<div class="">JobState=PENDING Reason=Resources
Dependency=(null)</div>
<div class="">Requeue=1 Restarts=0 BatchFlag=1 Reboot=0
ExitCode=0:0</div>
<div class="">DerivedExitCode=0:0</div>
<div class="">RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A</div>
<div class="">SubmitTime=2019-07-02T16:57:11
EligibleTime=2019-07-02T16:57:11</div>
<div class="">StartTime=Unknown EndTime=Unknown Deadline=N/A</div>
<div class="">PreemptTime=None SuspendTime=None SecsPreSuspend=0</div>
<div class="">LastSchedEval=2019-07-02T16:57:59</div>
<div class="">Partition=defq AllocNode:Sid=ourcluster:386851</div>
<div class="">ReqNodeList=(null) ExcNodeList=(null)</div>
<div class="">NodeList=(null)</div>
<div class="">NumNodes=3-3 NumCPUs=3 NumTasks=3 CPUs/Task=1
ReqB:S:C:T=0:0:*:*</div>
<div class="">TRES=cpu=3,node=3</div>
<div class="">Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*</div>
<div class="">MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0</div>
<div class="">Features=(null) DelayBoot=00:00:00</div>
<div class="">Gres=gpu:1 Reservation=(null)</div>
<div class="">OverSubscribe=YES Contiguous=0 Licenses=(null)
Network=(null)</div>
<div class="">Command=/home/myuser/cnn_gpu.sh</div>
<div class="">WorkDir=/home/myuser</div>
<div class="">StdErr=/home/myuser/slurm-352.out</div>
<div class="">StdIn=/dev/null</div>
<div class="">StdOut=/home/myuser/slurm-352.out</div>
<div class="">Power=</div>
</div>
<div class=""><br class="">
</div>
<div class="">Another test showed the below:</div>
<div class="">sinfo -N<br class="">
NODELIST NODES PARTITION STATE<br class="">
node001 1 defq* drain<br class="">
node002 1 defq* drain<br class="">
node003 1 defq* drain<br class="">
<br class="">
sinfo -R<br class="">
REASON USER TIMESTAMP NODELIST<br class="">
Low RealMemory slurm 2019-05-17T10:05:26 node[001-003]</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><span style="caret-color: rgb(102, 102, 102); color:
rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica;
font-size: 14px; background-color: rgb(255, 255, 255);" class="">[ciscluster]% jobqueue</span><br style="caret-color:
rgb(102, 102, 102); color: rgb(102, 102, 102); font-family:
Verdana, Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">[ciscluster->jobqueue(slurm)]%
ls</span><br style="caret-color: rgb(102, 102, 102); color:
rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica;
font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">Type
Name Nodes</span><br style="caret-color: rgb(102, 102, 102);
color: rgb(102, 102, 102); font-family: Verdana, Arial,
Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">------------
------------------------</span><br style="caret-color:
rgb(102, 102, 102); color: rgb(102, 102, 102); font-family:
Verdana, Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">----------------------------------------------------</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">Slurm
defq node001..node003</span><br style="caret-color: rgb(102,
102, 102); color: rgb(102, 102, 102); font-family: Verdana,
Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">Slurm
gpuq</span><br style="caret-color: rgb(102, 102, 102); color:
rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica;
font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">[ourcluster->jobqueue(slurm)]%
use defq</span><br style="caret-color: rgb(102, 102, 102);
color: rgb(102, 102, 102); font-family: Verdana, Arial,
Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">[ourcluster->jobqueue(slurm)->defq]%
get options</span><br style="caret-color: rgb(102, 102, 102);
color: rgb(102, 102, 102); font-family: Verdana, Arial,
Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">QoS=N/A
ExclusiveUser=NO OverSubscribe=FORCE:12 OverTimeL imit=0
State=UP</span></div>
<div class=""><font class="" face="Verdana, Arial, Helvetica" color="#666666"><span style="caret-color: rgb(102, 102, 102);
font-size: 14px; background-color: rgb(255, 255, 255);" class=""><br class="">
</span></font></div>
<div class=""><span style="caret-color: rgb(102, 102, 102); color:
rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica;
font-size: 14px; background-color: rgb(255, 255, 255);" class="">pdsh -w node00[1-3] "lscpu | grep -iE 'socket|core'" </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">node003:
Thread(s) per core: 1 </span><br style="caret-color: rgb(102,
102, 102); color: rgb(102, 102, 102); font-family: Verdana,
Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">node003:
Core(s) per socket: 12 </span><br style="caret-color: rgb(102,
102, 102); color: rgb(102, 102, 102); font-family: Verdana,
Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">node003:
Socket(s): 2 </span><br style="caret-color: rgb(102, 102,
102); color: rgb(102, 102, 102); font-family: Verdana, Arial,
Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">node001:
Thread(s) per core: 1 </span><br style="caret-color: rgb(102,
102, 102); color: rgb(102, 102, 102); font-family: Verdana,
Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">node001:
Core(s) per socket: 12 </span><br style="caret-color: rgb(102,
102, 102); color: rgb(102, 102, 102); font-family: Verdana,
Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">node001:
Socket(s): 2 </span><br style="caret-color: rgb(102, 102,
102); color: rgb(102, 102, 102); font-family: Verdana, Arial,
Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">node002:
Thread(s) per core: 1 </span><br style="caret-color: rgb(102,
102, 102); color: rgb(102, 102, 102); font-family: Verdana,
Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">node002:
Core(s) per socket: 12 </span><br style="caret-color: rgb(102,
102, 102); color: rgb(102, 102, 102); font-family: Verdana,
Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">node002:
Socket(s): 2 </span></div>
<div class=""><span style="background-color: rgb(255, 255, 255);" class=""><br class="">
</span></div>
<div class=""><span style="caret-color: rgb(102, 102, 102); color:
rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica;
font-size: 14px; background-color: rgb(255, 255, 255);" class="">scontrol show nodes node001 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">NodeName=node001
Arch=x86_64 CoresPerSocket=12 </span><br style="caret-color:
rgb(102, 102, 102); color: rgb(102, 102, 102); font-family:
Verdana, Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">CPUAlloc=0
CPUErr=0 CPUTot=24 CPULoad=0.01 </span><br style="caret-color:
rgb(102, 102, 102); color: rgb(102, 102, 102); font-family:
Verdana, Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">AvailableFeatures=(null) </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">Ac
tiveFeatures=(null) </span><br style="caret-color: rgb(102,
102, 102); color: rgb(102, 102, 102); font-family: Verdana,
Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">Gres=gpu:1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">NodeAddr=node001
NodeHostName=node001 Version=17.11 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">OS=Linux
3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">RealMemory=196489092
AllocMem=0 FreeMem=184912 Sockets=2 Boards=1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">State=IDLE+DRAIN
ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">Partitions=defq </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">BootTime=2019-06-28T15:33:47
SlurmdStartTime=2019-06-28T15:35:17 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">CfgTRES=cpu=24,mem=196489092M,billing=24 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">AllocTRES= </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">CapWatts=n/a </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">CurrentWatts=0
LowestJoules=0 ConsumedJoules=0 </span><br style="caret-color:
rgb(102, 102, 102); color: rgb(102, 102, 102); font-family:
Verdana, Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">ExtSensorsJoules=n/s
ExtSensorsWatts=0 ExtSensorsTemp=n/s </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">Reason=Low
RealMemory [slurm@2019-05-17T10:05:26] </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><span style="caret-color: rgb(102, 102, 102); color:
rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica;
font-size: 14px; background-color: rgb(255, 255, 255);" class="">sinfo </span><br style="caret-color: rgb(102, 102,
102); color: rgb(102, 102, 102); font-family: Verdana, Arial,
Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">PARTITION
AVAIL TI MELIMIT NODES STATE NODELIST </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">defq* up
infinite 3 drain node[001-003] </span><br style="caret-color:
rgb(102, 102, 102); color: rgb(102, 102, 102); font-family:
Verdana, Arial, Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">gpuq up
infinite 0 n/a </span><br style="caret-color: rgb(102, 102,
102); color: rgb(102, 102, 102); font-family: Verdana, Arial,
Helvetica; font-size: 14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class=""><br class="">
</span></div>
<div class=""><span style="caret-color: rgb(102, 102, 102); color:
rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica;
font-size: 14px; background-color: rgb(255, 255, 255);" class=""><br class="">
</span></div>
<div class=""><span style="caret-color: rgb(102, 102, 102); color:
rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica;
font-size: 14px; background-color: rgb(255, 255, 255);" class="">scontrol show nodes| grep -i mem </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">RealMemory=196489092
AllocMem=0 FreeMem=184907 Sockets=2 Boards=1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">CfgTRES=cpu=24,mem=196489092M,billing=24 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">Reason=Low
RealMemory [slurm@2019-05-17T10:05:26] </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">RealMemory=196489092
AllocMem=0 FreeMem=185084 Sockets=2 Boards=1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">CfgTRES=cpu=24,mem=196489092M,billing=24 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">Reason=Low
RealMemory [slurm@2019-05-17T10:05:26] </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">RealMemory=196489092
AllocMem=0 FreeMem=188720 Sockets=2 Boards=1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">CfgTRES=cpu=24,mem=196489092M,billing=24 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102,
102); font-family: Verdana, Arial, Helvetica; font-size:
14px;" class="">
<span style="caret-color: rgb(102, 102, 102); color: rgb(102,
102, 102); font-family: Verdana, Arial, Helvetica; font-size:
14px; background-color: rgb(255, 255, 255);" class="">Reason=Low
RealMemory [slurm@2019-05-17T10:05:26] </span></div>
</blockquote>
</div>
</div></blockquote></div><br class=""></body></html>