<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">I’m new to Slurm and we have a 3 node + head node cluster running Centos 7 and Bright Cluster 8.1. Their support sent me here as they say Slurm is configured optimally to allow multiple tasks to run. However at times a job will hold up new jobs. Are there any other logs I can look at and/or settings to change to prevent this or alert me when this is happening? Here are some tests and commands that I hope will illuminate where I may be going wrong. The slurn.conf file has these options set:</div><div class="">SelectType=select/cons_res</div><div class="">SelectTypeParameters=CR_CPU<br class="">SchedulerTimeSlice=60<br class=""><br class=""></div><div class="">I also see <span style="font-family: Menlo; font-size: 11px; background-color: rgb(255, 255, 255);" class="">/var/log/slurmctld is loaded with errors like these:</span></div><div class="">[2019-07-03T02:21:30.913] error: _slurm_rpc_node_registration node=node003: Invalid argument<br class="">[2019-07-03T02:54:50.655] error: Node node002 has low real_memory size (191879 < 196489092)<br class="">[2019-07-03T02:54:50.655] error: _slurm_rpc_node_registration node=node002: Invalid argument<br class="">[2019-07-03T02:54:50.655] error: Node node001 has low real_memory size (191883 < 196489092)<br class="">[2019-07-03T02:54:50.655] error: _slurm_rpc_node_registration node=node001: Invalid argument<br class="">[2019-07-03T02:54:50.655] error: Node node003 has low real_memory size (191879 < 196489092)<br class="">[2019-07-03T02:54:50.655] error: _slurm_rpc_node_registration node=node003: Invalid argument<br class="">[2019-07-03T03:28:10.293] error: Node node002 has low real_memory size (191879 < 196489092)<br class="">[2019-07-03T03:28:10.293] error: _slurm_rpc_node_registration node=node002: Invalid argument<br class="">[2019-07-03T03:28:10.293] error: Node node003 has low real_memory size (191879 < 196489092)</div><div class=""><br class=""></div><div class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">squeue</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><font face="Courier" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-size: 14px; background-color: rgb(255, 255, 255);" class="">JOBID PARTITION NAME <span class="Apple-tab-span" style="white-space:pre"> </span> USER ST TIME NODES </span><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-size: 14px; background-color: rgb(255, 255, 255);" class="">NODELIST(REASON)</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-size: 14px; background-color: rgb(255, 255, 255);" class="">352 defq <span class="Apple-tab-span" style="white-space:pre"> </span>TensorFl </span>myuser<span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-size: 14px; background-color: rgb(255, 255, 255);" class=""> PD 0:00 3 </span><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-size: 14px; background-color: rgb(255, 255, 255);" class="">(Resources)</span></font><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""></div><div class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class=""><br class=""></span></div><div class=""><div class=""> scontrol show jobid -dd 352</div><div class="">JobId=352 JobName=TensorFlowGPUTest</div><div class="">UserId=myuser(1001) GroupId=myuser(1001) MCS_label=N/A</div><div class="">Priority=4294901741 Nice=0 Account=(null) QOS=normal</div><div class="">JobState=PENDING Reason=Resources Dependency=(null)</div><div class="">Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0</div><div class="">DerivedExitCode=0:0</div><div class="">RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A</div><div class="">SubmitTime=2019-07-02T16:57:11 EligibleTime=2019-07-02T16:57:11</div><div class="">StartTime=Unknown EndTime=Unknown Deadline=N/A</div><div class="">PreemptTime=None SuspendTime=None SecsPreSuspend=0</div><div class="">LastSchedEval=2019-07-02T16:57:59</div><div class="">Partition=defq AllocNode:Sid=ourcluster:386851</div><div class="">ReqNodeList=(null) ExcNodeList=(null)</div><div class="">NodeList=(null)</div><div class="">NumNodes=3-3 NumCPUs=3 NumTasks=3 CPUs/Task=1 ReqB:S:C:T=0:0:*:*</div><div class="">TRES=cpu=3,node=3</div><div class="">Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*</div><div class="">MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0</div><div class="">Features=(null) DelayBoot=00:00:00</div><div class="">Gres=gpu:1 Reservation=(null)</div><div class="">OverSubscribe=YES Contiguous=0 Licenses=(null) Network=(null)</div><div class="">Command=/home/myuser/cnn_gpu.sh</div><div class="">WorkDir=/home/myuser</div><div class="">StdErr=/home/myuser/slurm-352.out</div><div class="">StdIn=/dev/null</div><div class="">StdOut=/home/myuser/slurm-352.out</div><div class="">Power=</div></div><div class=""><br class=""></div><div class="">Another test showed the below:</div><div class="">sinfo -N<br class="">NODELIST NODES PARTITION STATE<br class="">node001 1 defq* drain<br class="">node002 1 defq* drain<br class="">node003 1 defq* drain<br class=""><br class="">sinfo -R<br class="">REASON USER TIMESTAMP NODELIST<br class="">Low RealMemory slurm 2019-05-17T10:05:26 node[001-003]</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">[ciscluster]% jobqueue</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">[ciscluster->jobqueue(slurm)]% ls</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">Type Name Nodes</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">------------ ------------------------</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">----------------------------------------------------</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">Slurm defq node001..node003</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">Slurm gpuq</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">[ourcluster->jobqueue(slurm)]% use defq</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">[ourcluster->jobqueue(slurm)->defq]% get options</span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">QoS=N/A ExclusiveUser=NO OverSubscribe=FORCE:12 OverTimeL imit=0 State=UP</span></div><div class=""><font color="#666666" face="Verdana, Arial, Helvetica" class=""><span style="caret-color: rgb(102, 102, 102); font-size: 14px; background-color: rgb(255, 255, 255);" class=""><br class=""></span></font></div><div class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">pdsh -w node00[1-3] "lscpu | grep -iE 'socket|core'" </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">node003: Thread(s) per core: 1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">node003: Core(s) per socket: 12 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">node003: Socket(s): 2 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">node001: Thread(s) per core: 1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">node001: Core(s) per socket: 12 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">node001: Socket(s): 2 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">node002: Thread(s) per core: 1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">node002: Core(s) per socket: 12 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">node002: Socket(s): 2 </span></div><div class=""><span style="background-color: rgb(255, 255, 255);" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">scontrol show nodes node001 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">NodeName=node001 Arch=x86_64 CoresPerSocket=12 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">CPUAlloc=0 CPUErr=0 CPUTot=24 CPULoad=0.01 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">AvailableFeatures=(null) </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">Ac tiveFeatures=(null) </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">Gres=gpu:1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">NodeAddr=node001 NodeHostName=node001 Version=17.11 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">OS=Linux 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">RealMemory=196489092 AllocMem=0 FreeMem=184912 Sockets=2 Boards=1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">State=IDLE+DRAIN ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">Partitions=defq </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">BootTime=2019-06-28T15:33:47 SlurmdStartTime=2019-06-28T15:35:17 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">CfgTRES=cpu=24,mem=196489092M,billing=24 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">AllocTRES= </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">CapWatts=n/a </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">Reason=Low RealMemory [slurm@2019-05-17T10:05:26] </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><br class=""></div><div class=""><br class=""></div><div class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">sinfo </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">PARTITION AVAIL TI MELIMIT NODES STATE NODELIST </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">defq* up infinite 3 drain node[001-003] </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">gpuq up infinite 0 n/a </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">scontrol show nodes| grep -i mem </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">RealMemory=196489092 AllocMem=0 FreeMem=184907 Sockets=2 Boards=1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">CfgTRES=cpu=24,mem=196489092M,billing=24 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">Reason=Low RealMemory [slurm@2019-05-17T10:05:26] </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">RealMemory=196489092 AllocMem=0 FreeMem=185084 Sockets=2 Boards=1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">CfgTRES=cpu=24,mem=196489092M,billing=24 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">Reason=Low RealMemory [slurm@2019-05-17T10:05:26] </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">RealMemory=196489092 AllocMem=0 FreeMem=188720 Sockets=2 Boards=1 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">CfgTRES=cpu=24,mem=196489092M,billing=24 </span><br style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px;" class=""><span style="caret-color: rgb(102, 102, 102); color: rgb(102, 102, 102); font-family: Verdana, Arial, Helvetica; font-size: 14px; background-color: rgb(255, 255, 255);" class="">Reason=Low RealMemory [slurm@2019-05-17T10:05:26] </span></div></body></html>