<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">
<div>>Your running job is requesting 6 CPUs per node (4 nodes, 6 CPUs per node). That means 6 CPUs are being used on node hpc.</div>
<div>>Your queued job is requesting 5 CPUs per node (4 nodes, 5 CPUs per
node). In total, if it was running, that would require 11 CPUs on node
hpc. But hpc only has 10 cores, so it can't run.</div>
</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Right... I changed that but still the job is in pending state.</div><div class="gmail_default" style="font-family:tahoma,sans-serif"></div><div class="gmail_default" style="font-family:tahoma,sans-serif">I modified /etc/slurm/slurm.conf as below</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"># grep hpc /etc/slurm/slurm.conf<br>NodeName=hpc NodeAddr=10.1.1.1 CPUs=11</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div># for i in {0..2}; do scontrol show node compute-0-$i | grep RealMemory; done && scontrol show node hpc | grep RealMemory<br> RealMemory=64259 AllocMem=1024 FreeMem=57116 Sockets=32 Boards=1<br> RealMemory=120705 AllocMem=1024 FreeMem=66403 Sockets=32 Boards=1<br> RealMemory=64259 AllocMem=1024 FreeMem=39966 Sockets=32 Boards=1<br> RealMemory=64259 AllocMem=1024 FreeMem=49189 Sockets=11 Boards=1<br># for i in {0..2}; do scontrol show node compute-0-$i | grep CPUTot; done && scontrol show node hpc | grep CPUTot<br> CPUAlloc=6 CPUTot=32 CPULoad=5.18<br> CPUAlloc=6 CPUTot=32 CPULoad=18.94<br> CPUAlloc=6 CPUTot=32 CPULoad=5.41<br> CPUAlloc=6 CPUTot=11 CPULoad=5.21<br><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">But still the job is pending</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">$ scontrol show -d job 129<br>JobId=129 JobName=qe-fb<br> UserId=mahmood(1000) GroupId=mahmood(1000) MCS_label=N/A<br> Priority=1751 Nice=0 Account=fish QOS=normal WCKey=*default<br> JobState=PENDING Reason=Resources Dependency=(null)<br> Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0<br> DerivedExitCode=0:0<br> RunTime=00:00:00 TimeLimit=30-00:00:00 TimeMin=N/A<br> SubmitTime=2019-12-17T15:00:37 EligibleTime=2019-12-17T15:00:37<br> AccrueTime=2019-12-17T15:00:37<br> StartTime=Unknown EndTime=Unknown Deadline=N/A<br> SuspendTime=None SecsPreSuspend=0 LastSchedEval=2019-12-17T15:00:38<br> Partition=SEA AllocNode:Sid=<a href="http://hpc.scu.ac.ir:14534">hpc.scu.ac.ir:14534</a><br> ReqNodeList=(null) ExcNodeList=(null)<br> NodeList=(null)<br> NumNodes=4-4 NumCPUs=20 NumTasks=20 CPUs/Task=1 ReqB:S:C:T=0:0:*:*<br> TRES=cpu=20,mem=40G,node=4,billing=20<br> Socks/Node=* NtasksPerN:B:S:C=5:0:*:* CoreSpec=*<br> MinCPUsNode=5 MinMemoryNode=10G MinTmpDiskNode=0<br> Features=(null) DelayBoot=00:00:00<br> OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)<br> Command=/home/mahmood/qe/f_borophene/slurm_qe.sh<br> WorkDir=/home/mahmood/qe/f_borophene<br> StdErr=/home/mahmood/qe/f_borophene/my_fb.log<br> StdIn=/dev/null<br> StdOut=/home/mahmood/qe/f_borophene/my_fb.log<br> Power=</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">
<span class="gmail_default" style="font-family:tahoma,sans-serif"></span>>I'm not aware of any nodes, that have 32, or even 10 sockets. Are
you sure, you want to use the cluster like that?</div><div class="gmail_default" style="font-family:tahoma,sans-serif"> <br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Marcus,</div><div class="gmail_default" style="font-family:tahoma,sans-serif">I have installed slurm via slurm roll on Rocks. All 4 nodes are dual socket Opetron 6282 with the following specs</div><div class="gmail_default" style="font-family:tahoma,sans-serif">Thread(s) per core: 2<br>Core(s) per socket: 8<br>Socket(s): 2</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">I just wrote 11 CPUs for the head node in order to not fully utilize the head node with jobs.</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">For example, compute-0-0 is</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">$ scontrol show node compute-0-0<br>NodeName=compute-0-0 Arch=x86_64 CoresPerSocket=1<br> CPUAlloc=6 CPUTot=32 CPULoad=5.15<br> AvailableFeatures=rack-0,32CPUs<br> ActiveFeatures=rack-0,32CPUs<br> Gres=(null)<br> NodeAddr=10.1.1.254 NodeHostName=compute-0-0<br> OS=Linux 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019<br> RealMemory=64259 AllocMem=1024 FreeMem=57050 Sockets=32 Boards=1<br> State=MIXED ThreadsPerCore=1 TmpDisk=444124 Weight=20511900 Owner=N/A MCS_label=N/A<br> Partitions=CLUSTER,WHEEL,SEA<br> BootTime=2019-10-10T19:01:38 SlurmdStartTime=2019-12-17T13:50:37<br> CfgTRES=cpu=32,mem=64259M,billing=47<br> AllocTRES=cpu=6,mem=1G<br> CapWatts=n/a<br> CurrentWatts=0 AveWatts=0<br> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><font face="tahoma,sans-serif">Regards,<br>Mahmood</font><br><br><br></div></div></div><br></div><br></div>