<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">Yes. It seems that what user specifies, slurm will reserve that. The other jobs realtime memory is less than what users had been specified. I thought that slurm will dynamically handles that in order to put more jobs in running state.</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br clear="all"></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><font face="tahoma,sans-serif">Regards,<br>Mahmood</font><br><br><br></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 17, 2019 at 7:54 PM Prentice Bisbal <<a href="mailto:pbisbal@pppl.gov">pbisbal@pppl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Mahmood, <br>
</p>
<p>What do you see as the problem here? To me, there is no problem
and the scheduler is working exactly has it should. The reason
"Resources" means that there are not enough computing resources
available for your job to run right now, so the job is setting in
the queue in the pending state waiting for the necessary resources
to become available. This is exactly what schedulers are<br>
</p>
<p>As Andreas pointed out, looking at the output of 'scontrol show
node compute-0-0' that you provided, compute-0-0 has 32 cores and
63 GB of RAM. Out of that 9 cores and 55 GB of RAM have already
been allocated, leaving 23 cores and only 8 GB of RAM available
for other jobs. The job you submitted requested 20 cores (tasks,
technically) and 40 GB of RAM. Since compute-0-0 doesn't have
enough RAM available, Slurm is keeping your job in the queue until
enough RAM is available for it to run. This is exactly what Slurm
should be doing. <br>
</p>
<pre class="gmail-m_-746978286124028451moz-signature" cols="72">Prentice </pre>
<div class="gmail-m_-746978286124028451moz-cite-prefix">On 4/17/19 11:00 AM, Henkel, Andreas
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">I think there isn’t enough memory. </div>
<div dir="ltr">AllocTres Shows mem=55G</div>
<div dir="ltr">And your job wants another 40G although the node
only has 63G in total. </div>
<div dir="ltr">Best,</div>
<div dir="ltr">Andreas </div>
<div dir="ltr"><br>
Am 17.04.2019 um 16:45 schrieb Mahmood Naderan <<a href="mailto:mahmood.nt@gmail.com" target="_blank">mahmood.nt@gmail.com</a>>:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default" style="font-family:tahoma,sans-serif">Hi,</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">Although it
was fine for previous job runs, the following
script now stuck as PD with the reason about
resources.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">$ cat
slurm_script.sh<br>
#!/bin/bash<br>
#SBATCH --output=test.out<br>
#SBATCH --job-name=g09-test<br>
#SBATCH --ntasks=20<br>
#SBATCH --nodelist=compute-0-0<br>
#SBATCH --mem=40GB<br>
#SBATCH --account=z7<br>
#SBATCH --partition=EMERALD<br>
g09 test.gjf<br>
$ sbatch slurm_script.sh<br>
Submitted batch job 878<br>
$ squeue<br>
JOBID PARTITION NAME USER
ST TIME NODES NODELIST(REASON)<br>
878 EMERALD g09-test shakerza
PD 0:00 1 (Resources)<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">However, all
things look good.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">$ sacctmgr
list association
format=user,account,partition,grptres%20 | grep
shaker<br>
shakerzad+ local<br>
shakerzad+ z7 emerald
cpu=20,mem=40G<br>
$ scontrol show node compute-0-0<br>
NodeName=compute-0-0 Arch=x86_64 CoresPerSocket=1<br>
CPUAlloc=9 CPUTot=32 CPULoad=8.89<br>
AvailableFeatures=rack-0,32CPUs<br>
ActiveFeatures=rack-0,32CPUs<br>
Gres=(null)<br>
NodeAddr=10.1.1.254 NodeHostName=compute-0-0
Version=18.08<br>
OS=Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri
Oct 20 20:32:50 UTC 2017<br>
RealMemory=64261 AllocMem=56320 FreeMem=37715
Sockets=32 Boards=1<br>
State=MIXED ThreadsPerCore=1 TmpDisk=444124
Weight=20511900 Owner=N/A MCS_label=N/A<br>
Partitions=CLUSTER,WHEEL,EMERALD,QUARTZ<br>
BootTime=2019-04-06T10:03:47
SlurmdStartTime=2019-04-06T10:05:54<br>
CfgTRES=cpu=32,mem=64261M,billing=47<br>
AllocTRES=cpu=9,mem=55G<br>
CapWatts=n/a<br>
CurrentWatts=0 LowestJoules=0 ConsumedJoules=0<br>
ExtSensorsJoules=n/s ExtSensorsWatts=0
ExtSensorsTemp=n/s<br>
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">Any idea?</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div>
<div dir="ltr" class="gmail-m_-746978286124028451gmail_signature">
<div dir="ltr"><font face="tahoma,sans-serif">Regards,<br>
Mahmood</font><br>
<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</blockquote>
</div>
</blockquote></div>