<div dir="ltr">Hello,<div><br></div><div>this is my node configuration:</div><div><br></div><div>NodeName=slurm-gpu-1 NodeAddr=192.168.0.200 Procs=16 Gres=gpu:2 State=UNKNOWN<br>NodeName=slurm-gpu-2 NodeAddr=192.168.0.124 Procs=1 Gres=gpu:0 State=UNKNOWN<br>PartitionName=gpu Nodes=slurm-gpu-1 Default=NO MaxTime=INFINITE AllowAccounts=whitelist,gpu_users State=UP<br>PartitionName=compute Nodes=slurm-gpu-1,slurm-gpu-2 Default=YES MaxTime=INFINITE AllowAccounts=whitelist State=UP<br></div><div><br></div><div><br></div><div>and this is one of the job scripts. You can see mem is set to 1M, so very minimal.</div><div><br></div><div>#!/bin/bash<br>#SBATCH -J Test1<br>#SBATCH --nodelist=slurm-gpu-1 <br>#SBATCH --mem=1M <br>#SBATCH --ntasks=1<br>#SBATCH --cpus-per-task=1 <br>#SBATCH -o /home/centos/Test1-%j.out<br>#SBATCH -e /home/centos/Test1-%j.err<br>srun sleep 60<br></div><div><br></div><div>Thanks,</div><div>Durai</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 26, 2020 at 2:49 AM Jacqueline Scoggins <<a href="mailto:jscoggins@lbl.gov">jscoggins@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">What is the variable for Oversubscribe is set for your partitions? By default Oversubscribe=No which means that none of your Cores will be shared with other jobs. With oversubscribe set to Yes or Force you should set a number after the FORCE to allow the number of jobs that can run on each core of each node in the partition. <div>Look at this page for a better understanding: <font face="arial, sans-serif"><a href="https://slurm.schedmd.com/cons_res_share.html#:~:text=OverSubscribe%3DYES-,By%20default%20same%20as%20OverSubscribe%3DNO.,the%20srun%20%2D%2Doversubscribe%20option.&text=Each%20core%20can%20be%20allocated,default%204%20jobs%20per%20core).&text=CPUs%20are%20allocated%20to%20jobs" rel="noopener noreferrer" style="box-sizing:inherit;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)" target="_blank">https://slurm.schedmd.com/cons_res_share.html#:~:text=OverSubscribe%3DYES-,By%20default%20same%20as%20OverSubscribe%3DNO.,the%20srun%20%2D%2Doversubscribe%20option.&text=Each%20core%20can%20be%20allocated,default%204%20jobs%20per%20core).&text=CPUs%20are%20allocated%20to%20jobs</a><span style="color:rgb(29,28,29);font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">.</span></font></div><div><font face="arial, sans-serif"><span style="color:rgb(29,28,29);font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><br></span></font></div><div><font face="arial, sans-serif"><span style="color:rgb(29,28,29);font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">You can also check the oversubscribe on a partition using sinfo -o "%h" option. </span></font><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px"> </span></div><div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px">sinfo -o '%P %.5a %.10h %N ' | head</span></div><div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">PARTITION AVAIL OVERSUBSCR NODELIST<span> </span></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span><br></span></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span><font face="arial, sans-serif">Look at the sinfo options for further details.</font></span></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span><font face="arial, sans-serif"><br></font></span></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span><font face="arial, sans-serif">Jackie</font></span></span></p></div>
</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 25, 2020 at 9:58 AM Durai Arasan <<a href="mailto:arasan.durai@gmail.com" target="_blank">arasan.durai@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello,<br><div><br></div><div>On our cluster we have SelectTypeParameters set to "CR_Core_Memory".</div><div><br></div><div>Under these conditions multiple jobs should be able to run on the same node. But they refuse to be allocated on the same node and only one job runs on the node and rest of the jobs are in pending state.</div><div><br></div><div>When we changed SelectTypeParameters to "CR_Core" however, this issue was resolved and multiple jobs were successfully allocated to the same node and ran concurrently on the same node.</div><div><br></div><div>Does anyone know why such behavior is seen? Why does including memory as consumable resource lead to node exclusive behavior?</div><div><br></div><div>Thanks,</div><div>Durai</div><div><br></div></div>
</blockquote></div>
</blockquote></div>