<div dir="auto">I'd take a look at:<div dir="auto"><br><div dir="auto"><a href="https://slurm.schedmd.com/cpu_management.html#Example2">https://slurm.schedmd.com/cpu_management.html#Example2</a><br></div><div dir="auto"><br><div dir="auto"><br></div><div dir="auto">I think this might be what you want:</div><div dir="auto"><br></div><div dir="auto"><pre style="margin-top:0px;margin-bottom:1.5em;padding:12px 18px;border:0px;font-size:0.85em;line-height:1.5em;font-family:"source code pro",monospace;vertical-align:baseline;background:rgb(51,51,51);color:rgb(238,238,238)">SelectType=select/cons_res
SelectTypeParameters=CR_Core</pre></div><div dir="auto">Best,</div><div dir="auto"><br></div><div dir="auto">Steve</div><div dir="auto"><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 23, 2021, 7:35 PM Anne Hammond <<a href="mailto:hammond@txcorp.com">hammond@txcorp.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">We are running slurm 20.11.2-1 from CentOS 7 rpms.<div><br></div><div>The queue is set up to allow OverSubscribe:</div><div><br></div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><font face="Menlo">NodeName=ne[04-09] CPUs=32 Sockets=2 CoresPerSocket=16 ThreadsPerCore=1 State=UNKNOWN</font></span></div></div><div><span style="font-size:11px"><font face="Menlo">PartitionName=neon-noSMT Nodes=ne[04-09] Default=NO MaxTime=3-00:00:00 DefaultTime=4:00:00 State=UP OverSubscribe=YES</font></span></div><div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures">I requested a user submit the first job:</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">#SBATCH --partition=neon-noSMT</div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --job-name="ns072"</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --nodes=1</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --ntasks=8</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --time=24:00:00</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --exclusive</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --error=ns072.err</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --output=ns072.out</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --mail-type=ALL  # NONE, BEGIN, END, FAIL, REQUEUE, ALL</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --<a href="mailto:mail-user=tgjenkins@txcorp.com" target="_blank" rel="noreferrer">mail-user=u</a><a href="mailto:ser@corp.com" target="_blank" rel="noreferrer">ser@corp.com</a></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><br></div><div><div><span style="font-variant-ligatures:no-common-ligatures">I requested the user submit the second job using the same SBATCH</span></div></div><div><span style="font-variant-ligatures:no-common-ligatures">commands as above, but adding:</span></div><div><br></div></div></span></div>
<div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH —-oversubscribe</span></div></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">and the command to run the second job on the same node</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">as the first job:</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><span style="font-variant-ligatures:no-common-ligatures">sbatch —nodelist={node running first job} </span><span style="font-family:Menlo;font-size:11px">run.sbatch</span></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><span style="font-family:Menlo;font-size:11px"><br></span></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><span style="font-family:Menlo;font-size:11px"><span style="font-family:Helvetica;font-size:12px">Note each job only uses 8 ntasks/cores, out of 32 available.</span></span></span></div><div><br></div><div><font face="Menlo"><span style="font-size:11px">When he submits the second job, the first job slows</span></font></div><div><font face="Menlo"><span style="font-size:11px">down to 300x slower.</span></font></div><div><font face="Menlo"><span style="font-size:11px"><br></span></font></div><div><font face="Menlo"><span style="font-size:11px">If I login to the node running the 2 jobs, only the top 8 </span></font></div><div><font face="Menlo"><span style="font-size:11px">cores/ntasks </span></font><span style="font-size:11px;font-family:Menlo">are being used, not 8 for each job.</span></div><div><span style="font-size:11px;font-family:Menlo"><br></span></div><div><font face="Menlo"><span style="font-size:11px">These are the SCHEDULING parameters from /etc/slurm/slurm.conf:</span></font></div><div><font face="Menlo"><span style="font-size:11px"><br></span></font></div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># SCHEDULING</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># out 29Dec20 </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">#FastSchedule=1</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">SchedulerType=sched/backfill</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">SelectType=select/linear</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">SelectTypeParameters=CR_ONE_TASK_PER_CORE</span></div></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Is there a different parameter I should be looking at?</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Thanks in advance,</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Anne Hammond</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><br></div><div><br></div></div></blockquote></div>