<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">We are running slurm 20.11.2-1 from CentOS 7 rpms.<div class=""><br class=""></div><div class="">The queue is set up to allow OverSubscribe:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><font face="Menlo" class="">NodeName=ne[04-09] CPUs=32 Sockets=2 CoresPerSocket=16 ThreadsPerCore=1 State=UNKNOWN</font></span></div></div><div class=""><span style="font-size: 11px;" class=""><font face="Menlo" class="">PartitionName=neon-noSMT Nodes=ne[04-09] Default=NO MaxTime=3-00:00:00 DefaultTime=4:00:00 State=UP OverSubscribe=YES</font></span></div><div class=""><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">I requested a user submit the first job:</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><div class=""><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;">#SBATCH --partition=neon-noSMT</div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#SBATCH --job-name="ns072"</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#SBATCH --nodes=1</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#SBATCH --ntasks=8</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#SBATCH --time=24:00:00</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#SBATCH --exclusive</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#SBATCH --error=ns072.err</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#SBATCH --output=ns072.out</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#SBATCH --mail-type=ALL # NONE, BEGIN, END, FAIL, REQUEUE, ALL</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#SBATCH --<a href="mailto:mail-user=tgjenkins@txcorp.com" class="">mail-user=u</a><a href="mailto:ser@corp.com" class="">ser@corp.com</a></span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><br class=""></div><div class=""><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">I requested the user submit the second job using the same SBATCH</span></div></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">commands as above, but adding:</span></div><div class=""><br class=""></div></div></span></div>
<div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#SBATCH —-oversubscribe</span></div></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">and the command to run the second job on the same node</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;"><span class="" style="font-variant-ligatures: no-common-ligatures;">as the first job:</span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><span class="" style="font-variant-ligatures: no-common-ligatures;">sbatch —nodelist={node running first job} </span><span class="" style="font-family: Menlo; font-size: 11px;">run.sbatch</span></span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><span class="" style="font-family: Menlo; font-size: 11px;"><br class=""></span></span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><span class="" style="font-family: Menlo; font-size: 11px;"><span style="font-family: Helvetica; font-size: 12px;" class="">Note each job only uses 8 ntasks/cores, out of 32 available.</span></span></span></div><div class=""><br class=""></div><div class=""><font face="Menlo" class=""><span style="font-size: 11px;" class="">When he submits the second job, the first job slows</span></font></div><div class=""><font face="Menlo" class=""><span style="font-size: 11px;" class="">down to 300x slower.</span></font></div><div class=""><font face="Menlo" class=""><span style="font-size: 11px;" class=""><br class=""></span></font></div><div class=""><font face="Menlo" class=""><span style="font-size: 11px;" class="">If I login to the node running the 2 jobs, only the top 8 </span></font></div><div class=""><font face="Menlo" class=""><span style="font-size: 11px;" class="">cores/ntasks </span></font><span style="font-size: 11px; font-family: Menlo;" class="">are being used, not 8 for each job.</span></div><div class=""><span style="font-size: 11px; font-family: Menlo;" class=""><br class=""></span></div><div class=""><font face="Menlo" class=""><span style="font-size: 11px;" class="">These are the SCHEDULING parameters from /etc/slurm/slurm.conf:</span></font></div><div class=""><font face="Menlo" class=""><span style="font-size: 11px;" class=""><br class=""></span></font></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""># SCHEDULING</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""># out 29Dec20 </span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">#FastSchedule=1</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">SchedulerType=sched/backfill</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">SelectType=select/linear</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">SelectTypeParameters=CR_ONE_TASK_PER_CORE</span></div></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Is there a different parameter I should be looking at?</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Thanks in advance,</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Anne Hammond</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><br class=""></div><div class=""><br class=""></div></body></html>