<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m running Slurm 15.08 and I’m having a problem I’m a bit confused about why it is happening. I have a user that is submitting asking for 64 tasks, the default system wide is 1 cpu per task. But the user is getting varied numbers of cpus
for each task, sometimes as few as 2 sometimes all 64. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is an example of one with too few resources given.<o:p></o:p></p>
<p class="MsoNormal">Output from scontrol show job, with userdata removed.<o:p></o:p></p>
<p class="MsoNormal"> Priority=266 Nice=0 Account=mechanical QOS=normal WCKey=*default<o:p></o:p></p>
<p class="MsoNormal"> JobState=COMPLETING Reason=NonZeroExitCode Dependency=(null)<o:p></o:p></p>
<p class="MsoNormal"> Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=255:0<o:p></o:p></p>
<p class="MsoNormal"> RunTime=00:01:12 TimeLimit=6-16:00:00 TimeMin=N/A<o:p></o:p></p>
<p class="MsoNormal"> SubmitTime=2018-02-21T11:24:10 EligibleTime=2018-02-21T11:24:10<o:p></o:p></p>
<p class="MsoNormal"> StartTime=2018-02-21T11:57:48 EndTime=2018-02-21T11:59:00<o:p></o:p></p>
<p class="MsoNormal"> PreemptTime=None SuspendTime=None SecsPreSuspend=0<o:p></o:p></p>
<p class="MsoNormal"> ReqNodeList=(null) ExcNodeList=(null)<o:p></o:p></p>
<p class="MsoNormal"> NumNodes=1 NumCPUs=38 CPUs/Task=1 ReqB:S:C:T=0:0:*:*<o:p></o:p></p>
<p class="MsoNormal"> TRES=cpu=64,mem=224000,node=2<o:p></o:p></p>
<p class="MsoNormal"> Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*<o:p></o:p></p>
<p class="MsoNormal"> MinCPUsNode=1 MinMemoryCPU=3500M MinTmpDiskNode=0<o:p></o:p></p>
<p class="MsoNormal"> Features=(null) Gres=(null) Reservation=(null)<o:p></o:p></p>
<p class="MsoNormal"> Shared=OK Contiguous=0 Licenses=(null) Network=(null)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Excerpt from the slurmctld log where the job started…<o:p></o:p></p>
<p class="MsoNormal"> [2018-02-21T11:57:48.305] sched: Allocate JobID=882939 NodeList=edrcompute-22-12,edrcompute-43-3 #CPUs=64 Partition=free<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">That seems to suggest that it should have gotten 64 cpus, but the scontrol output, and the fact that the job fails because there aren’t the right number of threads for mpi to run on seems to suggest that it is indeed getting the number
of cpus listed in scontrol not the number suggested by the tres, or the scheduler log. Here is the head of the user’s submission file it’s pretty basic but it should give them 64 tasks, which should give them 64 cpus.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#!/bin/bash<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -J Vasp <o:p></o:p></p>
<p class="MsoNormal">#SBATCH --ntasks=64<o:p></o:p></p>
<p class="MsoNormal">#SBATCH --time=160:00:00<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for your input,<o:p></o:p></p>
<p class="MsoNormal">Buddy Scharfenberg.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>