<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:14pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hi Chance,<br>
</p>
<p><br>
</p>
<p>Can you check your slurm.conf's TaskPlugin and TaskPluginParam or cgroup settings. The tasks may not even be constrained to a group of cores.<br>
</p>
<p><br>
</p>
<p>The <span style="color: rgb(33, 33, 33); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 18.6667px; background-color: rgb(255, 255, 255);">00:02:16</span><span style="color: rgb(33, 33, 33); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 18.6667px; background-color: rgb(255, 255, 255);"> core-walltime</span>
seems odd though as you've set each job for 40 cpu minutes (20 minutes * 2 cores) Are you using a debug partition with restricted walltimes?<br>
</p>
<p><br>
</p>
<p>Regards,<br>
</p>
<p> Sam<br>
</p>
<p><br>
</p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div class="BodyFragment">
<hr style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:center">
<div class="PlainText">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right"><span style="font-size:large">Sam Hawarden</span></div>
<font size="4">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right">Assistant Research Fellow</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right">Pathology Department</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right">Dunedin School of Medicine</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right">sam.hawarden(at)otago.ac.nz<br>
</div>
</font></div>
</div>
</div>
</div>
<div style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Chance Bryce Carl Nelson <chance-nelson@nau.edu><br>
<b>Sent:</b> Saturday, 22 December 2018 08:11<br>
<b>To:</b> slurm-users@lists.schedmd.com<br>
<b>Subject:</b> [slurm-users] [Slurm 18.08.4] sacct/seff Inaccurate usercpu on Job Arrays</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi folks,
<div><br>
</div>
<div>calling sacct with the usercpu flag enabled seems to provide cpu times far above expected values for job array indices. This is also reported by seff. For example, executing the following job script:</div>
<div>
<div>________________________________________________________</div>
<br class="gmail-Apple-interchange-newline">
</div>
<div>
<div>#!/bin/bash</div>
<div>#SBATCH --job-name=array_test </div>
<div>#SBATCH --workdir=/scratch/cbn35/bigdata </div>
<div>#SBATCH --output=/scratch/cbn35/bigdata/logs/job_%A_%a.log</div>
<div>#SBATCH --time=20:00<span style="white-space:pre"> </span> </div>
<div>#SBATCH --array=1-5</div>
<div>#SBATCH -c2</div>
<div><br>
</div>
<div>srun stress -c 2 -m 1 --vm-bytes 500M --timeout 65s</div>
</div>
<div><br>
</div>
<div>________________________________________________________</div>
<div><br>
</div>
<div>...results in the following stats:</div>
<div>
<div>________________________________________________________</div>
</div>
<div><br>
</div>
<div>
<div> JobID ReqCPUS UserCPU Timelimit Elapsed </div>
<div>------------ -------- ---------- ---------- ---------- </div>
<div>15730924_5 2 02:30:14 00:20:00 00:01:08 </div>
<div>15730924_5.+ 2 00:00.004 00:01:08 </div>
<div>15730924_5.+ 2 00:00:00 00:01:09 </div>
<div>15730924_5.0 2 02:30:14 00:01:05 </div>
<div>15730924_1 2 02:30:48 00:20:00 00:01:08 </div>
<div>15730924_1.+ 2 00:00.013 00:01:08 </div>
<div>15730924_1.+ 2 00:00:00 00:01:09 </div>
<div>15730924_1.0 2 02:30:48 00:01:05 </div>
<div>15730924_2 2 02:15:52 00:20:00 00:01:07 </div>
<div>15730924_2.+ 2 00:00.007 00:01:07 </div>
<div>15730924_2.+ 2 00:00:00 00:01:07 </div>
<div>15730924_2.0 2 02:15:52 00:01:06 </div>
<div>15730924_3 2 02:30:20 00:20:00 00:01:08 </div>
<div>15730924_3.+ 2 00:00.010 00:01:08 </div>
<div>15730924_3.+ 2 00:00:00 00:01:09 </div>
<div>15730924_3.0 2 02:30:20 00:01:05 </div>
<div>15730924_4 2 02:30:26 00:20:00 00:01:08 </div>
<div>15730924_4.+ 2 00:00.006 00:01:08 </div>
<div>15730924_4.+ 2 00:00:00 00:01:09 </div>
<div>15730924_4.0 2 02:30:25 00:01:05 </div>
</div>
<div><br>
</div>
<div>
<div>________________________________________________________</div>
<br class="gmail-Apple-interchange-newline">
</div>
<div>This is also reported by seff, with several errors to boot:</div>
<div>________________________________________________________<br class="gmail-Apple-interchange-newline">
</div>
<div><br>
</div>
<div>
<div>Use of uninitialized value $lmem in numeric lt (<) at /usr/bin/seff line 130, <DATA> line 624.</div>
<div>Use of uninitialized value $lmem in numeric lt (<) at /usr/bin/seff line 130, <DATA> line 624.</div>
<div>Use of uninitialized value $lmem in numeric lt (<) at /usr/bin/seff line 130, <DATA> line 624.</div>
<div>Job ID: 15730924</div>
<div>Array Job ID: 15730924_5</div>
<div>Cluster: monsoon</div>
<div>User/Group: cbn35/clusterstu</div>
<div>State: COMPLETED (exit code 0)</div>
<div>Nodes: 1</div>
<div>Cores per node: 2</div>
<div>CPU Utilized: 03:19:15</div>
<div>CPU Efficiency: 8790.44% of 00:02:16 core-walltime</div>
<div>Job Wall-clock time: 00:01:08</div>
<div>Memory Utilized: 0.00 MB (estimated maximum)</div>
<div>Memory Efficiency: 0.00% of 1.95 GB (1000.00 MB/core)</div>
</div>
<div><br>
</div>
<div>
<div>________________________________________________________<br class="gmail-Apple-interchange-newline">
</div>
<br class="gmail-Apple-interchange-newline">
</div>
<div><br>
</div>
<div>As far as I can tell, I don't think a two core job with an elapsed time of around one minute would have a cpu time of two hours. Could this be a configuration issue, or is it a possible bug? </div>
<div><br>
</div>
<div>More info is available on request, and any help is appreciated!</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>