<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Appears that the filetxt accounting just doesn't record those fields. <br>
(The man page has a rather cryptic <span style="font-size: 12pt;">"</span><span style="font-size: 12pt;">Note: The filetxt plugin records only a</span><span style="font-size: 12pt;"> limited subset of accounting information and will prevent some sacct
options from</span><span style="font-size: 12pt;"> proper operation.</span><span style="font-size: 12pt;">"
<br>
I did not expect something like NCPUs to be outside of a useful but limited subset of information...)</span></p>
<p><br>
Assume this is similar to the situation with jobcomp/filetxt as here - <a href="https://bugs.schedmd.com/show_bug.cgi?id=3229">https://bugs.schedmd.com/show_bug.cgi?id=3229</a> <br>
<br>
If anyone else runs into this issue, the fix appears to be that one should just use slurmdbd - thought I'd send this out; all the previous posts I came across with the same issue never had a resolution.<br>
<br>
Cheers,<br>
</p>
<p><br>
</p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<font face="'Courier New',monospace" style=""><font face="Calibri, Arial, Helvetica, sans-serif">------------------------------------</font><br>
<font face="'Courier New',monospace">Eric Coulter jecoulte@iu.edu</font><br>
XSEDE Capabilities and Resource Integration Engineer <br>
IU Campus Bridging & Research Infrastructure<br>
RT/PTI/UITS<br>
</font><font face="Courier New, monospace"><a href="https://www.xsede.org/ecosystem/xcri-mission" id="NoLP">https://www.xsede.org/ecosystem/xcri-mission</a></font>
</div>
</div>
<div dir="ltr" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Coulter, John Eric <jecoulte@iu.edu><br>
<b>Sent:</b> Tuesday, January 30, 2018 3:13 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com<br>
<b>Subject:</b> [slurm-users] Slurm accounting problem - NCPUs=0</font>
<div> </div>
</div>
<div>
<p>Hi All, <br>
<br>
I've run into a strange problem with my slurm configuration. Trying to set up AccountingStorage properly so that I can use OpenXDMoD for producing usage reports, but the output I'm getting from sacct only has 0's for a huge number of fields like NCPUs and CPUTimeRaw
(which are rather important for useage reports). <br>
<br>
Has anyone here run into something similar before? It would be great if someone could point out what I've mis-configured. I've pasted the relevant bits of my slurm config and sacct output after my sig. <br>
<br>
Thanks!<br>
</p>
<p><br>
</p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<font face="'Courier New',monospace">------------------------------------<br>
<font face="'Courier New',monospace">Eric Coulter jecoulte@iu.edu</font><br>
XSEDE Capabilities and Resource Integration Engineer <br>
IU Campus Bridging & Research Infrastructure<br>
RT/PTI/UITS<br>
812-856-3250</font><br>
<br>
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">jecoulte@headnode ~]$ scontrol show config | grep Acc</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageBackupHost = (null)</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageEnforce = none </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageHost = headnode</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageLoc = /var/log/slurmacct.log</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStoragePort = 0</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageTRES = cpu,mem,energy,node #Added these in case the default wasn't being
respected for some reason... </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageType = accounting_storage/filetxt </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageUser = root</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStoreJobComment = Yes</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AcctGatherEnergyType = acct_gather_energy/none</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AcctGatherFilesystemType = acct_gather_filesystem/none</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AcctGatherInfinibandType = acct_gather_infiniband/none</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AcctGatherNodeFreq = 0 sec</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AcctGatherProfileType = acct_gather_profile/none</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">JobAcctGatherFrequency = 30</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">JobAcctGatherType = jobacct_gather/linux</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">JobAcctGatherParams = (null)</span><br>
<br>
For a job running on 2 nodes, 1 cpu per node, sacct shows:<br>
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">[jecoulte@headnode ~]$ sudo sacct -j 386 --format JobID,JobName,AllocNodes,TotalCPU,CPUTime,NCPUS,CPUTimeRaw,AllocCPUs</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)"> JobID JobName AllocNodes TotalCPU CPUTime NCPUS CPUTimeRAW AllocCPUS </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">386 fact_job.+ 2 00:49.345 00:00:00 0 0 0 </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">386.0 hostname 2 00:00.006 00:00:00 0 0 0 </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">386.1 fact-sum.g 2 00:49.338 00:00:00 0 0 0 <br>
</span><br>
For the same job, the record in AccountingStorageLoc is:<br>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
[jecoulte@headnode ~]$ grep ^386 /var/log/slurmacct.log </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
386 low 1517006536 1517006537 1000 1000 - - 0 fact_job.job 1 4294901759 2 compute-[0-1] (null)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
386 low 1517006536 1517006537 1000 1000 - - 0 fact_job.job 1 4294901759 2 compute-[0-1] (null)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
386 low 1517006536 1517006538 1000 1000 - - 1 0 1 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0 0 0.00 0 0 0.00 0 0 0.00 hostname compute-[0-1] 0 0 0 0 (null) 4294967295</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
386 low 1517006536 1517006538 1000 1000 - - 1 0 3 0 2 2 0 0 6466 0 5388 0 1078 0 0 0 0 0 0 0 0 0 0 0 0 0 0 269148 1 236380.00 620 1 618.00 0 1 0.00 0 1 0.00 hostname compute-[0-1] 1 1 1 1 (null) 4294967295</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
386 low 1517006536 1517006537 1000 1000 - - 0 fact_job.job 1 4294901759 2 compute-[0-1] (null)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
386 low 1517006536 1517006538 1000 1000 - - 1 1 1 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0 0 0.00 0 0 0.00 0 0 0.00 fact-sum.g compute-[0-1] 0 0 0 0 (null) 4294967295</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
386 low 1517006536 1517006565 1000 1000 - - 1 1 3 0 2 2 27 49 338902 48 94477 1 244425 0 0 0 0 0 0 0 0 0 0 0 0 0 0 269148 1 236380.00 620 1 618.00 0 1 0.00 0 1 0.00 fact-sum.g compute-[0-1] 1 1 1 1 (null) 4294967295</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
386 low 1517006536 1517006537 1000 1000 - - 0 fact_job.job 1 4294901759 2 compute-[0-1] (null)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
386 low 1517006536 1517006537 1000 1000 - - 0 fact_job.job 1 4294901759 2 compute-[0-1] (null)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
386 low 1517006536 1517006565 1000 1000 - - 3 28 3 4294967295 0<br>
</div>
<br>
</div>
</div>
</div>
</div>
</body>
</html>