<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hi All, <br>
<br>
I've run into a strange problem with my slurm configuration. Trying to set up AccountingStorage properly so that I can use OpenXDMoD for producing usage reports, but the output I'm getting from sacct only has 0's for a huge number of fields like NCPUs and CPUTimeRaw
(which are rather important for useage reports). <br>
<br>
Has anyone here run into something similar before? It would be great if someone could point out what I've mis-configured. I've pasted the relevant bits of my slurm config and sacct output after my sig. <br>
<br>
Thanks!<br>
</p>
<p><br>
</p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<font face="'Courier New',monospace">------------------------------------<br>
<font face="'Courier New',monospace">Eric Coulter jecoulte@iu.edu</font><br>
XSEDE Capabilities and Resource Integration Engineer <br>
IU Campus Bridging & Research Infrastructure<br>
RT/PTI/UITS<br>
812-856-3250</font><br>
<br>
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">jecoulte@headnode ~]$ scontrol show config | grep Acc</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageBackupHost = (null)</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageEnforce = none </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageHost = headnode</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageLoc = /var/log/slurmacct.log</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStoragePort = 0</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageTRES = cpu,mem,energy,node #Added these in case the default wasn't being
respected for some reason... </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageType = accounting_storage/filetxt </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStorageUser = root</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AccountingStoreJobComment = Yes</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AcctGatherEnergyType = acct_gather_energy/none</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AcctGatherFilesystemType = acct_gather_filesystem/none</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AcctGatherInfinibandType = acct_gather_infiniband/none</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AcctGatherNodeFreq = 0 sec</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">AcctGatherProfileType = acct_gather_profile/none</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">JobAcctGatherFrequency = 30</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">JobAcctGatherType = jobacct_gather/linux</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">JobAcctGatherParams = (null)</span><br>
<br>
For a job running on 2 nodes, 1 cpu per node, sacct shows:<br>
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">[jecoulte@headnode ~]$ sudo sacct -j 386 --format JobID,JobName,AllocNodes,TotalCPU,CPUTime,NCPUS,CPUTimeRaw,AllocCPUs</span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)"> JobID JobName AllocNodes TotalCPU CPUTime NCPUS CPUTimeRAW AllocCPUS </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">386 fact_job.+ 2 00:49.345 00:00:00 0 0 0 </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">386.0 hostname 2 00:00.006 00:00:00 0 0 0 </span><br style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">
<span style="color:rgb(33,33,33); font-family:"Segoe UI","Segoe WP","Segoe UI WPC",Tahoma,Arial,sans-serif; font-size:13.3333px; background-color:rgb(255,255,255)">386.1 fact-sum.g 2 00:49.338 00:00:00 0 0 0 <br>
</span><br>
For the same job, the record in AccountingStorageLoc is:<br>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
[jecoulte@headnode ~]$ grep ^386 /var/log/slurmacct.log </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
386 low 1517006536 1517006537 1000 1000 - - 0 fact_job.job 1 4294901759 2 compute-[0-1] (null)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
386 low 1517006536 1517006537 1000 1000 - - 0 fact_job.job 1 4294901759 2 compute-[0-1] (null)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
386 low 1517006536 1517006538 1000 1000 - - 1 0 1 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0 0 0.00 0 0 0.00 0 0 0.00 hostname compute-[0-1] 0 0 0 0 (null) 4294967295</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
386 low 1517006536 1517006538 1000 1000 - - 1 0 3 0 2 2 0 0 6466 0 5388 0 1078 0 0 0 0 0 0 0 0 0 0 0 0 0 0 269148 1 236380.00 620 1 618.00 0 1 0.00 0 1 0.00 hostname compute-[0-1] 1 1 1 1 (null) 4294967295</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
386 low 1517006536 1517006537 1000 1000 - - 0 fact_job.job 1 4294901759 2 compute-[0-1] (null)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
386 low 1517006536 1517006538 1000 1000 - - 1 1 1 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0 0 0.00 0 0 0.00 0 0 0.00 fact-sum.g compute-[0-1] 0 0 0 0 (null) 4294967295</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
386 low 1517006536 1517006565 1000 1000 - - 1 1 3 0 2 2 27 49 338902 48 94477 1 244425 0 0 0 0 0 0 0 0 0 0 0 0 0 0 269148 1 236380.00 620 1 618.00 0 1 0.00 0 1 0.00 fact-sum.g compute-[0-1] 1 1 1 1 (null) 4294967295</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
386 low 1517006536 1517006537 1000 1000 - - 0 fact_job.job 1 4294901759 2 compute-[0-1] (null)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
386 low 1517006536 1517006537 1000 1000 - - 0 fact_job.job 1 4294901759 2 compute-[0-1] (null)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
386 low 1517006536 1517006565 1000 1000 - - 3 28 3 4294967295 0<br>
</div>
<br>
</div>
</div>
</body>
</html>