[slurm-users] Hyperthreading and billing according to physical cores

[bill broadley]

 From what I can tell billing reports threads used, it does seem somewhat broken.

For instance on a 64c/128T node with hyperthreading enabled:

$ sbatch -A csc000 -t 12 -p debug -N 1 -n 64 --hint=nomultithread 
--ntasks-per-core=1 --mem=2000 --exclusive ./sleep.sh
Submitted batch job 623871
$ sbatch -A csc000 -t 12 -p debug -N 1 -n 128 --hint=multithread 
--ntasks-per-core=2 --mem=2000 --exclusive ./sleep.sh
Submitted batch job 624841
$ sacct -j 623871,624841  -o jobid,reqtres%40 | egrep -v "batch|exte"
JobID                                         ReqTRES
------------ ----------------------------------------
623871             billing=64,cpu=64,mem=2000M,node=1
624841           billing=128,cpu=128,mem=2000M,node=1

In both cases I'm using 100% of the CPUs on the node.  Yet in the first case I'm 
billed half as much.

I did verify that multiple of the first job would never schedule two jobs on the 
same node, even without --exclusive.

My node config:

NodeName=c1-[28-47,56-79] procs=128 RealMemory=256000 sockets=2 
CoresPerSocket=32 ThreadsPerCore=2 State=UNKNOWN

Has anyone got the billing to report an accurate number when running a mix of 
jobs that use 1 task per core and 2 tasks per core?  Do you just ignore the 
billing field?