Nodes TRES double what is requested
Hi, We are running slurm 23.02.6. Our nodes have hyperthreading disabled and we have slurm.conf set to CPU=32 for each node (each node has 2 processes with 16 cores). When we allocated a job, such as salloc -n 32, it will allocate a whole node but using sinfo shows double the allocation in the TRES=64. It also shows in sinfo that the node has 4294967264 idle CPUs. Not sure if its a known bug, or an issue with our config? I have tried various things, like setting the sockets/boards in slurm.conf. Thanks Jack
Jack, To make sure things are set right, run 'slurmd -C' on the node and use that output in your config. It can also give you insight as to what is being seen on the node versus what you may expect. Brian Andrus On 7/10/2024 1:25 AM, jack.mellor--- via slurm-users wrote:
Hi,
We are running slurm 23.02.6. Our nodes have hyperthreading disabled and we have slurm.conf set to CPU=32 for each node (each node has 2 processes with 16 cores). When we allocated a job, such as salloc -n 32, it will allocate a whole node but using sinfo shows double the allocation in the TRES=64. It also shows in sinfo that the node has 4294967264 idle CPUs.
Not sure if its a known bug, or an issue with our config? I have tried various things, like setting the sockets/boards in slurm.conf.
Thanks Jack
Hint: round down a bit the RAM reported by 'slurmd -C'. Or you risk the nodes not coming back up after an upgrade that leaves a bit less free RAM than configured. Diego Il 10/07/2024 17:29, Brian Andrus via slurm-users ha scritto:
Jack,
To make sure things are set right, run 'slurmd -C' on the node and use that output in your config.
It can also give you insight as to what is being seen on the node versus what you may expect.
Brian Andrus
On 7/10/2024 1:25 AM, jack.mellor--- via slurm-users wrote:
Hi,
We are running slurm 23.02.6. Our nodes have hyperthreading disabled and we have slurm.conf set to CPU=32 for each node (each node has 2 processes with 16 cores). When we allocated a job, such as salloc -n 32, it will allocate a whole node but using sinfo shows double the allocation in the TRES=64. It also shows in sinfo that the node has 4294967264 idle CPUs.
Not sure if its a known bug, or an issue with our config? I have tried various things, like setting the sockets/boards in slurm.conf.
Thanks Jack
-- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Not sure if this is correct but I think you need to leave a bit of RAM for the OS to use so best not to allow slurm to allocate ALL of it. I usually take 8G off to allow for that - negligible when our nodes have at least 768GB of RAM. At least this is my experience when using cgroups. Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation ________________________________ From: Diego Zuccato via slurm-users <slurm-users@lists.schedmd.com> Sent: 11 July 2024 08:06 To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com> Subject: [slurm-users] Re: Nodes TRES double what is requested Hint: round down a bit the RAM reported by 'slurmd -C'. Or you risk the nodes not coming back up after an upgrade that leaves a bit less free RAM than configured. Diego Il 10/07/2024 17:29, Brian Andrus via slurm-users ha scritto:
Jack,
To make sure things are set right, run 'slurmd -C' on the node and use that output in your config.
It can also give you insight as to what is being seen on the node versus what you may expect.
Brian Andrus
On 7/10/2024 1:25 AM, jack.mellor--- via slurm-users wrote:
Hi,
We are running slurm 23.02.6. Our nodes have hyperthreading disabled and we have slurm.conf set to CPU=32 for each node (each node has 2 processes with 16 cores). When we allocated a job, such as salloc -n 32, it will allocate a whole node but using sinfo shows double the allocation in the TRES=64. It also shows in sinfo that the node has 4294967264 idle CPUs.
Not sure if its a known bug, or an issue with our config? I have tried various things, like setting the sockets/boards in slurm.conf.
Thanks Jack
-- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786 -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
On 2024/07/10 16:25, jack.mellor--- via slurm-users wrote:
We are running slurm 23.02.6. Our nodes have hyperthreading disabled and we have slurm.conf set to CPU=32 for each node (each node has 2 processes with 16 cores). When we allocated a job, such as salloc -n 32, it will allocate a whole node but using sinfo shows double the allocation in the TRES=64. It also shows in sinfo that the node has 4294967264 idle CPUs.
What does an scontrol show node tell you about the node(s) On our systems, where, sadly, our vendor is unable/unwilling to turn off SMT/hyperthreading, we see (not all fields shown), for a fully allocated, AMD EPYC 7763: so 128 physical core, node CoresPerSocket=64 CPUAlloc=256 CPUEfctv=256 CPUTot=256 Sockets=2 Boards=1 ThreadsPerCore=2 CfgTRES=cpu=256 AllocTRES=cpu=256 so I guess the question would be, depending on exactly what you see, have you explictly set, or tried setting, ThreadsPerCore=1 in the config. -- Supercomputing Systems Administrator Pawsey Supercomputing Centre SMS: +61 4 7497 6266 Eml: kevin.buckley@pawsey.org.au
participants (5)
-
Brian Andrus -
Diego Zuccato -
Emyr James -
jack.mellor＠mbda.co.uk -
Kevin Buckley