Good afternoon all, after having run a Slurm cluster for a long time with exclusive node usage (the DEFAULT partition set to Shared=EXCLUSIVE, for historical reasons), I have now received a request to provide individual CPU cores as well, so "shared mode" would be it. I set up a new partition with OverSubscribe=NO, changed the DEFAULT stanza to OverSubscribe=EXCLUSIVE, and things seemed to work (after I changed SelectTypeParameters to CR_Core_Memory from CR_Core - apparently nobody ever hit a memory limit before...). But - now there are jobs that don't run anymore. After staring at them for hours, I found that they would request CPUs=32 (they always did, I suspect). Our compute nodes are 2s/16c/2t, "slurmd -C" on them reports CPUs=64 Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 Moving those jobs (that claim "never to be runnable") to the new partition now appears to work, and they claim enough memory not to share their node with other jobs, but I'd like to keep them in their original partition. Where's my mistake? Am I trying to mix things that can't be mixed? Thanks, S -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~
On Mon, Oct 12, 2026 at 01:58:12PM +0200, Steffen Grunewald via slurm-users wrote:
I have now received a request to provide individual CPU cores as well, so "shared mode" would be it.
I cannot quite get what you mean by this. Are you trying to utilize more than one job on the CPU cores?
Our compute nodes are 2s/16c/2t, "slurmd -C" on them reports CPUs=64 Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2
Moving those jobs (that claim "never to be runnable") to the new partition now appears to work, and they claim enough memory not to share their node with other jobs, but I'd like to keep them in their original partition.
If you set up `OverSubscribe=EXCLUSIVE` on the default partition, and your job is requesting CPUs=32, slurm will allocate CPUs=64 (max. accroding to your configuration) regardless, because of exclusiveness.
Where's my mistake? Am I trying to mix things that can't be mixed?
I cannot say for sure untill there is a glimpse of your slurm.conf :/ Maybe there are enough job on the exclusive partitions already? With the similar setup I know, slurm config goes something like ``` PartitionName=Default OverSubscribe=NO PartitionName=Priority_special OverSubscribe=EXCLUSIVE ``` -- Monish.
On 12/10/2026 14:58, Steffen Grunewald via slurm-users wrote:
Good afternoon all,
after having run a Slurm cluster for a long time with exclusive node usage (the DEFAULT partition set to Shared=EXCLUSIVE, for historical reasons), I have now received a request to provide individual CPU cores as well, so "shared mode" would be it. That would entail setting SelectType <https://slurm.schedmd.com/slurm.conf.html#OPT_SelectType_1> to *select/cons_tres *, instead of***select/linear *which you should have used previously. That is a site global parameter, not a partition parameter. I set up a new partition with OverSubscribe=NO, changed the DEFAULT stanza to OverSubscribe=EXCLUSIVE, and things seemed to work (after I changed SelectTypeParameters to CR_Core_Memory from CR_Core - apparently nobody ever hit a memory limit before...).
I suspect some mixup here. oversubscribe refers to your current SelectType. If you select whole nodes, over subscribe allows using multiple jobs in same node to the limit of job count per partition. Oversubscribe with select/con_tres allows multiple jobs on same cpu/core (in your case cpu is half a core). That is not the same as setting select/con_tres with no oversubscribe, which is the normal mode for what you are describing - allowing Slurm to allocate at the core level rather then the node level. The move from CR_Core to CR_Core_Memory doesn't affect this and is irrelevant to the question as asked. Further, using cr_core in multi-threaded environment is self defeating. I would suggest either set the environment to single threaded, or use cr_cpu.
But - now there are jobs that don't run anymore. After staring at them for hours, I found that they would request CPUs=32 (they always did, I suspect). Our compute nodes are 2s/16c/2t, "slurmd -C" on them reports CPUs=64 Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2
Moving those jobs (that claim "never to be runnable") to the new partition now appears to work, and they claim enough memory not to share their node with other jobs, but I'd like to keep them in their original partition.
Where's my mistake? Am I trying to mix things that can't be mixed?
Thanks, S
You have to decide if the cluster should be linear (entire node allocations) or consumable resources(cpu, cpu+mem etc.) I strongly suggest referring to https://slurm.schedmd.com/cons_tres_share.html and deciding on configuration based on desired resulting behavior (last column), and working backwards from there. Best, --Dani_L.
participants (3)
-
Daniel Letai -
Monish Balasubramaniam -
Steffen Grunewald