That would entail setting SelectType to select/cons_tres , instead of select/linear which you should have used previously. That is a site global parameter, not a partition parameter.Good afternoon all, after having run a Slurm cluster for a long time with exclusive node usage (the DEFAULT partition set to Shared=EXCLUSIVE, for historical reasons), I have now received a request to provide individual CPU cores as well, so "shared mode" would be it.
I set up a new partition with OverSubscribe=NO, changed the DEFAULT stanza to OverSubscribe=EXCLUSIVE, and things seemed to work (after I changed SelectTypeParameters to CR_Core_Memory from CR_Core - apparently nobody ever hit a memory limit before...).
I suspect some mixup here. oversubscribe refers to your current
SelectType. If you select whole nodes, over subscribe allows using
multiple jobs in same node to the limit of job count per
partition. Oversubscribe with select/con_tres allows multiple jobs
on same cpu/core (in your case cpu is half a core).
That is not the same as setting select/con_tres with no
oversubscribe, which is the normal mode for what you are
describing - allowing Slurm to allocate at the core level rather
then the node level.
The move from CR_Core to CR_Core_Memory doesn't affect this and is irrelevant to the question as asked. Further, using cr_core in multi-threaded environment is self defeating. I would suggest either set the environment to single threaded, or use cr_cpu.
You have to decide if the cluster should be linear (entire node allocations) or consumable resources(cpu, cpu+mem etc.)But - now there are jobs that don't run anymore. After staring at them for hours, I found that they would request CPUs=32 (they always did, I suspect). Our compute nodes are 2s/16c/2t, "slurmd -C" on them reports CPUs=64 Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 Moving those jobs (that claim "never to be runnable") to the new partition now appears to work, and they claim enough memory not to share their node with other jobs, but I'd like to keep them in their original partition. Where's my mistake? Am I trying to mix things that can't be mixed? Thanks, S
I strongly suggest referring to https://slurm.schedmd.com/cons_tres_share.html and deciding on configuration based on desired resulting behavior (last column), and working backwards from there.
Best,
--Dani_L.