[slurm-users] Node Specific Core Distribution

[Sam Gallop (NBI)]

Hi All,

I'm hoping that someone many have encountered this scenario and knows of a solution. Basically we wish to change the default core distribution but only for specific compute nodes. The current default distribution is cyclic, but for specific nodes we would like to override this behaviour to be a default block:block.

We're currently using Slurm 17.11.12. I'm aware that --distribution (-m) can set at a job level but this relies on the user setting it and I would prefer not to be dependent on that. I'm also aware that I can set CR_CORE_DEFAULT_DIST_BLOCK via slurm.conf but this is a global change. I'm not aware of a node specific setting.
I've tried a couple of methods using SPANK and LUA (LUA would be our preferred mechanism). With both SPANK and LUA I'm able to set SBATCH_DISTRIBUTION to block:block. With LUA I'm setting the SBATCH_DISTRIBUTION via the job_desc environment variable, as SBATCH_DISTRIBUTION doesn't appear to be directly available via job_desc.distribution. With SPANK the SLURM_DISTRIBUTION output variable doesn't appear to get set, whilst with LUA I'm able to see the SLURM_DISTRIBUTION getting populated with block:block. But even though SLURM_DISTRIBUTION is set the scheduler doesn't appear to honour the request as I still end up a cyclic distribution.

Has anyone encountered this before? Can anyone suggest an alternative solution to achieve the end?

---
Samuel Gallop
Computing infrastructure for Science
CiS Support & Development

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20190426/942617c4/attachment.html>