Hi,
I am looking at that useful documentation https://slurm.schedmd.com/cpu_management.html
We have people complaining about memory performance issues while running highly distributed jobs in a shared HPC cluster environment. After looking into it, we saw that is either because of concurrent access to the memory with other users, or because we have jump over memory from a socket to another (in a Numa architecture).
Until now, our "easy" answer was to use "exclusive" mode in sbatch, to ensure avoiding any concurrent memory access, and using numactl to check if the job is bound to the other socket.
However, I am looking to a better solution, as exclusive mode took all of a big node (we have only big nodes...), and numactl just check if we will have poor performances or not.
I checked the slurm.conf parameters, but all I saw is about modifying selectType. However, I cannot use other selectType as we have GPU, so we are using cons_tres. Moreover, we already have a very complicated Slurm configuration, and I would like to avoid any side effect.
I am thinking about talking to our user about the `--cpu-bind` (https://slurm.schedmd.com/srun.html#OPT_cpu-bind) method, but I am not sure how to use it, and it seems to be limited to srun ... ?
From a job point of view, I know there are some programs to deal with that, like likwid or placement (but not entirely ?), but that does not seem easy to use, and a slurm would be more suitable and more generic.
Do you have any idea on how to deal with these issues ?
Thanks,
Best regards,
Rémy Dernat