Still learning about SLURM, so please forgive me if I ask a naïve question

 

I like to use Anders Halager’s gnodes command to visualise the state of our nodes.  I’ve noticed lately that we fairly often see things like this (apologies for line wrap):

 

+- core - 46 cores & 186GB -----------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+

| seskscpn301    0G  ___OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO | seskscpn309  172G  ..........................................!!!! | seskscpn317    0G  OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO |

 

Now, you can see in this that nodes 301 and 317 are more or less fully loaded.  This is great.  But 309 is in an interesting state.  Four overloaded cores, and all other cores unused, and plenty of RAM available.

 

And yet SLURM is not scheduling any more work to that node.  Right now there are more than 2000 jobs pending, many of which could run on that node.  But SLURM is not scheduling them, and I don’t know why.

 

One thing I’ve seen cause this is a job trying to use more CPUs than it has been allocated.  The cgroup stops this being a real problem of course, but it does cause the load average to go high.  Is this what’s causing SLURM to stop sending anything to the node?  Is there a configuration change that might help in this situation?

 

Thanks in advance,

 

Tim