[slurm-users] Memory oversubscription and sheduling

[Michael Jennings]

On Thursday, 10 May 2018, at 20:02:37 (+1000),
Chris Samuel wrote:

> For instance there's the LBNL Node Health Check (NHC) system that plugs into 
> both Slurm and Torque.
> 
> https://slurm.schedmd.com/SUG14/node_health_check.pdf
> 
> https://github.com/mej/nhc
> 
> At ${JOB-1} we would run our in-house health check from cron and write to a 
> file in /dev/shm so that all the actual Slurm health check script would do is 
> send that to Slurm (and raise an error if it was missing).    This was because 
> we used to see health checks block due to issues and so slurmd would lock up 
> running them.   Decoupling them fixed that.

I'm surprised to hear that; this is the first time I've ever heard
that in regards to SLURM.  I'd only ever heard folks complain about
TORQUE having that issue.

FWIW, due to this exact situation, LBNL NHC has a built-in feature
called "Detached Mode" that will do something very similar, but it's
all self-contained within NHC.  The foreground process will fork off a
child and then return the results from the previous run while the
child goes off, runs all the tests, and stores its results to a file
for the next health check cycle to return.

You can read more about it here:  https://github.com/mej/nhc#detached-mode

HTH,
Michael

PS:  Hi Chris!  :-D

-- 
Michael E. Jennings <mej at lanl.gov>
HPC Systems Team, Los Alamos National Laboratory
Bldg. 03-2327, Rm. 2341     W: +1 (505) 606-0605