[slurm-users] Automatically cancel jobs not utilizing their GPUs

[Stephan Roth]

Hi all,

Does anyone have ideas or suggestions on how to automatically cancel 
jobs which don't utilize the GPUs allocated to them?

The Slurm version in use is 19.05.

I'm thinking about collecting GPU utilization per process on all nodes 
with NVML/nvidia-smi, update a mean value of the collected utilization 
per GPU and cancel a job if the mean value is below a to-be-defined 
threshold after a to-be-defined amount of minutes.

Thank you for any input,

Cheers,
Stephan