[slurm-users] Musing: Can GPUs be restricted by changing ownership permissions?

[Greg Wickham]

Hi All,

We have a flotilla of GPUs all protected by cgroups.

But occasionally we have users who _must_ ssh into the node.

Of course if they ssh in, then the cgroup protection doesn’t work (yes - there’s a slurm plugin
to tie an ssh session to a cgroup, but that seems more problematic with 8-GPU nodes and
a plethora of 1 GPU jobs - during heavy use the user may not have access to the GPU they require).

Has anyone any experience with changing GPU permissions during prolog / epilogue?

thanks,

   -greg

--
Dr. Greg Wickham
Advanced Computing Infrastructure Team Lead
Advanced Computing Core Laboratory
King Abdullah University of Science and Technology
Building #1, Office #0124
greg.wickham at kaust.edu.sa +966 544 700 330
--