[slurm-users] CUDA environment variable not being set

[Christopher Samuel]

On 10/8/20 3:48 pm, Sajesh Singh wrote:

>    Thank you. Looks like the fix is indeed the missing file /etc/slurm/cgroup_allowed_devices_file.conf

No, you don't want that, that will allow all access to GPUs whether 
people have requested them or not.

What you want is in gres.conf and looks like (hopefully not line wrapped!):

NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia0 Cores=0,2,4,6,8
NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia1 
Cores=10,12,14,16,18
NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia2 
Cores=20,22,24,26,28
NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia3 
Cores=30,32,34,36,38

All the best,
Chris
-- 
   Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA