[slurm-users] CUDA environment variable not being set

[Sajesh Singh]

Christopher,

 Thank you for the tip. That works as expected. 


-SS-

-----Original Message-----
From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Christopher Samuel
Sent: Thursday, October 8, 2020 6:52 PM
To: slurm-users at lists.schedmd.com
Subject: Re: [slurm-users] CUDA environment variable not being set

EXTERNAL SENDER


On 10/8/20 3:48 pm, Sajesh Singh wrote:

>    Thank you. Looks like the fix is indeed the missing file 
> /etc/slurm/cgroup_allowed_devices_file.conf

No, you don't want that, that will allow all access to GPUs whether people have requested them or not.

What you want is in gres.conf and looks like (hopefully not line wrapped!):

NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia0 Cores=0,2,4,6,8 NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia1
Cores=10,12,14,16,18
NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia2
Cores=20,22,24,26,28
NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia3
Cores=30,32,34,36,38

All the best,
Chris
--
   Chris Samuel  :  https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&data=01%7C01%7Cssingh%40amnh.org%7C19c93fb5353d43eae47f08d86bdcdebd%7Cbe0003e8c6b9496883aeb34586974b76%7C0&sdata=YO9kTd3TSKG6Y2B6NHx%2B59I5rNdZGPESatncINTPC5A%3D&reserved=0  :  Berkeley, CA, USA