[slurm-users] [External] Re: serious bug about CUDA_VISBLE_DEVICES in the slurm 17.11.7

[Chaofeng Zhang]

CUDA_VISBLE_DEVICES is used by many AI framework to determine which gpu to use, like tensorflow. So this environment is critical to us.

-----Original Message-----
From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Chris Samuel
Sent: Thursday, August 30, 2018 4:42 PM
To: slurm-users at lists.schedmd.com
Subject: [External] Re: [slurm-users] serious bug about CUDA_VISBLE_DEVICES in the slurm 17.11.7

On Thursday, 30 August 2018 6:38:08 PM AEST Chaofeng Zhang wrote:

> The CUDA_VISBLE_DEVICES can't be set NoDevFiles in Slurm 17.11.7.  
> This is worked when we use Slurm 17.02.

You probably should be using cgroups instead to constrain access to GPUs.  
Then it doesn't matter what you set CUDA_VISBLE_DEVICES to be as processes will only be able to access what they requested.

Hope that helps!
Chris
--
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC