[slurm-users] serious bug about CUDA_VISBLE_DEVICES in the slurm 17.11.7

[Chaofeng Zhang]

The CUDA_VISBLE_DEVICES can't be set NoDevFiles in Slurm 17.11.7.  This is worked when we use Slurm 17.02.

Slurm 17.02:
[root at head ~]# export CUDA_VISIBLE_DEVICES=0,1
[root at head ~]# srun -N1 -n1 --gres=none --nodelist=head  /usr/bin/env|grep CUDA
CUDA_HOME=/usr/local/cuda
CUDA_VISIBLE_DEVICES=NoDevFiles

Slurm 17.11.7:
[root at head ~]# export CUDA_VISIBLE_DEVICES=0,1
[root at head ~]# srun -N1 -n1 --gres=none  --nodelist=head /usr/bin/env|grep CUDA
CUDA_HOME=/usr/local/cuda
CUDA_VISIBLE_DEVICES=0,1

Even it is not work pass env through export.
[root at head ~]# export CUDA_VISIBLE_DEVICES=0,1
[root at head ~]# srun -N1 -n1 --gres=none --nodelist=head --export=All,CUDA_VISIBLE_DEVICES=NoDevFiles /usr/bin/env|grep CUDA
CUDA_HOME=/usr/local/cuda
CUDA_VISIBLE_DEVICES=0,1