[slurm-users] CUDA environment variable not being set

[Sajesh Singh]

Yes. It is located in the /etc/slurm directory

--

-SS-

From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Brian Andrus
Sent: Thursday, October 8, 2020 5:02 PM
To: slurm-users at lists.schedmd.com
Subject: Re: [slurm-users] CUDA environment variable not being set

EXTERNAL SENDER


do you have your gres.conf on the nodes also?

Brian Andrus
On 10/8/2020 11:57 AM, Sajesh Singh wrote:
Slurm 18.08
CentOS 7.7.1908

I have 2 M500 GPUs in a compute node which is defined in the slurm.conf and gres.conf of the cluster, but if I launch a job requesting GPUs the environment variable CUDA_VISIBLE_DEVICES Is never set and I see the following messages in the slurmd.log file:

debug:  common_gres_set_env: unable to set env vars, no device files configured

Has anyone encountered this before?

Thank you,

SS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20201008/570264df/attachment.htm>