[slurm-users] How does cgroups limit user access to GPUs?

[Randall Radmer]

Yes, I was just testing that.  Adding "Delegate=yes" seems to fix the
problem (see below), but wanted to try a few more things before saying
anything.

[computelab-136:~]$ grep ^Delegate /etc/systemd/system/slurmd.service
Delegate=yes
[computelab-136:~]$ nvidia-smi --query-gpu=index,name --format=csv
index, name
0, Tesla T4
[computelab-136:~]$ sudo systemctl daemon-reload; sudo systemctl restart
slurmd
[computelab-136:~]$ nvidia-smi --query-gpu=index,name --format=csv
index, name
0, Tesla T4



On Thu, Apr 11, 2019 at 7:53 AM Marcus Wagner <wagner at itc.rwth-aachen.de>
wrote:

> Hi Randall,
>
> could you please for a test add the following lines to the service part of
> the slurmd.service file (or add an override file).
>
> Delegate=yes
>
>
> Best
> Marcus
>
>
>
> On 4/11/19 3:11 PM, Randall Radmer wrote:
>
> It's now distressingly simple to reproduce this, based on Kilinan's clue
> (off topic, "Kilinan's Clue" sounds like a good title for a Hardy Boys
> Mystery Story).
>
> After limited testing, seems to me that running "systemctl
> daemon-reload"  followed by "systemctl restart slurmd" breaks it.  See
> below:
>
> [computelab-305:~]$ sudo systemctl restart slurmd
> [computelab-305:~]$ nvidia-smi --query-gpu=index,name --format=csv
> index, name
> 0, Tesla T4
> [computelab-305:~]$ sudo systemctl daemon-reload
> [computelab-305:~]$ nvidia-smi --query-gpu=index,name --format=csv
> index, name
> 0, Tesla T4
> [computelab-305:~]$ sudo systemctl restart slurmd
> [computelab-305:~]$ nvidia-smi --query-gpu=index,name --format=csv
> index, name
> 0, Tesla T4
> 1, Tesla T4
> 2, Tesla T4
> 3, Tesla T4
> 4, Tesla T4
> 5, Tesla T4
> 6, Tesla T4
> 7, Tesla T4
> [computelab-305:~]$ slurmd -V
> slurm 17.11.9-2
>
>
> On Wed, Apr 10, 2019 at 3:59 PM Kilian Cavalotti <
> kilian.cavalotti.work at gmail.com> wrote:
>
>> Hi Randy!
>>
>> > We have a slurm cluster with a number of nodes, some of which have more
>> than one GPU.  Users select how many or which GPUs they want with srun's
>> "--gres" option.  Nothing fancy here, and in general this works as
>> expected.  But starting a few days ago we've had problems on one machine.
>> A specific user started a single-gpu session with srun, and nvidia-smi
>> reported one GPU, as expected.  But about two hours later, he suddenly
>> could see all GPUs with nvidia-smi.  To be clear, this is all from the
>> iterative session provided by Slurm.  He did not ssh to the machine.  He's
>> not running Docker.  Nothing odd as far as we can tell.
>> >
>> > A big problem is I've been unable to reproduce the problem.  I have
>> confidence that what this user is telling me is correct, but I can't do
>> much until/unless I can reproduce it.
>>
>> I think this kind of behavior has already been reported a few times:
>> https://lists.schedmd.com/pipermail/slurm-users/2018-April/000885.html
>> https://bugs.schedmd.com/show_bug.cgi?id=5300
>>
>> As far as I can tell, it looks like this is probably systemd messing
>> up with cgroups and deciding it's the king of cgroups on the host.
>>
>> You'll find more context and details in
>> https://bugs.schedmd.com/show_bug.cgi?id=5292
>>
>> Cheers,
>> --
>> Kilian
>>
>>
> --
> Marcus Wagner, Dipl.-Inf.
>
> IT Center
> Abteilung: Systeme und Betrieb
> RWTH Aachen University
> Seffenter Weg 23
> 52074 Aachen
> Tel: +49 241 80-24383
> Fax: +49 241 80-624383wagner at itc.rwth-aachen.dewww.itc.rwth-aachen.de
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20190411/1564cd99/attachment-0001.html>