[slurm-users] GPU Jobs with Slurm

[Abhiram Chintangal]

Hello,

I recently set up a small cluster at work using Warewulf/Slurm. Currently,
I am not able to get the scheduler to
work well with GPU's (Gres).

While slurm is able to filter by GPU type, it allocates all the GPU's on
the node. See below:

[abhiram at whale ~]$ srun --gres=gpu:p100:2 -n 1 --partition=gpu nvidia-smi
> --query-gpu=index,name --format=csv
> index, name
> 0, Tesla P100-PCIE-16GB
> 1, Tesla P100-PCIE-16GB
> 2, Tesla P100-PCIE-16GB
> 3, Tesla P100-PCIE-16GB
> [abhiram at whale ~]$ srun --gres=gpu:titanrtx:2 -n 1 --partition=gpu
> nvidia-smi --query-gpu=index,name --format=csv
> index, name
> 0, TITAN RTX
> 1, TITAN RTX
> 2, TITAN RTX
> 3, TITAN RTX
> 4, TITAN RTX
> 5, TITAN RTX
> 6, TITAN RTX
> 7, TITAN RTX
>

I am fairly new to Slurm and still figuring out my way around it. I would
really appreciate any help with this.

For your reference, I attached the slurm.conf and gres.conf files.

Best,

Abhiram

-- 

Abhiram Chintangal
QB3 Nogales Lab
Bioinformatics Specialist @ Howard Hughes Medical Institute
University of California Berkeley
708D Stanley Hall, Berkeley, CA 94720
Phone (510)666-3344
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210113/4f618cb5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: slurm.conf
Type: application/octet-stream
Size: 2791 bytes
Desc: not available
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210113/4f618cb5/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gres.conf
Type: application/octet-stream
Size: 926 bytes
Desc: not available
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210113/4f618cb5/attachment-0001.obj>