[slurm-users] Slurm -- using GPU cards with NVLINK

[Ryan Novosielski]

I’m fairly sure that you set this up the same way you set up for a peer-to-peer setup. Here’s ours:

[root at cuda001 ~]# nvidia-smi topo --matrix
        GPU0    GPU1    GPU2    GPU3    mlx4_0  CPU Affinity
GPU0     X      PIX     SYS     SYS     PHB     0-11
GPU1    PIX      X      SYS     SYS     PHB     0-11
GPU2    SYS     SYS      X      PIX     SYS     12-23
GPU3    SYS     SYS     PIX      X      SYS     12-23
mlx4_0  PHB     PHB     SYS     SYS      X 

[root at cuda001 ~]# cat /etc/slurm/gres.conf 

…

# 2 x K80 (perceval)
NodeName=cuda[001-008] Name=gpu File=/dev/nvidia[0-1] CPUs=0-11
NodeName=cuda[001-008] Name=gpu File=/dev/nvidia[2-3] CPUs=12-23

This also seems to be related:

https://slurm.schedmd.com/SLUG19/GPU_Scheduling_and_Cons_Tres.pdf

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Sep 10, 2020, at 11:00 AM, David Baker <D.J.Baker at soton.ac.uk> wrote:
> 
> Hello,
> 
> We are installing a group of nodes which all contain 4 GPU cards. The GPUs are paired together using NVLINK as described in the matrix below. 
> 
> We are familiar with using Slurm to schedule and run jobs on GPU cards, but this is the first time we have dealt with NVLINK enabled GPUs. Could someone please advise us how to configure Slurm so that we can submit jobs to the cards and make use of the NVLINK? That is, what do we need to put in the gres.conf or slurm.conf, and how should users use the sbatch command? I presume, for example, that a user could make use of a GPU card, and potentially make use of memory on the paired card.
> 
> Best regards,
> David
> 
> [root at alpha51 ~]# nvidia-smi topo --matrix
>         GPU0    GPU1    GPU2    GPU3    CPU Affinity    NUMA Affinity
> GPU0     X      NV2     SYS     SYS     0,2,4,6,8,10    0
> GPU1    NV2      X      SYS     SYS     0,2,4,6,8,10    0
> GPU2    SYS     SYS      X      NV2     1,3,5,7,9,11    1
> GPU3    SYS     SYS     NV2      X      1,3,5,7,9,11    1