[slurm-users] GPU devices mapping with job's cgroup in cgroups v2 using eBPF

[Mahendra Paipuri]

Hello all,

Happy new year!

We have recently upgraded the cgroups on our SLURM cluster to v2. In
cgroups v1, the interface `/devices.list` used to have the information of
which device has been attached to that particular cgroup. From my
understanding, cgroups v2 use eBPF to manage devices and so as SLURM to
manage the GPUs.

I was looking for a way to be able to programatically determine the job
cgroups to device mapping and I came across this thread (
https://bugzilla.redhat.com/show_bug.cgi?id=1717396) which has a similar
discussion in the context of VMs.

So, I have used `bpftool` to inspect the job cgroups. An example output:
```
# /tmp/bpftool cgroup list
/sys/fs/cgroup/system.slice/slurmstepd.scope/job_1956132
ID       AttachType      AttachFlags     Name
```
When I add `effective` flag, I see the attached eBPF program

```
# /tmp/bpftool cgroup list
/sys/fs/cgroup/system.slice/slurmstepd.scope/job_1956132 effective
ID       AttachType      Name
4197     cgroup_device   Slurm_Cgroup_v2
```
>From my understand, `effective` flag shows the inherited eBPF programs as
well. So, my question is at which level of cgroups the eBPF program is
attached? I tried to inspect at various levels but all of them returned
none.

Then looking into translated byte code of eBPF program, I get the following
```
# /tmp/bpftool prog dump xlated id 4197
  0: (61) r2 = *(u32 *)(r1 +0)
  1: (54) w2 &= 65535
  2: (61) r3 = *(u32 *)(r1 +0)
  3: (74) w3 >>= 16
  4: (61) r4 = *(u32 *)(r1 +4)
  5: (61) r5 = *(u32 *)(r1 +8)
  6: (55) if r2 != 0x2 goto pc+4
  7: (55) if r4 != 0xc3 goto pc+3
  8: (55) if r5 != 0x0 goto pc+2
  9: (b7) r0 = 0
 10: (95) exit
 11: (55) if r2 != 0x2 goto pc+4
 12: (55) if r4 != 0xc3 goto pc+3
 13: (55) if r5 != 0x1 goto pc+2
 14: (b7) r0 = 0
 15: (95) exit
 16: (55) if r2 != 0x2 goto pc+4
 17: (55) if r4 != 0xc3 goto pc+3
 18: (55) if r5 != 0x2 goto pc+2
 19: (b7) r0 = 0
 20: (95) exit
 21: (55) if r2 != 0x2 goto pc+4
 22: (55) if r4 != 0xc3 goto pc+3
 23: (55) if r5 != 0x3 goto pc+2
 24: (b7) r0 = 1
 25: (95) exit
 26: (b7) r0 = 1
 27: (95) exit
```
>From the output, it is clear that GPU:3 (among 0,1,2,3) is the one that is
attached to that job's cgroup.

However, I was looking for a way to dump eBPF maps that can directly
provide the major, minor numbers and permissions of device as discussed in
the comment (https://bugzilla.redhat.com/show_bug.cgi?id=1717396#c5). When
I inspect eBPF program, I dont see any maps associated.

```
# /tmp/bpftool prog list id 4197
4197: cgroup_device  name Slurm_Cgroup_v2  tag 1a261c8a913ff67c  gpl
       loaded_at 2024-01-02T08:19:56+0100  uid 0
       xlated 224B  jited 142B  memlock 4096B
```
So, my second question is how can I get a similar information as `map dump`
that can give us the device's major minor numbers directly instead of
parsing the byte code from `prog dump`?

I am still discovering the eBPF ecosystem so if I am missing something very
obvious, please let me know. I would really appreciate that.

Cheers!

Regards
Mahendra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20240107/99c47a63/attachment.htm>