MIG and eBPF issues (Slurm 24.11.6)
Hi all, I'm having some trouble getting Slurm 24.11.6 to work with MIG, and the slurmd logs seem to point to an issue with eBPF. For some context, this is an LXD unprivileged container where I'm trying to get MIG to work with Slurm. Other compute nodes without MIG work fine and isolate the GPUs accordingly. What I'm seeing in slurmd logs: [2025-11-24T23:32:50.197] [331.interactive] cgroup/v2: cgroup_p_constrain_apply: CGROUP: EBPF Closing and loading bpf program into /sys/fs/cgroup/system.slice/slurmstepd.scope/job_331 [2025-11-24T23:32:50.197] [331.interactive] error: load_ebpf_prog: BPF load error (Operation not permitted). Please check your system limits (MEMLOCK). I've tried increasing the system limits for MEMLOCK by setting DefaultLimitMEMLOCK=infinity in /etc/systemd/system.conf, and I've copied my slurmd.service file below where I've set Delegate=yes and LimitMEMLOCK=infinity. Previously only Delegate=yes wasn't set (I had rifled through the cgroupv2 documentaton for Slurm and found that setting), but in both cases I see the same BPF load error. Just wondering if this was something that other people had come across before and maybe I'm doing something silly here. I've checked that my slurm.conf has the corresponding parameters set according to Slurm's own documentation for cgroup.conf and my cgroup.conf is also copied below. Some portion of the gres.conf is also copied below, and even though I tried AutoDetect=nvml for this node, it's still doesn't work, which was why I changed to manually setting it based off the output of slurmd -G. Maybe I should try switching back to cgroupv1 and see if that helps fix things, but I'm not sure at this point if MIG and Slurm are compatible using cgroupv1. I can send other parts of logs, configuration files etc. Any help would be greatly appreciated! ###### slurmd.service file [Unit] Description=Slurm node daemon After=network.target munge.service ConditionPathExists=/etc/slurm/slurm.conf [Service] Type=forking EnvironmentFile=-/etc/sysconfig/slurmd ExecStart=/usr/sbin/slurmd -d /usr/sbin/slurmstepd $SLURMD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID PIDFile=/var/run/slurmd.pid KillMode=process LimitNOFILE=51200 Delegate=yes LimitMEMLOCK=infinity LimitSTACK=infinity [Install] WantedBy=multi-user.target ###### cgroup.conf CgroupPlugin=autodetect ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ##### gres.conf NodeName=gpu-3 AutoDetect=nvml Name=gpu NodeName=gpu-4 Name=gpu MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap30,/dev/nvidia-caps/nvidia-cap31
How are you running the lxd container? Have you setup the device as a passthru to the container? I have not run lxd containers under slurm (I use apptainer/podman) but I have used lxd VMs as nodes with no issues, even using GPUs/IB cards in them. Brian Andrus On 11/26/2025 8:32 AM, wk5ng--- via slurm-users wrote:
Hi all,
I'm having some trouble getting Slurm 24.11.6 to work with MIG, and the slurmd logs seem to point to an issue with eBPF. For some context, this is an LXD unprivileged container where I'm trying to get MIG to work with Slurm. Other compute nodes without MIG work fine and isolate the GPUs accordingly.
What I'm seeing in slurmd logs: [2025-11-24T23:32:50.197] [331.interactive] cgroup/v2: cgroup_p_constrain_apply: CGROUP: EBPF Closing and loading bpf program into /sys/fs/cgroup/system.slice/slurmstepd.scope/job_331 [2025-11-24T23:32:50.197] [331.interactive] error: load_ebpf_prog: BPF load error (Operation not permitted). Please check your system limits (MEMLOCK).
I've tried increasing the system limits for MEMLOCK by setting DefaultLimitMEMLOCK=infinity in /etc/systemd/system.conf, and I've copied my slurmd.service file below where I've set Delegate=yes and LimitMEMLOCK=infinity. Previously only Delegate=yes wasn't set (I had rifled through the cgroupv2 documentaton for Slurm and found that setting), but in both cases I see the same BPF load error.
Just wondering if this was something that other people had come across before and maybe I'm doing something silly here. I've checked that my slurm.conf has the corresponding parameters set according to Slurm's own documentation for cgroup.conf and my cgroup.conf is also copied below.
Some portion of the gres.conf is also copied below, and even though I tried AutoDetect=nvml for this node, it's still doesn't work, which was why I changed to manually setting it based off the output of slurmd -G.
Maybe I should try switching back to cgroupv1 and see if that helps fix things, but I'm not sure at this point if MIG and Slurm are compatible using cgroupv1.
I can send other parts of logs, configuration files etc. Any help would be greatly appreciated!
###### slurmd.service file [Unit] Description=Slurm node daemon After=network.target munge.service ConditionPathExists=/etc/slurm/slurm.conf
[Service] Type=forking EnvironmentFile=-/etc/sysconfig/slurmd ExecStart=/usr/sbin/slurmd -d /usr/sbin/slurmstepd $SLURMD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID PIDFile=/var/run/slurmd.pid KillMode=process LimitNOFILE=51200 Delegate=yes LimitMEMLOCK=infinity LimitSTACK=infinity
[Install] WantedBy=multi-user.target
###### cgroup.conf CgroupPlugin=autodetect ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes
##### gres.conf NodeName=gpu-3 AutoDetect=nvml Name=gpu NodeName=gpu-4 Name=gpu MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap30,/dev/nvidia-caps/nvidia-cap31
Yes we have setup the NVIDIA GPUs as passthrough to the container based on LXD's documentation: https://documentation.ubuntu.com/lxd/latest/reference/devices_gpu/ We also have nvidia.runtime set to True to pass the host NVIDIA and CUDA runtime libraries into the container: https://documentation.ubuntu.com/lxd/latest/reference/instance_options/#nvid... With lxd VMs, have you had experience with MIG in the VMs? For example, it explicitly says that the MIG GPU type is not supported for VMs in LXD documentation, but at the same time, maybe I just need to pass in the GPUs, then enable MIG within the lxd VM.
On 11/26/25 11:32 am, wk5ng--- via slurm-users wrote:
I'm having some trouble getting Slurm 24.11.6 to work with MIG, and the slurmd logs seem to point to an issue with eBPF. For some context, this is an LXD unprivileged container where I'm trying to get MIG to work with Slurm. Other compute nodes without MIG work fine and isolate the GPUs accordingly.
To clarify, are you running the slurmd inside the container, or is this outside of it? Have you tried running other eBPF programs inside the container? For cgroups v2 Slurm uses eBPF as v2 removed the ability to constrain devices. All the best, Chris -- Chris Samuel : http://www.csamuel.org/ : Philadelphia, PA, USA
Slurmd is running inside the container, and no I've not tried running other eBPF programs inside the container - is there something that you would recommend trying just to see if eBPF is functional? One thing that I've tried is just running: bpftool prog show This works on the bare metal server running the LXD containers but within the containers, I had to install linux-tools-generic and linux-tools-common, then running the same command shows: root@gpu-4:~# bpftool prog show Error: can't get next program: Operation not permitted I think this possibly points to a limitation of unprivileged containers (or more likely, a misconfiguration of the container on my end due to my ignorance). For example, we added the GPUs and MIGs using lxc config device add, but maybe we also need to add lines specifying lxc.cgroup2.devices.allow based on https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html. Also possibly, I should look into privilege escalation (?) https://documentation.ubuntu.com/lxd/latest/explanation/bpf/ but I don't quite understand this yet.
participants (3)
-
Brian Andrus -
Christopher Samuel -
wk5ng＠uwaterloo.ca