- slurm-users - lists.schedmd.com

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 24 Jan '24

24 Jan '24

Hi, I just tested with 23.02.7-1 and the issue is gone. So it seems like the patch got released. Best, Tim On 1/24/24 16:55, Stefan Fleischmann wrote: > On Wed, 24 Jan 2024 12:37:04 -0300 Cristóbal Navarro > <cristobal.navarro.g(a)gmail.com> wrote: >> Many thanks >> One question? Do we have to apply this patch (and recompile slurm i >> guess) only on the compute-node with problems? >> Also, I noticed the patch now appears as "obsolete", is that ok? > We have Slurm installed on a NFS share, so what I did was to recompile > it and then I only replaced the library lib/slurm/cgroup_v2.so. Good > enough for now, I've been planning to update to 23.11 anyway soon. > > I suppose it's marked as obsolete because the patch went into a > release. According to the info in the bug report it should have been > included in 23.02.4. > > Cheers, > Stefan > >> On Wed, Jan 24, 2024 at 9:52 AM Stefan Fleischmann <sfle(a)kth.se> >> wrote: >> >>> Turns out I was wrong, this is not a problem in the kernel at all. >>> It's a known bug that is triggered by long bpf logs, see here >>> https://bugs.schedmd.com/show_bug.cgi?id=17210 >>> >>> There is a patch included there. >>> >>> Cheers, >>> Stefan >>> >>> On Tue, 23 Jan 2024 15:28:59 +0100 Stefan Fleischmann <sfle(a)kth.se> >>> wrote: >>>> I don't think there is much for SchedMD to do. As I said since it >>>> is working fine with newer kernels there doesn't seem to be any >>>> breaking change in cgroup2 in general, but only a regression >>>> introduced in one of the latest updates in 5.15. >>>> >>>> If Slurm was doing something wrong with cgroup2, and it >>>> accidentally worked until this recent change, then other kernel >>>> versions should show the same behavior. But as far as I can tell >>>> it still works just fine with newer kernels. >>>> >>>> Cheers, >>>> Stefan >>>> >>>> On Tue, 23 Jan 2024 15:20:56 +0100 >>>> Tim Schneider <tim.schneider1(a)tu-darmstadt.de> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have filed a bug report with SchedMD >>>>> (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the >>>>> support told me they cannot invest time in this issue since I >>>>> don't have a support contract. Maybe they will look into it >>>>> once it affects more people or someone important enough. >>>>> >>>>> So far, I have resorted to using 5.15.0-89-generic, but I am >>>>> also a bit concerned about the security aspect of this choice. >>>>> >>>>> Best, >>>>> >>>>> Tim >>>>> >>>>> On 23.01.24 14:59, Stefan Fleischmann wrote: >>>>>> Hi! >>>>>> >>>>>> I'm seeing the same in our environment. My conclusion is that >>>>>> it is a regression in the Ubuntu 5.15 kernel, introduced with >>>>>> 5.15.0-90-generic. Last working kernel version is >>>>>> 5.15.0-89-generic. I have filed a bug report here: >>>>>> https://bugs.launchpad.net/bugs/2050098 >>>>>> >>>>>> Please add yourself to the affected users in the bug report >>>>>> so it hopefully gets more attention. >>>>>> >>>>>> I've tested with newer kernels (6.5, 6.6 and 6.7) and the >>>>>> problem does not exist there. 6.5 is the latest hwe kernel >>>>>> for 22.04 and would be an option for now. Reverting back to >>>>>> 5.15.0-89 would work as well, but I haven't looked into the >>>>>> security aspects of that. >>>>>> >>>>>> Cheers, >>>>>> Stefan >>>>>> >>>>>> On Mon, 22 Jan 2024 13:31:15 -0300 >>>>>> cristobal.navarro.g at gmail.com wrote: >>>>>> >>>>>>> Hi Tim and community, >>>>>>> We are currently having the same issue (cgroups not working >>>>>>> it seems, showing all GPUs on jobs) on a GPU-compute node >>>>>>> (DGX A100) a couple of days ago after a full update (apt >>>>>>> upgrade). Now whenever we launch a job for that partition, >>>>>>> we get the error message mentioned by Tim. As a note, we >>>>>>> have another custom GPU-compute node with L40s, on a >>>>>>> different partition, and that one works fine. Before this >>>>>>> error, we always had small differences in kernel version >>>>>>> between nodes, so I am not sure if this can be the problem. >>>>>>> Nevertheless, here is the info of our nodes as well. >>>>>>> >>>>>>> *[Problem node]* The DGX A100 node has this kernel >>>>>>> cnavarro at nodeGPU01:~$ uname -a >>>>>>> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 >>>>>>> 20:28:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> *[Functioning node]* The Custom GPU node (L40s) has this >>>>>>> kernel cnavarro at nodeGPU02:~$ uname -a >>>>>>> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 >>>>>>> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> *And the login node *(slurmctld) >>>>>>> ? ~ uname -a >>>>>>> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue >>>>>>> Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> Any ideas what we should check? >>>>>>> >>>>>>> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 >>>>>>> at tu-darmstadt.de> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am using SLURM 22.05.9 on a small compute cluster. Since I >>>>>>>> reinstalled two of our nodes, I get the following error when >>>>>>>> launching a job: >>>>>>>> >>>>>>>> slurmstepd: error: load_ebpf_prog: BPF load error (No space >>>>>>>> left on device). Please check your system limits (MEMLOCK). >>>>>>>> >>>>>>>> Also the cgroups do not seem to work properly anymore, as I >>>>>>>> am able to see all GPUs even if I do not request them, >>>>>>>> which is not the case on the other nodes. >>>>>>>> >>>>>>>> One difference I found between the updated nodes and the >>>>>>>> original nodes (both are Ubuntu 22.04) is the kernel >>>>>>>> version, which is "5.15.0-89-generic #99-Ubuntu SMP" on the >>>>>>>> functioning nodes and "5.15.0-91-generic #101-Ubuntu SMP" >>>>>>>> on the updated nodes. I could not figure out how to install >>>>>>>> the exact first kernel version on the updated nodes, but I >>>>>>>> noticed that when I reinstall 5.15.0 with this tool: >>>>>>>> https://github.com/pimlie/ubuntu-mainline-kernel.sh, the >>>>>>>> error message disappears. However, once I do that, the >>>>>>>> network driver does not function properly anymore, so this >>>>>>>> does not seem to be a good solution. >>>>>>>> >>>>>>>> Has anyone seen this issue before or is there maybe >>>>>>>> something else I should take a look at? I am also happy to >>>>>>>> just find a workaround such that I can take these nodes >>>>>>>> back online. >>>>>>>> >>>>>>>> I appreciate any help! >>>>>>>> >>>>>>>> Thanks a lot in advance and best wishes, >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> >>>>>>>> >>>> >>> >>

2 1

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Cristóbal Navarro 24 Jan '24

24 Jan '24

Many thanks One question? Do we have to apply this patch (and recompile slurm i guess) only on the compute-node with problems? Also, I noticed the patch now appears as "obsolete", is that ok? On Wed, Jan 24, 2024 at 9:52 AM Stefan Fleischmann <sfle(a)kth.se> wrote: > Turns out I was wrong, this is not a problem in the kernel at all. It's > a known bug that is triggered by long bpf logs, see here > https://bugs.schedmd.com/show_bug.cgi?id=17210 > > There is a patch included there. > > Cheers, > Stefan > > On Tue, 23 Jan 2024 15:28:59 +0100 Stefan Fleischmann <sfle(a)kth.se> > wrote: > > I don't think there is much for SchedMD to do. As I said since it is > > working fine with newer kernels there doesn't seem to be any breaking > > change in cgroup2 in general, but only a regression introduced in one > > of the latest updates in 5.15. > > > > If Slurm was doing something wrong with cgroup2, and it accidentally > > worked until this recent change, then other kernel versions should > > show the same behavior. But as far as I can tell it still works just > > fine with newer kernels. > > > > Cheers, > > Stefan > > > > On Tue, 23 Jan 2024 15:20:56 +0100 > > Tim Schneider <tim.schneider1(a)tu-darmstadt.de> wrote: > > > > > Hi, > > > > > > I have filed a bug report with SchedMD > > > (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the support > > > told me they cannot invest time in this issue since I don't have a > > > support contract. Maybe they will look into it once it affects more > > > people or someone important enough. > > > > > > So far, I have resorted to using 5.15.0-89-generic, but I am also a > > > bit concerned about the security aspect of this choice. > > > > > > Best, > > > > > > Tim > > > > > > On 23.01.24 14:59, Stefan Fleischmann wrote: > > > > Hi! > > > > > > > > I'm seeing the same in our environment. My conclusion is that it > > > > is a regression in the Ubuntu 5.15 kernel, introduced with > > > > 5.15.0-90-generic. Last working kernel version is > > > > 5.15.0-89-generic. I have filed a bug report here: > > > > https://bugs.launchpad.net/bugs/2050098 > > > > > > > > Please add yourself to the affected users in the bug report so it > > > > hopefully gets more attention. > > > > > > > > I've tested with newer kernels (6.5, 6.6 and 6.7) and the problem > > > > does not exist there. 6.5 is the latest hwe kernel for 22.04 and > > > > would be an option for now. Reverting back to 5.15.0-89 would work > > > > as well, but I haven't looked into the security aspects of that. > > > > > > > > Cheers, > > > > Stefan > > > > > > > > On Mon, 22 Jan 2024 13:31:15 -0300 > > > > cristobal.navarro.g at gmail.com wrote: > > > > > > > >> Hi Tim and community, > > > >> We are currently having the same issue (cgroups not working it > > > >> seems, showing all GPUs on jobs) on a GPU-compute node (DGX A100) > > > >> a couple of days ago after a full update (apt upgrade). Now > > > >> whenever we launch a job for that partition, we get the error > > > >> message mentioned by Tim. As a note, we have another custom > > > >> GPU-compute node with L40s, on a different partition, and that > > > >> one works fine. Before this error, we always had small > > > >> differences in kernel version between nodes, so I am not sure if > > > >> this can be the problem. Nevertheless, here is the info of our > > > >> nodes as well. > > > >> > > > >> *[Problem node]* The DGX A100 node has this kernel > > > >> cnavarro at nodeGPU01:~$ uname -a > > > >> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 > > > >> 20:28:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> *[Functioning node]* The Custom GPU node (L40s) has this kernel > > > >> cnavarro at nodeGPU02:~$ uname -a > > > >> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 > > > >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> *And the login node *(slurmctld) > > > >> ? ~ uname -a > > > >> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 > > > >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> Any ideas what we should check? > > > >> > > > >> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 at > > > >> tu-darmstadt.de> wrote: > > > >> > > > >>> Hi, > > > >>> > > > >>> I am using SLURM 22.05.9 on a small compute cluster. Since I > > > >>> reinstalled two of our nodes, I get the following error when > > > >>> launching a job: > > > >>> > > > >>> slurmstepd: error: load_ebpf_prog: BPF load error (No space left > > > >>> on device). Please check your system limits (MEMLOCK). > > > >>> > > > >>> Also the cgroups do not seem to work properly anymore, as I am > > > >>> able to see all GPUs even if I do not request them, which is not > > > >>> the case on the other nodes. > > > >>> > > > >>> One difference I found between the updated nodes and the > > > >>> original nodes (both are Ubuntu 22.04) is the kernel version, > > > >>> which is "5.15.0-89-generic #99-Ubuntu SMP" on the functioning > > > >>> nodes and "5.15.0-91-generic #101-Ubuntu SMP" on the updated > > > >>> nodes. I could not figure out how to install the exact first > > > >>> kernel version on the updated nodes, but I noticed that when I > > > >>> reinstall 5.15.0 with this tool: > > > >>> https://github.com/pimlie/ubuntu-mainline-kernel.sh, the error > > > >>> message disappears. However, once I do that, the network driver > > > >>> does not function properly anymore, so this does not seem to be > > > >>> a good solution. > > > >>> > > > >>> Has anyone seen this issue before or is there maybe something > > > >>> else I should take a look at? I am also happy to just find a > > > >>> workaround such that I can take these nodes back online. > > > >>> > > > >>> I appreciate any help! > > > >>> > > > >>> Thanks a lot in advance and best wishes, > > > >>> > > > >>> Tim > > > >>> > > > >>> > > > >>> > > > -- Cristóbal A. Navarro

2 1

slurm-config on NFS-volume
by Werf, C.G. van der (Carel) 24 Jan '24

24 Jan '24

Hi, Among other clusters, I have a simple cluster with 2 nodes, running slurm. 1 node runs : mysqld, slurmdbd, slurmctld and slurmd. The other node, only runs slurmd. Slurm config is in node1: /etc/slurm. A copy of the config is in node2:/etc/slurm. This slurm configuration runs ok. But, as I am normally used to configure on a shared location, where there is only 1 copy of the config, I tried to change the current setup by moving the slurm-config to a shared (NFS) disk. So on both nodes : /etc/slurm -> nfs-server:/shared/etc/slurm But, I have tried several ways... The slurmctld and slurmd daemons will not start when I refer to this "nfs-shared" location. Any ideas what is missing here ? Authorizations seem ok. With Regards, | Carel van der Werf | | Developer/Administrator Linux | ICT-Bèta | Department of Science | Utrecht University |

3 3

error: Couldn't find the specified plugin name for cred/munge looking at all files
by Jesse Aiton 24 Jan '24

24 Jan '24

Hello Slurm Folks, I have a weird issue where on the same server, which acts as both a controller and a node, slurmctld can’t find cred_munge.so slurmctld: debug3: Trying to load plugin /app/slurm-24.0.8/lib/slurm/cred_munge.so slurmctld: debug4: /app/slurm-24.0.8/lib/slurm/cred_munge.so: Does not exist or not a regular file. slurmctld: error: Couldn't find the specified plugin name for cred/munge looking at all files slurmctld: error: cannot open plugin directory /app/slurm-24.0.8/lib/slurm slurmctld: error: cannot find cred plugin for cred/munge slurmctld: error: cannot create cred context for cred/munge slurmctld: fatal: failed to initialize cred plugin But slurmd can: slurmd: debug3: Trying to load plugin /app/slurm-24.0.8/lib/slurm/cred_munge.so slurmd: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x180800 slurmd: cred/munge: init: Munge credential signature plugin loaded slurmd: debug3: Success. This is on Ubuntu 20.04 and happens both with Slurm 20.11.09 and 24.0.8 Thank you, Jesse # slurm.conf file generated by configurator easy.html. # Put this file on all nodes of your cluster. # See the slurm.conf man page for more information. # ClusterName=prod-cluster SlurmctldHost=controller # #MailProg=/bin/mail #MpiDefault= #MpiParams=ports=#-# ProctrackType=proctrack/cgroup ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid #SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid #SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm #SlurmdUser=root StateSaveLocation=/var/spool/slurmctld #SwitchType= TaskPlugin=task/affinity,task/cgroup # # # TIMERS #KillWait=30 #MinJobAge=300 #SlurmctldTimeout=120 #SlurmdTimeout=300 # # # SCHEDULING SchedulerType=sched/backfill SelectType=select/cons_tres # # # LOGGING AND ACCOUNTING #AccountingStorageType= #JobAcctGatherFrequency=30 #JobAcctGatherType= #SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld.log #SlurmdDebug=info SlurmdLogFile=/var/log/slurmd.log # # # COMPUTE NODES NodeName=controller CPUs=1 State=UNKNOWN NodeName=node CPUs=1 State=UNKNOWN PartitionName=prod-part Nodes=ALL Default=YES MaxTime=INFINITE State=UP

3 5

Slurm version 23.11.2 is now available
by Tim McMullan 23 Jan '24

23 Jan '24

We are pleased to announce the availability of Slurm version 23.11.2. The 23.11.2 release includes a number of fixes to stability and various bug fixes. Some notable changes include several fixes to the new scontrol reconfigure method, including one that could result in jobs getting cancelled prematurely, a couple errors that resulted in the backup slurmctld stopping on fail-back, and an issue during upgrades with older MySQL versions with a small max_allowed_packet value for sites with a large number of associations. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.2 > ========================== > -- slurmrestd - Reject single http query with multiple path requests. > -- Fix launching Singularity v4.x containers with srun --container by setting > .process.terminal to true in generated config.json when step has > pseudoterminal (--pty) requested. > -- Fix loading in dyanmic/cloud node jobs after net_cred expired. > -- Fix cgroup null path error on slurmd/slurmstepd tear down. > -- data_parser/v0.0.40 - Prevent failure if accounting is disabled, instead > issue a warning if needed data from the database can not be retrieved. > -- openapi/slurmctld - Prevent failure if accounting is disabled. > -- Prevent slurmscriptd processing delays from blocking other threads in > slurmctld while trying to launch various scripts. This is additional work > for a fix in 23.02.6. > -- Fix memory leak when receiving alias addrs from controller. > -- scontrol - Accept `scontrol token lifespan=infinite` to create tokens that > effectively do not expire. > -- Avoid errors when Slurmdb accounting disabled when '--json' or '--yaml' is > invoked with CLI commands and slurmrestd. Add warnings when query would > have populated data from Slurmdb instead of errors. > -- Fix slurmctld memory leak when running job with --tres-per-task=gres:shard:# > -- Fix backfill trying to start jobs outside of backfill window. > -- Fix oversubscription on partitions with PreemptMode=OFF. > -- Preserve node reason on power up if the node is downed or drained. > -- data_parser/v0.0.40 - Avoid aborting when invoking a not implemented > parser. > -- data_parser/v0.0.40 - Fix how nice values are parsed for job submissions. > -- data_parser/v0.0.40 - Fix regression where parsing error did not result in > invalid request being rejected. > -- Fix segfault in front-end node registration. > -- Prevent jobs using none typed gpus from being killed by the controller after > a reconfig or restart. > -- Fix deadlock situation in the dbd when adding associations. > -- Update default values of text/blob columns when updating from old mysql > versions in more situations. This improves a previous fix to handle an > uncommon case when upgrading mysql/mariadb. > -- Fix rpmbuild in openSUSE/SLES due to incorrect mariadb dependency. > -- Fix compilation on RHEL 7. > -- When upgrading the slurmdbd to 23.11, avoid generating a query to update > the association table that is larger than max_allowed_packet which would > result in an upgrade failure. > -- Fix rare deadlock when a dynamic node registers at the same time that a > once per minute background task occurs. > -- Fix build issue on 32-bit systems. > -- data_parser/v0.0.40 - Fix enumerated strings in OpenAPI specification not > have type field specified. > -- Improve scontrol show job -d information of used shared gres (shard/mps) > topology. > -- Allow Slurm to compile without MUNGE if --without-munge is used as an > argument to configure. > -- accounting_storage/mysql - Fix usage query to use new lineage column > instead of lft/rgt. > -- slurmrestd - Improve handling of missing parsers when content plugins > expect parsers not loaded. > -- slurmrestd - Correct parsing of StepIds when querying jobs. > -- slurmrestd - Improve error from parsing failures of lists. > -- slurmrestd - Improve parsing of singular values for lists. > -- accounting_storage/mysql - Fix PrivateData=User when listing associations. > -- Disable sorting of dynamic nodes to avoid issues when restarting with > heterogenous jobs that cause jobs to abort on restart. > -- Don't allow deletion of non-dynamic nodes. > -- accounting_storage/mysql - Fix issue adding partition based associations. > -- Respect non-"slurm" settings for I_MPI_HYDRA_BOOTSTRAP and HYDRA_BOOTSTRAP > and avoid injecting the --external-launcher option which will cause > mpirun/mpiexec to fail with an unexpected argument error. > -- Fix bug where scontrol hold would change node count for jobs with > implicitly defined node counts. > -- data_parser/v0.0.40 - Fix regression of support for "hold" in > job description. > -- Avoid sending KILL RPCs to unresolvable POWERING_UP and POWERED_DOWN nodes. > -- data_parser/v0.0.38 - Fix several potential NULL dereferences that could > cause slurmrestd to crash. > -- Add --gres-flags=one-task-per-sharing. Do not allow different tasks in to be > allocated shared gres from the same sharing gres. > -- Add SelectTypeParameters=ENFORCE_BINDING_GRES and ONE_TASK_PER_SHARING_GRES. > This gives default behavior for a job's --gres-flags. > -- Alter the networking code to try connecting to the backup controllers > if the DNS lookup for the primary SlurmctldHost fails. > -- Alter the name resolution to only log at verbose() in client commands > on failures. This allows for HA setups where the DNS entries are withdrawn > for some SlurmctldHost entries without flooding the user with errors. > -- Prevent slurmscriptd PID leaks when running slurmctld in foreground mode. > -- Open all slurmctld listening ports at startup, and persist throughout. > This also changes the backup slurmctld process to open the SlurmctldPort > range, instead of only the first. > -- Fix backup slurmctld shutting down instead of resuming standby duty if it > took control. > -- Fix race condition that delayed the primary slurmctld resuming when taking > control from a backup controller. > -- srun - Ensure processed messages are meant for this job in case of a > rapidly-reused TCP port. > -- srun - Prevent step launch failure while waiting for step allocation if > a stray message is received. > -- Fix backup slurmctld to be able to respond to configless config file > requests correctly. > -- Fix slurmctld crashing when recovering from a failed reconfigure. > -- Fix slurmscriptd operation after recovering from a failed reconfigure. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

GPU devices mapping with job's cgroup in cgroups v2 using eBPF
by Mahendra Paipuri 23 Jan '24

23 Jan '24

Hello all, Happy new year! We have recently upgraded the cgroups on our SLURM cluster to v2. In cgroups v1, the interface `/devices.list` used to have the information of which device has been attached to that particular cgroup. From my understanding, cgroups v2 use eBPF to manage devices and so as SLURM to manage the GPUs. I was looking for a way to be able to programatically determine the job cgroups to device mapping and I came across this thread ( https://bugzilla.redhat.com/show_bug.cgi?id=1717396) which has a similar discussion in the context of VMs. So, I have used `bpftool` to inspect the job cgroups. An example output: ``` # /tmp/bpftool cgroup list /sys/fs/cgroup/system.slice/slurmstepd.scope/job_1956132 ID AttachType AttachFlags Name ``` When I add `effective` flag, I see the attached eBPF program ``` # /tmp/bpftool cgroup list /sys/fs/cgroup/system.slice/slurmstepd.scope/job_1956132 effective ID AttachType Name 4197 cgroup_device Slurm_Cgroup_v2 ``` >From my understand, `effective` flag shows the inherited eBPF programs as well. So, my question is at which level of cgroups the eBPF program is attached? I tried to inspect at various levels but all of them returned none. Then looking into translated byte code of eBPF program, I get the following ``` # /tmp/bpftool prog dump xlated id 4197 0: (61) r2 = *(u32 *)(r1 +0) 1: (54) w2 &= 65535 2: (61) r3 = *(u32 *)(r1 +0) 3: (74) w3 >>= 16 4: (61) r4 = *(u32 *)(r1 +4) 5: (61) r5 = *(u32 *)(r1 +8) 6: (55) if r2 != 0x2 goto pc+4 7: (55) if r4 != 0xc3 goto pc+3 8: (55) if r5 != 0x0 goto pc+2 9: (b7) r0 = 0 10: (95) exit 11: (55) if r2 != 0x2 goto pc+4 12: (55) if r4 != 0xc3 goto pc+3 13: (55) if r5 != 0x1 goto pc+2 14: (b7) r0 = 0 15: (95) exit 16: (55) if r2 != 0x2 goto pc+4 17: (55) if r4 != 0xc3 goto pc+3 18: (55) if r5 != 0x2 goto pc+2 19: (b7) r0 = 0 20: (95) exit 21: (55) if r2 != 0x2 goto pc+4 22: (55) if r4 != 0xc3 goto pc+3 23: (55) if r5 != 0x3 goto pc+2 24: (b7) r0 = 1 25: (95) exit 26: (b7) r0 = 1 27: (95) exit ``` >From the output, it is clear that GPU:3 (among 0,1,2,3) is the one that is attached to that job's cgroup. However, I was looking for a way to dump eBPF maps that can directly provide the major, minor numbers and permissions of device as discussed in the comment (https://bugzilla.redhat.com/show_bug.cgi?id=1717396#c5). When I inspect eBPF program, I dont see any maps associated. ``` # /tmp/bpftool prog list id 4197 4197: cgroup_device name Slurm_Cgroup_v2 tag 1a261c8a913ff67c gpl loaded_at 2024-01-02T08:19:56+0100 uid 0 xlated 224B jited 142B memlock 4096B ``` So, my second question is how can I get a similar information as `map dump` that can give us the device's major minor numbers directly instead of parsing the byte code from `prog dump`? I am still discovering the eBPF ecosystem so if I am missing something very obvious, please let me know. I would really appreciate that. Cheers! Regards Mahendra

2 1

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 23 Jan '24

23 Jan '24

Hi, I have filed a bug report with SchedMD (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the support told me they cannot invest time in this issue since I don't have a support contract. Maybe they will look into it once it affects more people or someone important enough. So far, I have resorted to using 5.15.0-89-generic, but I am also a bit concerned about the security aspect of this choice. Best, Tim On 23.01.24 14:59, Stefan Fleischmann wrote: > Hi! > > I'm seeing the same in our environment. My conclusion is that it is a > regression in the Ubuntu 5.15 kernel, introduced with 5.15.0-90-generic. > Last working kernel version is 5.15.0-89-generic. I have filed a bug > report here: https://bugs.launchpad.net/bugs/2050098 > > Please add yourself to the affected users in the bug report so it > hopefully gets more attention. > > I've tested with newer kernels (6.5, 6.6 and 6.7) and the problem does > not exist there. 6.5 is the latest hwe kernel for 22.04 and would be an > option for now. Reverting back to 5.15.0-89 would work as well, but I > haven't looked into the security aspects of that. > > Cheers, > Stefan > > On Mon, 22 Jan 2024 13:31:15 -0300 > cristobal.navarro.g at gmail.com wrote: > >> Hi Tim and community, >> We are currently having the same issue (cgroups not working it seems, >> showing all GPUs on jobs) on a GPU-compute node (DGX A100) a couple >> of days ago after a full update (apt upgrade). Now whenever we launch >> a job for that partition, we get the error message mentioned by Tim. >> As a note, we have another custom GPU-compute node with L40s, on a >> different partition, and that one works fine. >> Before this error, we always had small differences in kernel version >> between nodes, so I am not sure if this can be the problem. >> Nevertheless, here is the info of our nodes as well. >> >> *[Problem node]* The DGX A100 node has this kernel >> cnavarro at nodeGPU01:~$ uname -a >> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 20:28:30 >> UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> *[Functioning node]* The Custom GPU node (L40s) has this kernel >> cnavarro at nodeGPU02:~$ uname -a >> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 >> UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> *And the login node *(slurmctld) >> ? ~ uname -a >> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> Any ideas what we should check? >> >> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 at >> tu-darmstadt.de> wrote: >> >>> Hi, >>> >>> I am using SLURM 22.05.9 on a small compute cluster. Since I >>> reinstalled two of our nodes, I get the following error when >>> launching a job: >>> >>> slurmstepd: error: load_ebpf_prog: BPF load error (No space left on >>> device). Please check your system limits (MEMLOCK). >>> >>> Also the cgroups do not seem to work properly anymore, as I am able >>> to see all GPUs even if I do not request them, which is not the >>> case on the other nodes. >>> >>> One difference I found between the updated nodes and the original >>> nodes (both are Ubuntu 22.04) is the kernel version, which is >>> "5.15.0-89-generic #99-Ubuntu SMP" on the functioning nodes and >>> "5.15.0-91-generic #101-Ubuntu SMP" on the updated nodes. I could >>> not figure out how to install the exact first kernel version on the >>> updated nodes, but I noticed that when I reinstall 5.15.0 with this >>> tool: https://github.com/pimlie/ubuntu-mainline-kernel.sh, the >>> error message disappears. However, once I do that, the network >>> driver does not function properly anymore, so this does not seem to >>> be a good solution. >>> >>> Has anyone seen this issue before or is there maybe something else I >>> should take a look at? I am also happy to just find a workaround >>> such that I can take these nodes back online. >>> >>> I appreciate any help! >>> >>> Thanks a lot in advance and best wishes, >>> >>> Tim >>> >>> >>>

2 1

Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres)
by Kherfani, Hafedh (Professional Services, TC) 23 Jan '24

23 Jan '24

Hello Experts, I'm a new Slurm user (so please bare with me :) ...). Recently we've deployed Slurm version 23.11 on a very simple cluster, which consists of a Master node (acting as a Login & Slurmdbd node as well), a Compute Node which has a NVIDIA HGX A100-SXM4-40GB GPU, detected as 4 x GPU's: GPU [0-4], and a Storage Array presenting/sharing the NFS disk (where users' home directories will be created as well). The problem is that I've never been able to run a simple/dummy batch script in a parallel way using the 4 GPU's. In fact, running the same command "sbatch gpu-job.sh" multiple times shows that only one single job is running, while the other jobs are in a pending state: [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 214 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 215 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 216 [slurmtest@c-a100-master test-batch-scripts]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 216 gpu gpu-job slurmtest PD 0:00 1 (None) 215 gpu gpu-job slurmtest PD 0:00 1 (Priority) 214 gpu gpu-job slurmtest PD 0:00 1 (Priority) 213 gpu gpu-job slurmtest PD 0:00 1 (Priority) 212 gpu gpu-job slurmtest PD 0:00 1 (Resources) 211 gpu gpu-job slurmtest R 0:14 1 c-a100-cn01 PS: CPU jobs (i.e. using the default debug partition, without call the GPU Gres) can be run in parallel. The issue with running parallel jobs is only seen when using the GPU's as Gres. I've tried many combinations of settings in gres.conf and slurm.conf, many (if not most) of these combinations would result in error messages in slurmctld and slurmd logs. The current gres.conf and slurm.conf contents is shown below. Even though it doesn't give errors when restarting slurmctld and slurmd services (on master and compute nodes, resp.), but as I said, it doesn't allow jobs to be executed in parallel. Batch script contents shared below as well, in order to give more clarity on what I'm trying to do: [root@c-a100-master slurm]# cat gres.conf | grep -v "^#" NodeName=c-a100-cn01 AutoDetect=nvml Name=gpu Type=A100 File=/dev/nvidia[0-3] [root@c-a100-master slurm]# cat slurm.conf | grep -v "^#" | egrep -i "AccountingStorageTRES|GresTypes|NodeName|partition" GresTypes=gpu AccountingStorageTRES=gres/gpu NodeName=c-a100-cn01 Gres=gpu:A100:4 CPUs=64 Boards=1 SocketsPerBoard=1 CoresPerSocket=32 ThreadsPerCore=2 RealMemory=515181 State=UNKNOWN PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP PartitionName=gpu Nodes=ALL MaxTime=10:0:0 [slurmtest@c-a100-master test-batch-scripts]$ cat gpu-job.sh #!/bin/bash #SBATCH --job-name=gpu-job #SBATCH --partition=gpu #SBATCH --nodes=1 #SBATCH --gpus-per-node=4 #SBATCH --gres=gpu:4 #SBATCH --tasks-per-node=1 #SBATCH --output=gpu_job_output.%j # Output file name (replaces %j with job ID) #SBATCH --error=gpu_job_error.%j # Error file name (replaces %j with job ID) hostname date sleep 40 pwd Any help on which changes need to be made to the config files (mainly slurm.conf and gres.cong) and/or the batch script, so that multiple jobs can be in a "Running" state at the same time (in parallel) ? Thanks in advance for your help ! Best regards, Hafedh Kherfani

8 10

propose environment variables SLURM_STDOUT, SLURM_STDERR, SLURM_STDIN
by urbanjost 22 Jan '24

22 Jan '24

RE: Placing the full pathname of the job stdout in an environment variable Would others find it useful if new variables were added that contained the full pathnames of the standard input, error and input files of batch jobs? ## SYNOPSIS Proposed new environment variables SLURM_STDOUT,SLURM_STDERR, and SLURM_STDIN pointing to the full pathnames of the output, error output, and input files of a task would be a useful feature for batch jobs in particular. ## PROBLEM There are cases where it is desirable to have a job process the standard files (stdin, stdout, stderr) from a batch job from within the job instead of later via an epilog or via post-processing. Just a few examples where the job might want to reference the full pathname of stdout: * copy the file to a remote machine (via scp(1), for example) * move the file from local or scratch space to permanent globally accessible storage * remove the file depending on completion status * mail the output * archive it * post-process it Slurm commands like scontrol(1) and squeue(1) can help you locate the file names. But the names returned do not always expand all the macros allowed when specifying the names, and require calling a Slurm command (which might overload Slurm if millions of jobs are submitted or cause problems if Slurm is not responding for any reason(?)). For example: #!/bin/bash # Minimalist submit script for sbatch(1) #SBATCH --nodes 1-1 --ntasks=1 --time 0-0:1:00 --chdir=/tmp #SBATCH --output=stdout.%j #SBATCH --error=stderr.%A:%a:%J:%j:%N:%n:%s:%t:%u.out # query filenames via squeue(1) export SLURM_STDOUT=$(squeue --noheader --Format=STDOUT: --job=$SLURM_JOBID) export SLURM_STDERR=$(squeue --noheader --Format=STDERR: --job=$SLURM_JOBID) # PS: Current documentation for squeue(1) does not explicitly tell the # user a null size works. # query filenames via scontrol(1) declare -x $(scontrol show job=$SLURM_JOBID|grep StdOut) declare -x $(scontrol show job=$SLURM_JOBID|grep StdErr) cat <<EOF SLURM_STDOUT=$SLURM_STDOUT SLURM_STDERR=$SLURM_STDERR StdOut=$StdOut StdErr=$StdErr EOF ls stdout.* stderr.* The resulting output shows either none or some macros are expanded by the commands (and I am not sure all commands will always return a full pathname(?)) SLURM_STDOUT=/home/urbanjs/venus/V600/stdout.%j SLURM_STDERR=/home/urbanjs/venus/V600/stderr.%A:%a:%J:%j:%N:%n:%s:%t:%u.out StdOut=/home/urbanjs/venus/V600/stdout.96 StdErr=/home/urbanjs/venus/V600/stderr.96:4294967294:%J:96:%N:%n:%s:%t:urbanjs.out stderr.96:4294967294:96:96:mercury:0:4294967294:0:urbanjs.out stdout.96 One currently available work-around would be that the user just avoid the filename macros and always specify the filenames using a standard convention (which would obviously have to be specific to a specific platform). This is error-prone as it requires the user to strictly follow a protocol where the pathnames are specified that can be overridden by command-line options on sbatch(1) and so-on. Alternatively instead of the new environment variables Slurm could have a command option that returns the fully expanded names always expanding the macros. But if you query Slurm you have to worry about scaleability. Is it OK for a 100 000, jobs to query Slurm simulataneously via squeue(1) or scontrol(1)? What happens if for external reasons the Slurm daemons are down or not responding? As a current work-around using commands like realpath(1), stat(1), ls(1),find(1),and getting pids with fuser(1), pidof(1), and ps(1) become attractive. I find using the realpath(1) command at the top of the job and $SLURM_TASK_ID works the best. realpath(1) may not be available on all platforms(?) as well as the /proc/$PID/fd/1 file, in which case the other commands might be used to the same effect but it would be nice to have a simple standard way that does not require calling a command. The proposed environment variables seem the most obvious solution. # do this early in the job in case the user changes or closes # file descriptor 1 ... export SLURM_STDOUT=$(realpath /proc/$SLURM_TASK_PID/fd/1) # create softlink OUTPUT to stdout of job ln -s -f $SLURM_STDOUT OUTPUT # now user can do things like mailx(1) or scp(1) the OUTPUT file. Other reasons I might want to know the stdout pathname are to minimize network traffic and filespace on central servers. I might specify the output of a job goes to scratch or local storage or a memory-resident file system and move it all to a location on another system or a long-term archive instead of a central area that might be filled by other jobs or that I have a quota on, etc. So since the stdout might not be easily accessible from my other sessions also having something like "scontrol write batch_stdout=$SLURM_JOBID" that shows the stdout of the job would also be useful. Note the LSF job scheduler bpeek(1) command allows for a tail(1)-like interface to the job stdout to support the stdout not being accessible from all system access points, just as described. The current situation where in typical usage the output file is often in a globally mounted area and I can just access it from other nodes with grep(1), tail(1), and so on covers many users needs in simple cluster configurations; but there are several scenarios as described above primarily encountered by people running many thousands of single-node jobs where assigning output to a local scratch device and then selectively processing at job termination is preferable. As mentioned, those users can specify such a local name with "#SBATCH --output" and then, knowing it, process it as desired but having something like $SLURM_STDOUT being the full pathname to the file is much more generic. ## SUMMARY So adding the variables creates an ability for users to reliably access the standard files of a job with a low-load scalable method; and being a SLURM_* module it could be much more reliably accessed via prolog and epilog scripts and module(1) scripts as well. ## PS: Related to this would be instead of just --chdir a --mkdir and --scratch #SBATCH option would be useful that expanded file macros that would be similar to --chdir but would make sure the directory existed (using only user privelages) and a --scratch that would do the same as --mkdir but remove the directory at job termination. urbanjs@mercury:~/SLURM/proposals$ Sent with [Proton Mail](https://proton.me/) secure email.

3 2

slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 22 Jan '24

22 Jan '24

Hi, I am using SLURM 22.05.9 on a small compute cluster. Since I reinstalled two of our nodes, I get the following error when launching a job: slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK). Also the cgroups do not seem to work properly anymore, as I am able to see all GPUs even if I do not request them, which is not the case on the other nodes. One difference I found between the updated nodes and the original nodes (both are Ubuntu 22.04) is the kernel version, which is "5.15.0-89-generic #99-Ubuntu SMP" on the functioning nodes and "5.15.0-91-generic #101-Ubuntu SMP" on the updated nodes. I could not figure out how to install the exact first kernel version on the updated nodes, but I noticed that when I reinstall 5.15.0 with this tool: https://github.com/pimlie/ubuntu-mainline-kernel.sh, the error message disappears. However, once I do that, the network driver does not function properly anymore, so this does not seem to be a good solution. Has anyone seen this issue before or is there maybe something else I should take a look at? I am also happy to just find a workaround such that I can take these nodes back online. I appreciate any help! Thanks a lot in advance and best wishes, Tim

2 1

2024

slurm-users ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users