- slurm-users - lists.schedmd.com

Question about CPUs and cores
by Gestió Servidors 25 Jan '24

25 Jan '24

Hi, I want to run a simple test that uses one node and four cores. Also, in my script, I execute a binary that reports me in what core is running one of the four tasks. These are my files: * submit script: #!/bin/bash #SBATCH --job-name=test_jobs # Job name #SBATCH --output=test_job_%j.out # Output file (using array index) #SBATCH --error=test_job_%j.err # Error file (using array index) #SBATCH --ntasks=4 #SBATCH --ntasks-per-core=1 #SBATCH --cpus-per-task=1 # Number of CPU cores per task echo "START : $(date)" srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & # Wait for all jobs to finish before exiting the job submission script wait echo "END : $(date)" * check_cores.c file: #include <stdio.h> #include<unistd.h> struct pstat { int pid; //process id char tcomm[256];//filename of the executable char state[2]; //state (R is running, S is sleeping, D is sleeping in an //uninterruptible wait, Z is zombie, T is traced or stopped) int ppid;// process id of the parent process int pgrp;// pgrp of the process int sid;// session id int tty_nr;// tty the process uses int tty_pgrp;// pgrp of the tty int flags;// task flags int min_flt;// number of minor faults int cmin_flt;// number of minor faults with child's int maj_flt;// number of major faults int cmaj_flt;// number of major faults with child's int utime;// user mode jiffies int stime;// kernel mode jiffies int cutime;// user mode jiffies with child's int cstime;// kernel mode jiffies with child's int priority;// priority level int nice;// nice level int num_threads;// number of threads int it_real_value;// (obsolete, always 0) int start_time;// time the process started after system boot int vsize;// virtual memory size int rss;// resident set memory size int rsslim;// current limit in bytes on the rss int start_code;// address above which program text can run int end_code;// address below which program text can run int start_stack;// address of the start of the stack int esp;// current value of ESP int eip;// current value of EIP int pending;// bitmap of pending signals int blocked;// bitmap of blocked signals int sigign;// bitmap of ignored signals int sigcatch;// bitmap of catched signals int wchan;// address where process went to sleep int i0;// (place holder) int i1;// (place holder) int exit_signal;// signal to send to parent thread on exit int task_cpu;// which CPU the task is scheduled on int rt_priority;// realtime priority int policy;// scheduling policy (man sched_setscheduler) int blkio_ticks;// time spent waiting for block IO int gtime;// guest time of the task in jiffies int cgtime;// guest time of the task children in jiffies } p ; int main() { char name[256]; char state[8]; FILE* f = fopen("/proc/self/stat", "r"); char hostname[1024]; gethostname(hostname, 1024); fscanf(f, "%d%s%s%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d", &p.pid, &p.tcomm, &p.state, &p.ppid, &p.pgrp, &p.sid, &p.tty_nr, &p.tty_pgrp, &p.flags, &p.min_flt, &p.cmin_flt, &p.maj_flt, &p.cmaj_flt, &p.utime, &p.stime, &p.cutime, &p.cstime, &p.priority, &p.nice, &p.num_threads, &p.it_real_value, &p.start_time, &p.vsize, &p.rss, &p.rsslim, &p.start_code, &p.end_code, &p.start_stack, &p.esp, &p.eip, &p.pending, &p.blocked, &p.sigign, &p.sigcatch, &p.wchan, &p.i0, &p.i1, &p.exit_signal, &p.task_cpu, &p.rt_priority, &p.policy, &p.blkio_ticks, &p.gtime, &p.cgtime); sleep (15); printf("HOST %s\n", hostname); printf("CPU %d\n", p.task_cpu); return 0; } When I run this script, I should receive an output like this: HOST clus04.hpc.local CPU 0 HOST clus04.hpc.local CPU 7 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 6 because system has scheduled my job in node "clus04.hpc.local" (a node with two 6-cores processors) and has reserved cores 0-1 of each processor. However, the real output I receive is HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 6 (and others outputs where there are one CPU repeated), so it seems SLURMs is reserving correctly four cores but, then, process is running two or more times in the same core. Slurmctld.log file in server shows this information: [...] [2024-01-25T09:38:42.867] DEBUG: JobId=206129 node clus04 vpus 1 cpus 4 [2024-01-25T09:38:42.867] ==================== [2024-01-25T09:38:42.867] JobId=206129 nhosts:1 ncpus:4 node_req:1 nodes=clus04 [2024-01-25T09:38:42.867] Node[0]: [2024-01-25T09:38:42.867] Mem(MB):0:0 Sockets:2 Cores:6 CPUs:4:0 [2024-01-25T09:38:42.867] Socket[0] Core[0] is allocated [2024-01-25T09:38:42.867] Socket[0] Core[1] is allocated [2024-01-25T09:38:42.867] Socket[1] Core[0] is allocated [2024-01-25T09:38:42.867] Socket[1] Core[1] is allocated [2024-01-25T09:38:42.867] -------------------- [2024-01-25T09:38:42.867] cpu_array_value[0]:4 reps:1 [2024-01-25T09:38:42.867] ==================== [2024-01-25T09:38:42.867] DEBUG: Dump job_resources: nhosts 1 cb 0-1,6-7 [...] [2024-01-25T09:38:58.213] _job_complete: JobId=206129 WEXITSTATUS 0 [2024-01-25T09:38:58.213] deallocate_nodes: JobId=206129 [2024-01-25T09:38:58.213] DEBUG: Dump job_resources: nhosts 1 cb 0-1,6-7 [2024-01-25T09:38:58.213] _job_complete: JobId=206129 done And slurmd.log file in node clus04 shows this information: [...] [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: step physical cores are '0-1,6-7' It seems that SLURM is reserving me correctly four cores for my job but, when inside my script I run four "srun", then each srun can freely take one of the four reserved cores, not exactly one to each task. What am I doing wrong? Thanks a lot!

1 0

Slurm version 23.11.3 is now available
by Tim McMullan 24 Jan '24

24 Jan '24

We are pleased to announce the availability of Slurm version 23.11.3. The 23.11.3 fixes a single issue related to the new Debian package support. For 23.11.2, the debian changelog was not updated correctly and would generate packages as 23.11.1. This release will correctly generate 23.11.3 packages, with no other changes. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.3 > ========================== > -- Fix debian/changelog file to reflect the correct version so the packages > are generated correctl. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 24 Jan '24

24 Jan '24

Hi, I just tested with 23.02.7-1 and the issue is gone. So it seems like the patch got released. Best, Tim On 1/24/24 16:55, Stefan Fleischmann wrote: > On Wed, 24 Jan 2024 12:37:04 -0300 Cristóbal Navarro > <cristobal.navarro.g(a)gmail.com> wrote: >> Many thanks >> One question? Do we have to apply this patch (and recompile slurm i >> guess) only on the compute-node with problems? >> Also, I noticed the patch now appears as "obsolete", is that ok? > We have Slurm installed on a NFS share, so what I did was to recompile > it and then I only replaced the library lib/slurm/cgroup_v2.so. Good > enough for now, I've been planning to update to 23.11 anyway soon. > > I suppose it's marked as obsolete because the patch went into a > release. According to the info in the bug report it should have been > included in 23.02.4. > > Cheers, > Stefan > >> On Wed, Jan 24, 2024 at 9:52 AM Stefan Fleischmann <sfle(a)kth.se> >> wrote: >> >>> Turns out I was wrong, this is not a problem in the kernel at all. >>> It's a known bug that is triggered by long bpf logs, see here >>> https://bugs.schedmd.com/show_bug.cgi?id=17210 >>> >>> There is a patch included there. >>> >>> Cheers, >>> Stefan >>> >>> On Tue, 23 Jan 2024 15:28:59 +0100 Stefan Fleischmann <sfle(a)kth.se> >>> wrote: >>>> I don't think there is much for SchedMD to do. As I said since it >>>> is working fine with newer kernels there doesn't seem to be any >>>> breaking change in cgroup2 in general, but only a regression >>>> introduced in one of the latest updates in 5.15. >>>> >>>> If Slurm was doing something wrong with cgroup2, and it >>>> accidentally worked until this recent change, then other kernel >>>> versions should show the same behavior. But as far as I can tell >>>> it still works just fine with newer kernels. >>>> >>>> Cheers, >>>> Stefan >>>> >>>> On Tue, 23 Jan 2024 15:20:56 +0100 >>>> Tim Schneider <tim.schneider1(a)tu-darmstadt.de> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have filed a bug report with SchedMD >>>>> (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the >>>>> support told me they cannot invest time in this issue since I >>>>> don't have a support contract. Maybe they will look into it >>>>> once it affects more people or someone important enough. >>>>> >>>>> So far, I have resorted to using 5.15.0-89-generic, but I am >>>>> also a bit concerned about the security aspect of this choice. >>>>> >>>>> Best, >>>>> >>>>> Tim >>>>> >>>>> On 23.01.24 14:59, Stefan Fleischmann wrote: >>>>>> Hi! >>>>>> >>>>>> I'm seeing the same in our environment. My conclusion is that >>>>>> it is a regression in the Ubuntu 5.15 kernel, introduced with >>>>>> 5.15.0-90-generic. Last working kernel version is >>>>>> 5.15.0-89-generic. I have filed a bug report here: >>>>>> https://bugs.launchpad.net/bugs/2050098 >>>>>> >>>>>> Please add yourself to the affected users in the bug report >>>>>> so it hopefully gets more attention. >>>>>> >>>>>> I've tested with newer kernels (6.5, 6.6 and 6.7) and the >>>>>> problem does not exist there. 6.5 is the latest hwe kernel >>>>>> for 22.04 and would be an option for now. Reverting back to >>>>>> 5.15.0-89 would work as well, but I haven't looked into the >>>>>> security aspects of that. >>>>>> >>>>>> Cheers, >>>>>> Stefan >>>>>> >>>>>> On Mon, 22 Jan 2024 13:31:15 -0300 >>>>>> cristobal.navarro.g at gmail.com wrote: >>>>>> >>>>>>> Hi Tim and community, >>>>>>> We are currently having the same issue (cgroups not working >>>>>>> it seems, showing all GPUs on jobs) on a GPU-compute node >>>>>>> (DGX A100) a couple of days ago after a full update (apt >>>>>>> upgrade). Now whenever we launch a job for that partition, >>>>>>> we get the error message mentioned by Tim. As a note, we >>>>>>> have another custom GPU-compute node with L40s, on a >>>>>>> different partition, and that one works fine. Before this >>>>>>> error, we always had small differences in kernel version >>>>>>> between nodes, so I am not sure if this can be the problem. >>>>>>> Nevertheless, here is the info of our nodes as well. >>>>>>> >>>>>>> *[Problem node]* The DGX A100 node has this kernel >>>>>>> cnavarro at nodeGPU01:~$ uname -a >>>>>>> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 >>>>>>> 20:28:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> *[Functioning node]* The Custom GPU node (L40s) has this >>>>>>> kernel cnavarro at nodeGPU02:~$ uname -a >>>>>>> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 >>>>>>> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> *And the login node *(slurmctld) >>>>>>> ? ~ uname -a >>>>>>> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue >>>>>>> Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> Any ideas what we should check? >>>>>>> >>>>>>> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 >>>>>>> at tu-darmstadt.de> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am using SLURM 22.05.9 on a small compute cluster. Since I >>>>>>>> reinstalled two of our nodes, I get the following error when >>>>>>>> launching a job: >>>>>>>> >>>>>>>> slurmstepd: error: load_ebpf_prog: BPF load error (No space >>>>>>>> left on device). Please check your system limits (MEMLOCK). >>>>>>>> >>>>>>>> Also the cgroups do not seem to work properly anymore, as I >>>>>>>> am able to see all GPUs even if I do not request them, >>>>>>>> which is not the case on the other nodes. >>>>>>>> >>>>>>>> One difference I found between the updated nodes and the >>>>>>>> original nodes (both are Ubuntu 22.04) is the kernel >>>>>>>> version, which is "5.15.0-89-generic #99-Ubuntu SMP" on the >>>>>>>> functioning nodes and "5.15.0-91-generic #101-Ubuntu SMP" >>>>>>>> on the updated nodes. I could not figure out how to install >>>>>>>> the exact first kernel version on the updated nodes, but I >>>>>>>> noticed that when I reinstall 5.15.0 with this tool: >>>>>>>> https://github.com/pimlie/ubuntu-mainline-kernel.sh, the >>>>>>>> error message disappears. However, once I do that, the >>>>>>>> network driver does not function properly anymore, so this >>>>>>>> does not seem to be a good solution. >>>>>>>> >>>>>>>> Has anyone seen this issue before or is there maybe >>>>>>>> something else I should take a look at? I am also happy to >>>>>>>> just find a workaround such that I can take these nodes >>>>>>>> back online. >>>>>>>> >>>>>>>> I appreciate any help! >>>>>>>> >>>>>>>> Thanks a lot in advance and best wishes, >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> >>>>>>>> >>>> >>> >>

2 1

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Cristóbal Navarro 24 Jan '24

24 Jan '24

Many thanks One question? Do we have to apply this patch (and recompile slurm i guess) only on the compute-node with problems? Also, I noticed the patch now appears as "obsolete", is that ok? On Wed, Jan 24, 2024 at 9:52 AM Stefan Fleischmann <sfle(a)kth.se> wrote: > Turns out I was wrong, this is not a problem in the kernel at all. It's > a known bug that is triggered by long bpf logs, see here > https://bugs.schedmd.com/show_bug.cgi?id=17210 > > There is a patch included there. > > Cheers, > Stefan > > On Tue, 23 Jan 2024 15:28:59 +0100 Stefan Fleischmann <sfle(a)kth.se> > wrote: > > I don't think there is much for SchedMD to do. As I said since it is > > working fine with newer kernels there doesn't seem to be any breaking > > change in cgroup2 in general, but only a regression introduced in one > > of the latest updates in 5.15. > > > > If Slurm was doing something wrong with cgroup2, and it accidentally > > worked until this recent change, then other kernel versions should > > show the same behavior. But as far as I can tell it still works just > > fine with newer kernels. > > > > Cheers, > > Stefan > > > > On Tue, 23 Jan 2024 15:20:56 +0100 > > Tim Schneider <tim.schneider1(a)tu-darmstadt.de> wrote: > > > > > Hi, > > > > > > I have filed a bug report with SchedMD > > > (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the support > > > told me they cannot invest time in this issue since I don't have a > > > support contract. Maybe they will look into it once it affects more > > > people or someone important enough. > > > > > > So far, I have resorted to using 5.15.0-89-generic, but I am also a > > > bit concerned about the security aspect of this choice. > > > > > > Best, > > > > > > Tim > > > > > > On 23.01.24 14:59, Stefan Fleischmann wrote: > > > > Hi! > > > > > > > > I'm seeing the same in our environment. My conclusion is that it > > > > is a regression in the Ubuntu 5.15 kernel, introduced with > > > > 5.15.0-90-generic. Last working kernel version is > > > > 5.15.0-89-generic. I have filed a bug report here: > > > > https://bugs.launchpad.net/bugs/2050098 > > > > > > > > Please add yourself to the affected users in the bug report so it > > > > hopefully gets more attention. > > > > > > > > I've tested with newer kernels (6.5, 6.6 and 6.7) and the problem > > > > does not exist there. 6.5 is the latest hwe kernel for 22.04 and > > > > would be an option for now. Reverting back to 5.15.0-89 would work > > > > as well, but I haven't looked into the security aspects of that. > > > > > > > > Cheers, > > > > Stefan > > > > > > > > On Mon, 22 Jan 2024 13:31:15 -0300 > > > > cristobal.navarro.g at gmail.com wrote: > > > > > > > >> Hi Tim and community, > > > >> We are currently having the same issue (cgroups not working it > > > >> seems, showing all GPUs on jobs) on a GPU-compute node (DGX A100) > > > >> a couple of days ago after a full update (apt upgrade). Now > > > >> whenever we launch a job for that partition, we get the error > > > >> message mentioned by Tim. As a note, we have another custom > > > >> GPU-compute node with L40s, on a different partition, and that > > > >> one works fine. Before this error, we always had small > > > >> differences in kernel version between nodes, so I am not sure if > > > >> this can be the problem. Nevertheless, here is the info of our > > > >> nodes as well. > > > >> > > > >> *[Problem node]* The DGX A100 node has this kernel > > > >> cnavarro at nodeGPU01:~$ uname -a > > > >> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 > > > >> 20:28:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> *[Functioning node]* The Custom GPU node (L40s) has this kernel > > > >> cnavarro at nodeGPU02:~$ uname -a > > > >> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 > > > >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> *And the login node *(slurmctld) > > > >> ? ~ uname -a > > > >> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 > > > >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> Any ideas what we should check? > > > >> > > > >> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 at > > > >> tu-darmstadt.de> wrote: > > > >> > > > >>> Hi, > > > >>> > > > >>> I am using SLURM 22.05.9 on a small compute cluster. Since I > > > >>> reinstalled two of our nodes, I get the following error when > > > >>> launching a job: > > > >>> > > > >>> slurmstepd: error: load_ebpf_prog: BPF load error (No space left > > > >>> on device). Please check your system limits (MEMLOCK). > > > >>> > > > >>> Also the cgroups do not seem to work properly anymore, as I am > > > >>> able to see all GPUs even if I do not request them, which is not > > > >>> the case on the other nodes. > > > >>> > > > >>> One difference I found between the updated nodes and the > > > >>> original nodes (both are Ubuntu 22.04) is the kernel version, > > > >>> which is "5.15.0-89-generic #99-Ubuntu SMP" on the functioning > > > >>> nodes and "5.15.0-91-generic #101-Ubuntu SMP" on the updated > > > >>> nodes. I could not figure out how to install the exact first > > > >>> kernel version on the updated nodes, but I noticed that when I > > > >>> reinstall 5.15.0 with this tool: > > > >>> https://github.com/pimlie/ubuntu-mainline-kernel.sh, the error > > > >>> message disappears. However, once I do that, the network driver > > > >>> does not function properly anymore, so this does not seem to be > > > >>> a good solution. > > > >>> > > > >>> Has anyone seen this issue before or is there maybe something > > > >>> else I should take a look at? I am also happy to just find a > > > >>> workaround such that I can take these nodes back online. > > > >>> > > > >>> I appreciate any help! > > > >>> > > > >>> Thanks a lot in advance and best wishes, > > > >>> > > > >>> Tim > > > >>> > > > >>> > > > >>> > > > -- Cristóbal A. Navarro

2 1

slurm-config on NFS-volume
by Werf, C.G. van der (Carel) 24 Jan '24

24 Jan '24

Hi, Among other clusters, I have a simple cluster with 2 nodes, running slurm. 1 node runs : mysqld, slurmdbd, slurmctld and slurmd. The other node, only runs slurmd. Slurm config is in node1: /etc/slurm. A copy of the config is in node2:/etc/slurm. This slurm configuration runs ok. But, as I am normally used to configure on a shared location, where there is only 1 copy of the config, I tried to change the current setup by moving the slurm-config to a shared (NFS) disk. So on both nodes : /etc/slurm -> nfs-server:/shared/etc/slurm But, I have tried several ways... The slurmctld and slurmd daemons will not start when I refer to this "nfs-shared" location. Any ideas what is missing here ? Authorizations seem ok. With Regards, | Carel van der Werf | | Developer/Administrator Linux | ICT-Bèta | Department of Science | Utrecht University |

3 3

error: Couldn't find the specified plugin name for cred/munge looking at all files
by Jesse Aiton 24 Jan '24

24 Jan '24

Hello Slurm Folks, I have a weird issue where on the same server, which acts as both a controller and a node, slurmctld can’t find cred_munge.so slurmctld: debug3: Trying to load plugin /app/slurm-24.0.8/lib/slurm/cred_munge.so slurmctld: debug4: /app/slurm-24.0.8/lib/slurm/cred_munge.so: Does not exist or not a regular file. slurmctld: error: Couldn't find the specified plugin name for cred/munge looking at all files slurmctld: error: cannot open plugin directory /app/slurm-24.0.8/lib/slurm slurmctld: error: cannot find cred plugin for cred/munge slurmctld: error: cannot create cred context for cred/munge slurmctld: fatal: failed to initialize cred plugin But slurmd can: slurmd: debug3: Trying to load plugin /app/slurm-24.0.8/lib/slurm/cred_munge.so slurmd: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x180800 slurmd: cred/munge: init: Munge credential signature plugin loaded slurmd: debug3: Success. This is on Ubuntu 20.04 and happens both with Slurm 20.11.09 and 24.0.8 Thank you, Jesse # slurm.conf file generated by configurator easy.html. # Put this file on all nodes of your cluster. # See the slurm.conf man page for more information. # ClusterName=prod-cluster SlurmctldHost=controller # #MailProg=/bin/mail #MpiDefault= #MpiParams=ports=#-# ProctrackType=proctrack/cgroup ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid #SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid #SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm #SlurmdUser=root StateSaveLocation=/var/spool/slurmctld #SwitchType= TaskPlugin=task/affinity,task/cgroup # # # TIMERS #KillWait=30 #MinJobAge=300 #SlurmctldTimeout=120 #SlurmdTimeout=300 # # # SCHEDULING SchedulerType=sched/backfill SelectType=select/cons_tres # # # LOGGING AND ACCOUNTING #AccountingStorageType= #JobAcctGatherFrequency=30 #JobAcctGatherType= #SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld.log #SlurmdDebug=info SlurmdLogFile=/var/log/slurmd.log # # # COMPUTE NODES NodeName=controller CPUs=1 State=UNKNOWN NodeName=node CPUs=1 State=UNKNOWN PartitionName=prod-part Nodes=ALL Default=YES MaxTime=INFINITE State=UP

3 5

Slurm version 23.11.2 is now available
by Tim McMullan 23 Jan '24

23 Jan '24

We are pleased to announce the availability of Slurm version 23.11.2. The 23.11.2 release includes a number of fixes to stability and various bug fixes. Some notable changes include several fixes to the new scontrol reconfigure method, including one that could result in jobs getting cancelled prematurely, a couple errors that resulted in the backup slurmctld stopping on fail-back, and an issue during upgrades with older MySQL versions with a small max_allowed_packet value for sites with a large number of associations. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.2 > ========================== > -- slurmrestd - Reject single http query with multiple path requests. > -- Fix launching Singularity v4.x containers with srun --container by setting > .process.terminal to true in generated config.json when step has > pseudoterminal (--pty) requested. > -- Fix loading in dyanmic/cloud node jobs after net_cred expired. > -- Fix cgroup null path error on slurmd/slurmstepd tear down. > -- data_parser/v0.0.40 - Prevent failure if accounting is disabled, instead > issue a warning if needed data from the database can not be retrieved. > -- openapi/slurmctld - Prevent failure if accounting is disabled. > -- Prevent slurmscriptd processing delays from blocking other threads in > slurmctld while trying to launch various scripts. This is additional work > for a fix in 23.02.6. > -- Fix memory leak when receiving alias addrs from controller. > -- scontrol - Accept `scontrol token lifespan=infinite` to create tokens that > effectively do not expire. > -- Avoid errors when Slurmdb accounting disabled when '--json' or '--yaml' is > invoked with CLI commands and slurmrestd. Add warnings when query would > have populated data from Slurmdb instead of errors. > -- Fix slurmctld memory leak when running job with --tres-per-task=gres:shard:# > -- Fix backfill trying to start jobs outside of backfill window. > -- Fix oversubscription on partitions with PreemptMode=OFF. > -- Preserve node reason on power up if the node is downed or drained. > -- data_parser/v0.0.40 - Avoid aborting when invoking a not implemented > parser. > -- data_parser/v0.0.40 - Fix how nice values are parsed for job submissions. > -- data_parser/v0.0.40 - Fix regression where parsing error did not result in > invalid request being rejected. > -- Fix segfault in front-end node registration. > -- Prevent jobs using none typed gpus from being killed by the controller after > a reconfig or restart. > -- Fix deadlock situation in the dbd when adding associations. > -- Update default values of text/blob columns when updating from old mysql > versions in more situations. This improves a previous fix to handle an > uncommon case when upgrading mysql/mariadb. > -- Fix rpmbuild in openSUSE/SLES due to incorrect mariadb dependency. > -- Fix compilation on RHEL 7. > -- When upgrading the slurmdbd to 23.11, avoid generating a query to update > the association table that is larger than max_allowed_packet which would > result in an upgrade failure. > -- Fix rare deadlock when a dynamic node registers at the same time that a > once per minute background task occurs. > -- Fix build issue on 32-bit systems. > -- data_parser/v0.0.40 - Fix enumerated strings in OpenAPI specification not > have type field specified. > -- Improve scontrol show job -d information of used shared gres (shard/mps) > topology. > -- Allow Slurm to compile without MUNGE if --without-munge is used as an > argument to configure. > -- accounting_storage/mysql - Fix usage query to use new lineage column > instead of lft/rgt. > -- slurmrestd - Improve handling of missing parsers when content plugins > expect parsers not loaded. > -- slurmrestd - Correct parsing of StepIds when querying jobs. > -- slurmrestd - Improve error from parsing failures of lists. > -- slurmrestd - Improve parsing of singular values for lists. > -- accounting_storage/mysql - Fix PrivateData=User when listing associations. > -- Disable sorting of dynamic nodes to avoid issues when restarting with > heterogenous jobs that cause jobs to abort on restart. > -- Don't allow deletion of non-dynamic nodes. > -- accounting_storage/mysql - Fix issue adding partition based associations. > -- Respect non-"slurm" settings for I_MPI_HYDRA_BOOTSTRAP and HYDRA_BOOTSTRAP > and avoid injecting the --external-launcher option which will cause > mpirun/mpiexec to fail with an unexpected argument error. > -- Fix bug where scontrol hold would change node count for jobs with > implicitly defined node counts. > -- data_parser/v0.0.40 - Fix regression of support for "hold" in > job description. > -- Avoid sending KILL RPCs to unresolvable POWERING_UP and POWERED_DOWN nodes. > -- data_parser/v0.0.38 - Fix several potential NULL dereferences that could > cause slurmrestd to crash. > -- Add --gres-flags=one-task-per-sharing. Do not allow different tasks in to be > allocated shared gres from the same sharing gres. > -- Add SelectTypeParameters=ENFORCE_BINDING_GRES and ONE_TASK_PER_SHARING_GRES. > This gives default behavior for a job's --gres-flags. > -- Alter the networking code to try connecting to the backup controllers > if the DNS lookup for the primary SlurmctldHost fails. > -- Alter the name resolution to only log at verbose() in client commands > on failures. This allows for HA setups where the DNS entries are withdrawn > for some SlurmctldHost entries without flooding the user with errors. > -- Prevent slurmscriptd PID leaks when running slurmctld in foreground mode. > -- Open all slurmctld listening ports at startup, and persist throughout. > This also changes the backup slurmctld process to open the SlurmctldPort > range, instead of only the first. > -- Fix backup slurmctld shutting down instead of resuming standby duty if it > took control. > -- Fix race condition that delayed the primary slurmctld resuming when taking > control from a backup controller. > -- srun - Ensure processed messages are meant for this job in case of a > rapidly-reused TCP port. > -- srun - Prevent step launch failure while waiting for step allocation if > a stray message is received. > -- Fix backup slurmctld to be able to respond to configless config file > requests correctly. > -- Fix slurmctld crashing when recovering from a failed reconfigure. > -- Fix slurmscriptd operation after recovering from a failed reconfigure. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

GPU devices mapping with job's cgroup in cgroups v2 using eBPF
by Mahendra Paipuri 23 Jan '24

23 Jan '24

Hello all, Happy new year! We have recently upgraded the cgroups on our SLURM cluster to v2. In cgroups v1, the interface `/devices.list` used to have the information of which device has been attached to that particular cgroup. From my understanding, cgroups v2 use eBPF to manage devices and so as SLURM to manage the GPUs. I was looking for a way to be able to programatically determine the job cgroups to device mapping and I came across this thread ( https://bugzilla.redhat.com/show_bug.cgi?id=1717396) which has a similar discussion in the context of VMs. So, I have used `bpftool` to inspect the job cgroups. An example output: ``` # /tmp/bpftool cgroup list /sys/fs/cgroup/system.slice/slurmstepd.scope/job_1956132 ID AttachType AttachFlags Name ``` When I add `effective` flag, I see the attached eBPF program ``` # /tmp/bpftool cgroup list /sys/fs/cgroup/system.slice/slurmstepd.scope/job_1956132 effective ID AttachType Name 4197 cgroup_device Slurm_Cgroup_v2 ``` >From my understand, `effective` flag shows the inherited eBPF programs as well. So, my question is at which level of cgroups the eBPF program is attached? I tried to inspect at various levels but all of them returned none. Then looking into translated byte code of eBPF program, I get the following ``` # /tmp/bpftool prog dump xlated id 4197 0: (61) r2 = *(u32 *)(r1 +0) 1: (54) w2 &= 65535 2: (61) r3 = *(u32 *)(r1 +0) 3: (74) w3 >>= 16 4: (61) r4 = *(u32 *)(r1 +4) 5: (61) r5 = *(u32 *)(r1 +8) 6: (55) if r2 != 0x2 goto pc+4 7: (55) if r4 != 0xc3 goto pc+3 8: (55) if r5 != 0x0 goto pc+2 9: (b7) r0 = 0 10: (95) exit 11: (55) if r2 != 0x2 goto pc+4 12: (55) if r4 != 0xc3 goto pc+3 13: (55) if r5 != 0x1 goto pc+2 14: (b7) r0 = 0 15: (95) exit 16: (55) if r2 != 0x2 goto pc+4 17: (55) if r4 != 0xc3 goto pc+3 18: (55) if r5 != 0x2 goto pc+2 19: (b7) r0 = 0 20: (95) exit 21: (55) if r2 != 0x2 goto pc+4 22: (55) if r4 != 0xc3 goto pc+3 23: (55) if r5 != 0x3 goto pc+2 24: (b7) r0 = 1 25: (95) exit 26: (b7) r0 = 1 27: (95) exit ``` >From the output, it is clear that GPU:3 (among 0,1,2,3) is the one that is attached to that job's cgroup. However, I was looking for a way to dump eBPF maps that can directly provide the major, minor numbers and permissions of device as discussed in the comment (https://bugzilla.redhat.com/show_bug.cgi?id=1717396#c5). When I inspect eBPF program, I dont see any maps associated. ``` # /tmp/bpftool prog list id 4197 4197: cgroup_device name Slurm_Cgroup_v2 tag 1a261c8a913ff67c gpl loaded_at 2024-01-02T08:19:56+0100 uid 0 xlated 224B jited 142B memlock 4096B ``` So, my second question is how can I get a similar information as `map dump` that can give us the device's major minor numbers directly instead of parsing the byte code from `prog dump`? I am still discovering the eBPF ecosystem so if I am missing something very obvious, please let me know. I would really appreciate that. Cheers! Regards Mahendra

2 1

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 23 Jan '24

23 Jan '24

Hi, I have filed a bug report with SchedMD (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the support told me they cannot invest time in this issue since I don't have a support contract. Maybe they will look into it once it affects more people or someone important enough. So far, I have resorted to using 5.15.0-89-generic, but I am also a bit concerned about the security aspect of this choice. Best, Tim On 23.01.24 14:59, Stefan Fleischmann wrote: > Hi! > > I'm seeing the same in our environment. My conclusion is that it is a > regression in the Ubuntu 5.15 kernel, introduced with 5.15.0-90-generic. > Last working kernel version is 5.15.0-89-generic. I have filed a bug > report here: https://bugs.launchpad.net/bugs/2050098 > > Please add yourself to the affected users in the bug report so it > hopefully gets more attention. > > I've tested with newer kernels (6.5, 6.6 and 6.7) and the problem does > not exist there. 6.5 is the latest hwe kernel for 22.04 and would be an > option for now. Reverting back to 5.15.0-89 would work as well, but I > haven't looked into the security aspects of that. > > Cheers, > Stefan > > On Mon, 22 Jan 2024 13:31:15 -0300 > cristobal.navarro.g at gmail.com wrote: > >> Hi Tim and community, >> We are currently having the same issue (cgroups not working it seems, >> showing all GPUs on jobs) on a GPU-compute node (DGX A100) a couple >> of days ago after a full update (apt upgrade). Now whenever we launch >> a job for that partition, we get the error message mentioned by Tim. >> As a note, we have another custom GPU-compute node with L40s, on a >> different partition, and that one works fine. >> Before this error, we always had small differences in kernel version >> between nodes, so I am not sure if this can be the problem. >> Nevertheless, here is the info of our nodes as well. >> >> *[Problem node]* The DGX A100 node has this kernel >> cnavarro at nodeGPU01:~$ uname -a >> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 20:28:30 >> UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> *[Functioning node]* The Custom GPU node (L40s) has this kernel >> cnavarro at nodeGPU02:~$ uname -a >> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 >> UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> *And the login node *(slurmctld) >> ? ~ uname -a >> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> Any ideas what we should check? >> >> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 at >> tu-darmstadt.de> wrote: >> >>> Hi, >>> >>> I am using SLURM 22.05.9 on a small compute cluster. Since I >>> reinstalled two of our nodes, I get the following error when >>> launching a job: >>> >>> slurmstepd: error: load_ebpf_prog: BPF load error (No space left on >>> device). Please check your system limits (MEMLOCK). >>> >>> Also the cgroups do not seem to work properly anymore, as I am able >>> to see all GPUs even if I do not request them, which is not the >>> case on the other nodes. >>> >>> One difference I found between the updated nodes and the original >>> nodes (both are Ubuntu 22.04) is the kernel version, which is >>> "5.15.0-89-generic #99-Ubuntu SMP" on the functioning nodes and >>> "5.15.0-91-generic #101-Ubuntu SMP" on the updated nodes. I could >>> not figure out how to install the exact first kernel version on the >>> updated nodes, but I noticed that when I reinstall 5.15.0 with this >>> tool: https://github.com/pimlie/ubuntu-mainline-kernel.sh, the >>> error message disappears. However, once I do that, the network >>> driver does not function properly anymore, so this does not seem to >>> be a good solution. >>> >>> Has anyone seen this issue before or is there maybe something else I >>> should take a look at? I am also happy to just find a workaround >>> such that I can take these nodes back online. >>> >>> I appreciate any help! >>> >>> Thanks a lot in advance and best wishes, >>> >>> Tim >>> >>> >>>

2 1

Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres)
by Kherfani, Hafedh (Professional Services, TC) 23 Jan '24

23 Jan '24

Hello Experts, I'm a new Slurm user (so please bare with me :) ...). Recently we've deployed Slurm version 23.11 on a very simple cluster, which consists of a Master node (acting as a Login & Slurmdbd node as well), a Compute Node which has a NVIDIA HGX A100-SXM4-40GB GPU, detected as 4 x GPU's: GPU [0-4], and a Storage Array presenting/sharing the NFS disk (where users' home directories will be created as well). The problem is that I've never been able to run a simple/dummy batch script in a parallel way using the 4 GPU's. In fact, running the same command "sbatch gpu-job.sh" multiple times shows that only one single job is running, while the other jobs are in a pending state: [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 214 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 215 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 216 [slurmtest@c-a100-master test-batch-scripts]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 216 gpu gpu-job slurmtest PD 0:00 1 (None) 215 gpu gpu-job slurmtest PD 0:00 1 (Priority) 214 gpu gpu-job slurmtest PD 0:00 1 (Priority) 213 gpu gpu-job slurmtest PD 0:00 1 (Priority) 212 gpu gpu-job slurmtest PD 0:00 1 (Resources) 211 gpu gpu-job slurmtest R 0:14 1 c-a100-cn01 PS: CPU jobs (i.e. using the default debug partition, without call the GPU Gres) can be run in parallel. The issue with running parallel jobs is only seen when using the GPU's as Gres. I've tried many combinations of settings in gres.conf and slurm.conf, many (if not most) of these combinations would result in error messages in slurmctld and slurmd logs. The current gres.conf and slurm.conf contents is shown below. Even though it doesn't give errors when restarting slurmctld and slurmd services (on master and compute nodes, resp.), but as I said, it doesn't allow jobs to be executed in parallel. Batch script contents shared below as well, in order to give more clarity on what I'm trying to do: [root@c-a100-master slurm]# cat gres.conf | grep -v "^#" NodeName=c-a100-cn01 AutoDetect=nvml Name=gpu Type=A100 File=/dev/nvidia[0-3] [root@c-a100-master slurm]# cat slurm.conf | grep -v "^#" | egrep -i "AccountingStorageTRES|GresTypes|NodeName|partition" GresTypes=gpu AccountingStorageTRES=gres/gpu NodeName=c-a100-cn01 Gres=gpu:A100:4 CPUs=64 Boards=1 SocketsPerBoard=1 CoresPerSocket=32 ThreadsPerCore=2 RealMemory=515181 State=UNKNOWN PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP PartitionName=gpu Nodes=ALL MaxTime=10:0:0 [slurmtest@c-a100-master test-batch-scripts]$ cat gpu-job.sh #!/bin/bash #SBATCH --job-name=gpu-job #SBATCH --partition=gpu #SBATCH --nodes=1 #SBATCH --gpus-per-node=4 #SBATCH --gres=gpu:4 #SBATCH --tasks-per-node=1 #SBATCH --output=gpu_job_output.%j # Output file name (replaces %j with job ID) #SBATCH --error=gpu_job_error.%j # Error file name (replaces %j with job ID) hostname date sleep 40 pwd Any help on which changes need to be made to the config files (mainly slurm.conf and gres.cong) and/or the batch script, so that multiple jobs can be in a "Running" state at the same time (in parallel) ? Thanks in advance for your help ! Best regards, Hafedh Kherfani

8 10

2024

slurm-users ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users