January 2024 - slurm-users - lists.schedmd.com

srun only runs one job on a node
by Rike-Benjamin Schuppner 27 Jan '24

27 Jan '24

Hi, Not sure if I do not understand the scheduling algorithm correctly of if this is a bug. The bug seems similar to the one described and fixed here https://github.com/SchedMD/slurm/commit/6a2c99edbf96e50463cc2f16d8e5eb955c8… Given the node definition: NodeName=cluster01 CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=64283 NodeName=cluster02 CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=128814 PartitionName=DEFAULT MaxTime=2-0 MaxNodes=2 PartitionName=mini Nodes=cluster01,cluster02 State=UP and the batch script: for i in {1..6} do srun --exclusive --mem=100 --threads-per-core=1 -N1 -c1 -n1 bash -c 'echo $(hostname) $(date); sleep 10 ; echo done $(hostname) $(date)' & done wait When run with: time sbatch --wait --partition=mini --ntasks=10 --cpus-per-task=1 --mem=1000 mini.slurm I get the following log after 2m 18s: cluster01 Sat Jan 27 02:05:12 PM CET 2024 cluster01 Sat Jan 27 02:05:12 PM CET 2024 cluster01 Sat Jan 27 02:05:13 PM CET 2024 cluster01 Sat Jan 27 02:05:13 PM CET 2024 cluster02 Sat Jan 27 02:05:13 PM CET 2024 done cluster01 Sat Jan 27 02:05:22 PM CET 2024 done cluster01 Sat Jan 27 02:05:22 PM CET 2024 done cluster01 Sat Jan 27 02:05:23 PM CET 2024 done cluster01 Sat Jan 27 02:05:23 PM CET 2024 done cluster02 Sat Jan 27 02:05:23 PM CET 2024 srun: Job 650 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=650.6 cluster01 Sat Jan 27 02:07:14 PM CET 2024 done cluster01 Sat Jan 27 02:07:24 PM CET 2024 So apparently I am getting 4 sruns on cluster01 (why not 6 when I have 8 CPUs?), but only 1 is scheduled on cluster02. Then it is exhausted and waits endlessly long (almost 2 minutes?) before scheduling again. The step log tells me: slurmctld: debug: laying out the 1 tasks on 1 hosts cluster01 dist 1 slurmctld: STEPS: _step_alloc_lps: JobId=650 StepId=4 node 0 (cluster01) gres_cpus_alloc=0 tasks=1 cpus_per_task=1 slurmctld: STEPS: _pick_step_cores: step JobId=650 StepId=4 requires 1 cores on node 0 with cpus_per_core=1, available cpus from job: 8 slurmctld: STEPS: _pick_step_core: alloc Node:0 Socket:0 Core:3 slurmctld: STEPS: step alloc on job node 0 (cluster01) used 4 of 8 CPUs slurmctld: STEPS: _slurm_rpc_job_step_create: JobId=650 StepId=4 cluster01 usec=5502 slurmctld: STEPS: _pick_step_nodes: JobId=650 Currently running steps use 4 of allocated 8 CPUs on node cluster01 slurmctld: STEPS: _pick_step_nodes: JobId=650 Currently running steps use 1 of allocated 2 CPUs on node cluster02 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=0 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=1 has nodes cluster02 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=2 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=3 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=4 has nodes cluster01 slurmctld: STEPS: _pick_step_nodes: step pick 1-1 nodes, avail:cluster[01-02] idle: picked:NONE slurmctld: STEPS: _pick_step_nodes: step picked 0 of 1 nodes slurmctld: STEPS: Picked nodes cluster01 when accumulating from cluster01 slurmctld: debug: laying out the 1 tasks on 1 hosts cluster01 dist 1 slurmctld: STEPS: _step_alloc_lps: JobId=650 StepId=5 node 0 (cluster01) gres_cpus_alloc=0 tasks=1 cpus_per_task=1 slurmctld: STEPS: _pick_step_cores: step JobId=650 StepId=5 requires 1 cores on node 0 with cpus_per_core=1, available cpus from job: 8 slurmctld: STEPS: unable to pick step cores for job node 0 (cluster01): Requested nodes are busy slurmctld: STEPS: Deallocating 100MB of memory on node 0 (cluster01) now used: 400 of 1000 slurmctld: STEPS: step dealloc on job node 0 (cluster01) used: 4 of 8 CPUs slurmctld: STEPS: _slurm_rpc_job_step_create for JobId=650: Requested nodes are busy So why is it unable to allocate on cluster02? And why does its need to when it is only using 4 of 8 CPUs on cluster01 anyway? Now if I change to --ntasks=8, the algorithm doesn’t even schedule cluster02 at all: cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 srun: Job 651 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=651.6 cluster01 Sat Jan 27 02:17:35 PM CET 2024 srun: Job 651 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=651.7 cluster01 Sat Jan 27 02:17:36 PM CET 2024 done cluster01 Sat Jan 27 02:17:45 PM CET 2024 done cluster01 Sat Jan 27 02:17:46 PM CET 2024 Thanks for your help. Best /rike

1 0

LuaSQLite3 with slurm-lua-spank
by Jackson, Gary L. 26 Jan '24

26 Jan '24

What do I need to do to use the LuaSQLite3 library with Lua SPANK? For what it’s worth, I don’t know anything about Lua. -- Gary

1 0

sinfo: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host
by Michael Lewis 26 Jan '24

26 Jan '24

Hi All, I’m trying to get slurm-23.11.3 running on Ubuntu 20.04 and running on a stand alone system. I’m running into an issue I can not find the answer to. After compiling and installing when I fire up slurmctld and slurmd I get an error from sinfo: sinfo: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host sinfo: error: fetch_config: DNS SRV lookup failed sinfo: error: _establish_config_source: failed to fetch config sinfo: fatal: Could not establish a configuration source I looks like a DNS issue but the system has no issue resolving to its hostname or localhost. The slurm.conf file is also being read properly as I have the logs directed to a place convenient to me. I see lots have had these same issues but cannot find a clear resolution. I have slurm running on a stand alone system in another lab with and identical setup without issue. Any advice would be greatly appreciated. Thanks, Mike

2 2

Database cluster
by Daniel L'Hommedieu 26 Jan '24

26 Jan '24

Community: What do you do to ensure database reliability in your SLURM environment? We can have multiple controllers and multiple slurmdbds, but my understanding is that slurmdbd can be configured with a single MySQL server, so what do you do? Do you have that “single MySQL server” be a cluster, such as Percona XtraDB? Do you use MySQL replication, then manually switch to slurmdbd to a replication slave if the master goes down? Do you do something else? Thanks. Daniel

6 7

Problem using Podman with scrun on SLURM 23.11.3
by Marcus Lauer 25 Jan '24

25 Jan '24

I am getting an unusual error when trying to run Podman containers using scrun on SLURM 23.11.3 (and 23.11.1 previously). In short, Podman works when not configured to use scrun, but when configured to use scrun it fails. Podman gives this error: scrun: fatal: Unable to request job allocation: Job cannot be submitted without the current working directory specified. The slurmctld logs show this error: _slurm_rpc_allocate_resources: Job cannot be submitted without the current working directory specified. The scrun command does not appear to have a --chdir option so the working directory must be detected automatically, and this appears to be failing. Perhaps someone has encountered this or a similar error before? If anyone has seen this before I would love to hear about your experiences. A bit more detail in case anyone is interested. This is a two-node test system with one head node and one compute node, both running Rocky Linux 8.9. I have created /etc/containers/storage.conf and containers.conf files which are identical to the ones in the SLURM Containers Guide except that I set the rootless_storage_path to the same value as the graphroot in storage.conf. When I get rid of the /etc/containers/containers.conf file I can run containers on the head node. The backing store is ext4, though I have tried NFS (which had other problems). I have tried the vfs and overlay storage drivers but get the same results. On the SLURM side I created an oci.conf which is the same as the "oci.conf example for crun using run (suggested)" from the SLURM Container Guide. I'm using crun because I have forced cgroup v2 on both systems ( systemd.unified_cgroup_hierarchy=1 on the kernel cmdline) and crun seems to support cgroup v2 better. I am using an scrun.lua which is similar to the one in the scrun manual page with some paths modified for my setup. I should also mention that sbatch jobs run just fine and srun works as well. the test cluster seems to be working fine in general. -- Marcus Lauer Systems Administrator CETS Group, Research Support

1 0

Question about CPUs and cores
by Gestió Servidors 25 Jan '24

25 Jan '24

Hi, I want to run a simple test that uses one node and four cores. Also, in my script, I execute a binary that reports me in what core is running one of the four tasks. These are my files: * submit script: #!/bin/bash #SBATCH --job-name=test_jobs # Job name #SBATCH --output=test_job_%j.out # Output file (using array index) #SBATCH --error=test_job_%j.err # Error file (using array index) #SBATCH --ntasks=4 #SBATCH --ntasks-per-core=1 #SBATCH --cpus-per-task=1 # Number of CPU cores per task echo "START : $(date)" srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & # Wait for all jobs to finish before exiting the job submission script wait echo "END : $(date)" * check_cores.c file: #include <stdio.h> #include<unistd.h> struct pstat { int pid; //process id char tcomm[256];//filename of the executable char state[2]; //state (R is running, S is sleeping, D is sleeping in an //uninterruptible wait, Z is zombie, T is traced or stopped) int ppid;// process id of the parent process int pgrp;// pgrp of the process int sid;// session id int tty_nr;// tty the process uses int tty_pgrp;// pgrp of the tty int flags;// task flags int min_flt;// number of minor faults int cmin_flt;// number of minor faults with child's int maj_flt;// number of major faults int cmaj_flt;// number of major faults with child's int utime;// user mode jiffies int stime;// kernel mode jiffies int cutime;// user mode jiffies with child's int cstime;// kernel mode jiffies with child's int priority;// priority level int nice;// nice level int num_threads;// number of threads int it_real_value;// (obsolete, always 0) int start_time;// time the process started after system boot int vsize;// virtual memory size int rss;// resident set memory size int rsslim;// current limit in bytes on the rss int start_code;// address above which program text can run int end_code;// address below which program text can run int start_stack;// address of the start of the stack int esp;// current value of ESP int eip;// current value of EIP int pending;// bitmap of pending signals int blocked;// bitmap of blocked signals int sigign;// bitmap of ignored signals int sigcatch;// bitmap of catched signals int wchan;// address where process went to sleep int i0;// (place holder) int i1;// (place holder) int exit_signal;// signal to send to parent thread on exit int task_cpu;// which CPU the task is scheduled on int rt_priority;// realtime priority int policy;// scheduling policy (man sched_setscheduler) int blkio_ticks;// time spent waiting for block IO int gtime;// guest time of the task in jiffies int cgtime;// guest time of the task children in jiffies } p ; int main() { char name[256]; char state[8]; FILE* f = fopen("/proc/self/stat", "r"); char hostname[1024]; gethostname(hostname, 1024); fscanf(f, "%d%s%s%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d", &p.pid, &p.tcomm, &p.state, &p.ppid, &p.pgrp, &p.sid, &p.tty_nr, &p.tty_pgrp, &p.flags, &p.min_flt, &p.cmin_flt, &p.maj_flt, &p.cmaj_flt, &p.utime, &p.stime, &p.cutime, &p.cstime, &p.priority, &p.nice, &p.num_threads, &p.it_real_value, &p.start_time, &p.vsize, &p.rss, &p.rsslim, &p.start_code, &p.end_code, &p.start_stack, &p.esp, &p.eip, &p.pending, &p.blocked, &p.sigign, &p.sigcatch, &p.wchan, &p.i0, &p.i1, &p.exit_signal, &p.task_cpu, &p.rt_priority, &p.policy, &p.blkio_ticks, &p.gtime, &p.cgtime); sleep (15); printf("HOST %s\n", hostname); printf("CPU %d\n", p.task_cpu); return 0; } When I run this script, I should receive an output like this: HOST clus04.hpc.local CPU 0 HOST clus04.hpc.local CPU 7 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 6 because system has scheduled my job in node "clus04.hpc.local" (a node with two 6-cores processors) and has reserved cores 0-1 of each processor. However, the real output I receive is HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 6 (and others outputs where there are one CPU repeated), so it seems SLURMs is reserving correctly four cores but, then, process is running two or more times in the same core. Slurmctld.log file in server shows this information: [...] [2024-01-25T09:38:42.867] DEBUG: JobId=206129 node clus04 vpus 1 cpus 4 [2024-01-25T09:38:42.867] ==================== [2024-01-25T09:38:42.867] JobId=206129 nhosts:1 ncpus:4 node_req:1 nodes=clus04 [2024-01-25T09:38:42.867] Node[0]: [2024-01-25T09:38:42.867] Mem(MB):0:0 Sockets:2 Cores:6 CPUs:4:0 [2024-01-25T09:38:42.867] Socket[0] Core[0] is allocated [2024-01-25T09:38:42.867] Socket[0] Core[1] is allocated [2024-01-25T09:38:42.867] Socket[1] Core[0] is allocated [2024-01-25T09:38:42.867] Socket[1] Core[1] is allocated [2024-01-25T09:38:42.867] -------------------- [2024-01-25T09:38:42.867] cpu_array_value[0]:4 reps:1 [2024-01-25T09:38:42.867] ==================== [2024-01-25T09:38:42.867] DEBUG: Dump job_resources: nhosts 1 cb 0-1,6-7 [...] [2024-01-25T09:38:58.213] _job_complete: JobId=206129 WEXITSTATUS 0 [2024-01-25T09:38:58.213] deallocate_nodes: JobId=206129 [2024-01-25T09:38:58.213] DEBUG: Dump job_resources: nhosts 1 cb 0-1,6-7 [2024-01-25T09:38:58.213] _job_complete: JobId=206129 done And slurmd.log file in node clus04 shows this information: [...] [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: step physical cores are '0-1,6-7' It seems that SLURM is reserving me correctly four cores for my job but, when inside my script I run four "srun", then each srun can freely take one of the four reserved cores, not exactly one to each task. What am I doing wrong? Thanks a lot!

1 0

Slurm version 23.11.3 is now available
by Tim McMullan 24 Jan '24

24 Jan '24

We are pleased to announce the availability of Slurm version 23.11.3. The 23.11.3 fixes a single issue related to the new Debian package support. For 23.11.2, the debian changelog was not updated correctly and would generate packages as 23.11.1. This release will correctly generate 23.11.3 packages, with no other changes. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.3 > ========================== > -- Fix debian/changelog file to reflect the correct version so the packages > are generated correctl. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 24 Jan '24

24 Jan '24

Hi, I just tested with 23.02.7-1 and the issue is gone. So it seems like the patch got released. Best, Tim On 1/24/24 16:55, Stefan Fleischmann wrote: > On Wed, 24 Jan 2024 12:37:04 -0300 Cristóbal Navarro > <cristobal.navarro.g(a)gmail.com> wrote: >> Many thanks >> One question? Do we have to apply this patch (and recompile slurm i >> guess) only on the compute-node with problems? >> Also, I noticed the patch now appears as "obsolete", is that ok? > We have Slurm installed on a NFS share, so what I did was to recompile > it and then I only replaced the library lib/slurm/cgroup_v2.so. Good > enough for now, I've been planning to update to 23.11 anyway soon. > > I suppose it's marked as obsolete because the patch went into a > release. According to the info in the bug report it should have been > included in 23.02.4. > > Cheers, > Stefan > >> On Wed, Jan 24, 2024 at 9:52 AM Stefan Fleischmann <sfle(a)kth.se> >> wrote: >> >>> Turns out I was wrong, this is not a problem in the kernel at all. >>> It's a known bug that is triggered by long bpf logs, see here >>> https://bugs.schedmd.com/show_bug.cgi?id=17210 >>> >>> There is a patch included there. >>> >>> Cheers, >>> Stefan >>> >>> On Tue, 23 Jan 2024 15:28:59 +0100 Stefan Fleischmann <sfle(a)kth.se> >>> wrote: >>>> I don't think there is much for SchedMD to do. As I said since it >>>> is working fine with newer kernels there doesn't seem to be any >>>> breaking change in cgroup2 in general, but only a regression >>>> introduced in one of the latest updates in 5.15. >>>> >>>> If Slurm was doing something wrong with cgroup2, and it >>>> accidentally worked until this recent change, then other kernel >>>> versions should show the same behavior. But as far as I can tell >>>> it still works just fine with newer kernels. >>>> >>>> Cheers, >>>> Stefan >>>> >>>> On Tue, 23 Jan 2024 15:20:56 +0100 >>>> Tim Schneider <tim.schneider1(a)tu-darmstadt.de> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have filed a bug report with SchedMD >>>>> (https://bugs.schedmd.com/show_bug.cgi?id=18623) but the >>>>> support told me they cannot invest time in this issue since I >>>>> don't have a support contract. Maybe they will look into it >>>>> once it affects more people or someone important enough. >>>>> >>>>> So far, I have resorted to using 5.15.0-89-generic, but I am >>>>> also a bit concerned about the security aspect of this choice. >>>>> >>>>> Best, >>>>> >>>>> Tim >>>>> >>>>> On 23.01.24 14:59, Stefan Fleischmann wrote: >>>>>> Hi! >>>>>> >>>>>> I'm seeing the same in our environment. My conclusion is that >>>>>> it is a regression in the Ubuntu 5.15 kernel, introduced with >>>>>> 5.15.0-90-generic. Last working kernel version is >>>>>> 5.15.0-89-generic. I have filed a bug report here: >>>>>> https://bugs.launchpad.net/bugs/2050098 >>>>>> >>>>>> Please add yourself to the affected users in the bug report >>>>>> so it hopefully gets more attention. >>>>>> >>>>>> I've tested with newer kernels (6.5, 6.6 and 6.7) and the >>>>>> problem does not exist there. 6.5 is the latest hwe kernel >>>>>> for 22.04 and would be an option for now. Reverting back to >>>>>> 5.15.0-89 would work as well, but I haven't looked into the >>>>>> security aspects of that. >>>>>> >>>>>> Cheers, >>>>>> Stefan >>>>>> >>>>>> On Mon, 22 Jan 2024 13:31:15 -0300 >>>>>> cristobal.navarro.g at gmail.com wrote: >>>>>> >>>>>>> Hi Tim and community, >>>>>>> We are currently having the same issue (cgroups not working >>>>>>> it seems, showing all GPUs on jobs) on a GPU-compute node >>>>>>> (DGX A100) a couple of days ago after a full update (apt >>>>>>> upgrade). Now whenever we launch a job for that partition, >>>>>>> we get the error message mentioned by Tim. As a note, we >>>>>>> have another custom GPU-compute node with L40s, on a >>>>>>> different partition, and that one works fine. Before this >>>>>>> error, we always had small differences in kernel version >>>>>>> between nodes, so I am not sure if this can be the problem. >>>>>>> Nevertheless, here is the info of our nodes as well. >>>>>>> >>>>>>> *[Problem node]* The DGX A100 node has this kernel >>>>>>> cnavarro at nodeGPU01:~$ uname -a >>>>>>> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 >>>>>>> 20:28:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> *[Functioning node]* The Custom GPU node (L40s) has this >>>>>>> kernel cnavarro at nodeGPU02:~$ uname -a >>>>>>> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 >>>>>>> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> *And the login node *(slurmctld) >>>>>>> ? ~ uname -a >>>>>>> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue >>>>>>> Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> Any ideas what we should check? >>>>>>> >>>>>>> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 >>>>>>> at tu-darmstadt.de> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am using SLURM 22.05.9 on a small compute cluster. Since I >>>>>>>> reinstalled two of our nodes, I get the following error when >>>>>>>> launching a job: >>>>>>>> >>>>>>>> slurmstepd: error: load_ebpf_prog: BPF load error (No space >>>>>>>> left on device). Please check your system limits (MEMLOCK). >>>>>>>> >>>>>>>> Also the cgroups do not seem to work properly anymore, as I >>>>>>>> am able to see all GPUs even if I do not request them, >>>>>>>> which is not the case on the other nodes. >>>>>>>> >>>>>>>> One difference I found between the updated nodes and the >>>>>>>> original nodes (both are Ubuntu 22.04) is the kernel >>>>>>>> version, which is "5.15.0-89-generic #99-Ubuntu SMP" on the >>>>>>>> functioning nodes and "5.15.0-91-generic #101-Ubuntu SMP" >>>>>>>> on the updated nodes. I could not figure out how to install >>>>>>>> the exact first kernel version on the updated nodes, but I >>>>>>>> noticed that when I reinstall 5.15.0 with this tool: >>>>>>>> https://github.com/pimlie/ubuntu-mainline-kernel.sh, the >>>>>>>> error message disappears. However, once I do that, the >>>>>>>> network driver does not function properly anymore, so this >>>>>>>> does not seem to be a good solution. >>>>>>>> >>>>>>>> Has anyone seen this issue before or is there maybe >>>>>>>> something else I should take a look at? I am also happy to >>>>>>>> just find a workaround such that I can take these nodes >>>>>>>> back online. >>>>>>>> >>>>>>>> I appreciate any help! >>>>>>>> >>>>>>>> Thanks a lot in advance and best wishes, >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> >>>>>>>> >>>> >>> >>

2 1

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Cristóbal Navarro 24 Jan '24

24 Jan '24

Many thanks One question? Do we have to apply this patch (and recompile slurm i guess) only on the compute-node with problems? Also, I noticed the patch now appears as "obsolete", is that ok? On Wed, Jan 24, 2024 at 9:52 AM Stefan Fleischmann <sfle(a)kth.se> wrote: > Turns out I was wrong, this is not a problem in the kernel at all. It's > a known bug that is triggered by long bpf logs, see here > https://bugs.schedmd.com/show_bug.cgi?id=17210 > > There is a patch included there. > > Cheers, > Stefan > > On Tue, 23 Jan 2024 15:28:59 +0100 Stefan Fleischmann <sfle(a)kth.se> > wrote: > > I don't think there is much for SchedMD to do. As I said since it is > > working fine with newer kernels there doesn't seem to be any breaking > > change in cgroup2 in general, but only a regression introduced in one > > of the latest updates in 5.15. > > > > If Slurm was doing something wrong with cgroup2, and it accidentally > > worked until this recent change, then other kernel versions should > > show the same behavior. But as far as I can tell it still works just > > fine with newer kernels. > > > > Cheers, > > Stefan > > > > On Tue, 23 Jan 2024 15:20:56 +0100 > > Tim Schneider <tim.schneider1(a)tu-darmstadt.de> wrote: > > > > > Hi, > > > > > > I have filed a bug report with SchedMD > > > (https://bugs.schedmd.com/show_bug.cgi?id=18623) but the support > > > told me they cannot invest time in this issue since I don't have a > > > support contract. Maybe they will look into it once it affects more > > > people or someone important enough. > > > > > > So far, I have resorted to using 5.15.0-89-generic, but I am also a > > > bit concerned about the security aspect of this choice. > > > > > > Best, > > > > > > Tim > > > > > > On 23.01.24 14:59, Stefan Fleischmann wrote: > > > > Hi! > > > > > > > > I'm seeing the same in our environment. My conclusion is that it > > > > is a regression in the Ubuntu 5.15 kernel, introduced with > > > > 5.15.0-90-generic. Last working kernel version is > > > > 5.15.0-89-generic. I have filed a bug report here: > > > > https://bugs.launchpad.net/bugs/2050098 > > > > > > > > Please add yourself to the affected users in the bug report so it > > > > hopefully gets more attention. > > > > > > > > I've tested with newer kernels (6.5, 6.6 and 6.7) and the problem > > > > does not exist there. 6.5 is the latest hwe kernel for 22.04 and > > > > would be an option for now. Reverting back to 5.15.0-89 would work > > > > as well, but I haven't looked into the security aspects of that. > > > > > > > > Cheers, > > > > Stefan > > > > > > > > On Mon, 22 Jan 2024 13:31:15 -0300 > > > > cristobal.navarro.g at gmail.com wrote: > > > > > > > >> Hi Tim and community, > > > >> We are currently having the same issue (cgroups not working it > > > >> seems, showing all GPUs on jobs) on a GPU-compute node (DGX A100) > > > >> a couple of days ago after a full update (apt upgrade). Now > > > >> whenever we launch a job for that partition, we get the error > > > >> message mentioned by Tim. As a note, we have another custom > > > >> GPU-compute node with L40s, on a different partition, and that > > > >> one works fine. Before this error, we always had small > > > >> differences in kernel version between nodes, so I am not sure if > > > >> this can be the problem. Nevertheless, here is the info of our > > > >> nodes as well. > > > >> > > > >> *[Problem node]* The DGX A100 node has this kernel > > > >> cnavarro at nodeGPU01:~$ uname -a > > > >> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 > > > >> 20:28:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> *[Functioning node]* The Custom GPU node (L40s) has this kernel > > > >> cnavarro at nodeGPU02:~$ uname -a > > > >> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 > > > >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> *And the login node *(slurmctld) > > > >> ? ~ uname -a > > > >> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 > > > >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> Any ideas what we should check? > > > >> > > > >> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 at > > > >> tu-darmstadt.de> wrote: > > > >> > > > >>> Hi, > > > >>> > > > >>> I am using SLURM 22.05.9 on a small compute cluster. Since I > > > >>> reinstalled two of our nodes, I get the following error when > > > >>> launching a job: > > > >>> > > > >>> slurmstepd: error: load_ebpf_prog: BPF load error (No space left > > > >>> on device). Please check your system limits (MEMLOCK). > > > >>> > > > >>> Also the cgroups do not seem to work properly anymore, as I am > > > >>> able to see all GPUs even if I do not request them, which is not > > > >>> the case on the other nodes. > > > >>> > > > >>> One difference I found between the updated nodes and the > > > >>> original nodes (both are Ubuntu 22.04) is the kernel version, > > > >>> which is "5.15.0-89-generic #99-Ubuntu SMP" on the functioning > > > >>> nodes and "5.15.0-91-generic #101-Ubuntu SMP" on the updated > > > >>> nodes. I could not figure out how to install the exact first > > > >>> kernel version on the updated nodes, but I noticed that when I > > > >>> reinstall 5.15.0 with this tool: > > > >>> https://github.com/pimlie/ubuntu-mainline-kernel.sh, the error > > > >>> message disappears. However, once I do that, the network driver > > > >>> does not function properly anymore, so this does not seem to be > > > >>> a good solution. > > > >>> > > > >>> Has anyone seen this issue before or is there maybe something > > > >>> else I should take a look at? I am also happy to just find a > > > >>> workaround such that I can take these nodes back online. > > > >>> > > > >>> I appreciate any help! > > > >>> > > > >>> Thanks a lot in advance and best wishes, > > > >>> > > > >>> Tim > > > >>> > > > >>> > > > >>> > > > -- Cristóbal A. Navarro

2 1

slurm-config on NFS-volume
by Werf, C.G. van der (Carel) 24 Jan '24

24 Jan '24

Hi, Among other clusters, I have a simple cluster with 2 nodes, running slurm. 1 node runs : mysqld, slurmdbd, slurmctld and slurmd. The other node, only runs slurmd. Slurm config is in node1: /etc/slurm. A copy of the config is in node2:/etc/slurm. This slurm configuration runs ok. But, as I am normally used to configure on a shared location, where there is only 1 copy of the config, I tried to change the current setup by moving the slurm-config to a shared (NFS) disk. So on both nodes : /etc/slurm -> nfs-server:/shared/etc/slurm But, I have tried several ways... The slurmctld and slurmd daemons will not start when I refer to this "nfs-shared" location. Any ideas what is missing here ? Authorizations seem ok. With Regards, | Carel van der Werf | | Developer/Administrator Linux | ICT-Bèta | Department of Science | Utrecht University |

3 3