- slurm-users - lists.schedmd.com

srun only runs one job on a node
by Rike-Benjamin Schuppner 27 Jan '24

27 Jan '24

Hi, Not sure if I do not understand the scheduling algorithm correctly of if this is a bug. The bug seems similar to the one described and fixed here https://github.com/SchedMD/slurm/commit/6a2c99edbf96e50463cc2f16d8e5eb955c8… Given the node definition: NodeName=cluster01 CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=64283 NodeName=cluster02 CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=128814 PartitionName=DEFAULT MaxTime=2-0 MaxNodes=2 PartitionName=mini Nodes=cluster01,cluster02 State=UP and the batch script: for i in {1..6} do srun --exclusive --mem=100 --threads-per-core=1 -N1 -c1 -n1 bash -c 'echo $(hostname) $(date); sleep 10 ; echo done $(hostname) $(date)' & done wait When run with: time sbatch --wait --partition=mini --ntasks=10 --cpus-per-task=1 --mem=1000 mini.slurm I get the following log after 2m 18s: cluster01 Sat Jan 27 02:05:12 PM CET 2024 cluster01 Sat Jan 27 02:05:12 PM CET 2024 cluster01 Sat Jan 27 02:05:13 PM CET 2024 cluster01 Sat Jan 27 02:05:13 PM CET 2024 cluster02 Sat Jan 27 02:05:13 PM CET 2024 done cluster01 Sat Jan 27 02:05:22 PM CET 2024 done cluster01 Sat Jan 27 02:05:22 PM CET 2024 done cluster01 Sat Jan 27 02:05:23 PM CET 2024 done cluster01 Sat Jan 27 02:05:23 PM CET 2024 done cluster02 Sat Jan 27 02:05:23 PM CET 2024 srun: Job 650 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=650.6 cluster01 Sat Jan 27 02:07:14 PM CET 2024 done cluster01 Sat Jan 27 02:07:24 PM CET 2024 So apparently I am getting 4 sruns on cluster01 (why not 6 when I have 8 CPUs?), but only 1 is scheduled on cluster02. Then it is exhausted and waits endlessly long (almost 2 minutes?) before scheduling again. The step log tells me: slurmctld: debug: laying out the 1 tasks on 1 hosts cluster01 dist 1 slurmctld: STEPS: _step_alloc_lps: JobId=650 StepId=4 node 0 (cluster01) gres_cpus_alloc=0 tasks=1 cpus_per_task=1 slurmctld: STEPS: _pick_step_cores: step JobId=650 StepId=4 requires 1 cores on node 0 with cpus_per_core=1, available cpus from job: 8 slurmctld: STEPS: _pick_step_core: alloc Node:0 Socket:0 Core:3 slurmctld: STEPS: step alloc on job node 0 (cluster01) used 4 of 8 CPUs slurmctld: STEPS: _slurm_rpc_job_step_create: JobId=650 StepId=4 cluster01 usec=5502 slurmctld: STEPS: _pick_step_nodes: JobId=650 Currently running steps use 4 of allocated 8 CPUs on node cluster01 slurmctld: STEPS: _pick_step_nodes: JobId=650 Currently running steps use 1 of allocated 2 CPUs on node cluster02 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=0 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=1 has nodes cluster02 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=2 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=3 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=4 has nodes cluster01 slurmctld: STEPS: _pick_step_nodes: step pick 1-1 nodes, avail:cluster[01-02] idle: picked:NONE slurmctld: STEPS: _pick_step_nodes: step picked 0 of 1 nodes slurmctld: STEPS: Picked nodes cluster01 when accumulating from cluster01 slurmctld: debug: laying out the 1 tasks on 1 hosts cluster01 dist 1 slurmctld: STEPS: _step_alloc_lps: JobId=650 StepId=5 node 0 (cluster01) gres_cpus_alloc=0 tasks=1 cpus_per_task=1 slurmctld: STEPS: _pick_step_cores: step JobId=650 StepId=5 requires 1 cores on node 0 with cpus_per_core=1, available cpus from job: 8 slurmctld: STEPS: unable to pick step cores for job node 0 (cluster01): Requested nodes are busy slurmctld: STEPS: Deallocating 100MB of memory on node 0 (cluster01) now used: 400 of 1000 slurmctld: STEPS: step dealloc on job node 0 (cluster01) used: 4 of 8 CPUs slurmctld: STEPS: _slurm_rpc_job_step_create for JobId=650: Requested nodes are busy So why is it unable to allocate on cluster02? And why does its need to when it is only using 4 of 8 CPUs on cluster01 anyway? Now if I change to --ntasks=8, the algorithm doesn’t even schedule cluster02 at all: cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 srun: Job 651 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=651.6 cluster01 Sat Jan 27 02:17:35 PM CET 2024 srun: Job 651 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=651.7 cluster01 Sat Jan 27 02:17:36 PM CET 2024 done cluster01 Sat Jan 27 02:17:45 PM CET 2024 done cluster01 Sat Jan 27 02:17:46 PM CET 2024 Thanks for your help. Best /rike

1 0

LuaSQLite3 with slurm-lua-spank
by Jackson, Gary L. 26 Jan '24

26 Jan '24

What do I need to do to use the LuaSQLite3 library with Lua SPANK? For what it’s worth, I don’t know anything about Lua. -- Gary

1 0

sinfo: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host
by Michael Lewis 26 Jan '24

26 Jan '24

Hi All, I’m trying to get slurm-23.11.3 running on Ubuntu 20.04 and running on a stand alone system. I’m running into an issue I can not find the answer to. After compiling and installing when I fire up slurmctld and slurmd I get an error from sinfo: sinfo: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host sinfo: error: fetch_config: DNS SRV lookup failed sinfo: error: _establish_config_source: failed to fetch config sinfo: fatal: Could not establish a configuration source I looks like a DNS issue but the system has no issue resolving to its hostname or localhost. The slurm.conf file is also being read properly as I have the logs directed to a place convenient to me. I see lots have had these same issues but cannot find a clear resolution. I have slurm running on a stand alone system in another lab with and identical setup without issue. Any advice would be greatly appreciated. Thanks, Mike

2 2

Database cluster
by Daniel L'Hommedieu 26 Jan '24

26 Jan '24

Community: What do you do to ensure database reliability in your SLURM environment? We can have multiple controllers and multiple slurmdbds, but my understanding is that slurmdbd can be configured with a single MySQL server, so what do you do? Do you have that “single MySQL server” be a cluster, such as Percona XtraDB? Do you use MySQL replication, then manually switch to slurmdbd to a replication slave if the master goes down? Do you do something else? Thanks. Daniel

6 7

Problem using Podman with scrun on SLURM 23.11.3
by Marcus Lauer 25 Jan '24

25 Jan '24

I am getting an unusual error when trying to run Podman containers using scrun on SLURM 23.11.3 (and 23.11.1 previously). In short, Podman works when not configured to use scrun, but when configured to use scrun it fails. Podman gives this error: scrun: fatal: Unable to request job allocation: Job cannot be submitted without the current working directory specified. The slurmctld logs show this error: _slurm_rpc_allocate_resources: Job cannot be submitted without the current working directory specified. The scrun command does not appear to have a --chdir option so the working directory must be detected automatically, and this appears to be failing. Perhaps someone has encountered this or a similar error before? If anyone has seen this before I would love to hear about your experiences. A bit more detail in case anyone is interested. This is a two-node test system with one head node and one compute node, both running Rocky Linux 8.9. I have created /etc/containers/storage.conf and containers.conf files which are identical to the ones in the SLURM Containers Guide except that I set the rootless_storage_path to the same value as the graphroot in storage.conf. When I get rid of the /etc/containers/containers.conf file I can run containers on the head node. The backing store is ext4, though I have tried NFS (which had other problems). I have tried the vfs and overlay storage drivers but get the same results. On the SLURM side I created an oci.conf which is the same as the "oci.conf example for crun using run (suggested)" from the SLURM Container Guide. I'm using crun because I have forced cgroup v2 on both systems ( systemd.unified_cgroup_hierarchy=1 on the kernel cmdline) and crun seems to support cgroup v2 better. I am using an scrun.lua which is similar to the one in the scrun manual page with some paths modified for my setup. I should also mention that sbatch jobs run just fine and srun works as well. the test cluster seems to be working fine in general. -- Marcus Lauer Systems Administrator CETS Group, Research Support

1 0

Question about CPUs and cores
by Gestió Servidors 25 Jan '24

25 Jan '24

Hi, I want to run a simple test that uses one node and four cores. Also, in my script, I execute a binary that reports me in what core is running one of the four tasks. These are my files: * submit script: #!/bin/bash #SBATCH --job-name=test_jobs # Job name #SBATCH --output=test_job_%j.out # Output file (using array index) #SBATCH --error=test_job_%j.err # Error file (using array index) #SBATCH --ntasks=4 #SBATCH --ntasks-per-core=1 #SBATCH --cpus-per-task=1 # Number of CPU cores per task echo "START : $(date)" srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & # Wait for all jobs to finish before exiting the job submission script wait echo "END : $(date)" * check_cores.c file: #include <stdio.h> #include<unistd.h> struct pstat { int pid; //process id char tcomm[256];//filename of the executable char state[2]; //state (R is running, S is sleeping, D is sleeping in an //uninterruptible wait, Z is zombie, T is traced or stopped) int ppid;// process id of the parent process int pgrp;// pgrp of the process int sid;// session id int tty_nr;// tty the process uses int tty_pgrp;// pgrp of the tty int flags;// task flags int min_flt;// number of minor faults int cmin_flt;// number of minor faults with child's int maj_flt;// number of major faults int cmaj_flt;// number of major faults with child's int utime;// user mode jiffies int stime;// kernel mode jiffies int cutime;// user mode jiffies with child's int cstime;// kernel mode jiffies with child's int priority;// priority level int nice;// nice level int num_threads;// number of threads int it_real_value;// (obsolete, always 0) int start_time;// time the process started after system boot int vsize;// virtual memory size int rss;// resident set memory size int rsslim;// current limit in bytes on the rss int start_code;// address above which program text can run int end_code;// address below which program text can run int start_stack;// address of the start of the stack int esp;// current value of ESP int eip;// current value of EIP int pending;// bitmap of pending signals int blocked;// bitmap of blocked signals int sigign;// bitmap of ignored signals int sigcatch;// bitmap of catched signals int wchan;// address where process went to sleep int i0;// (place holder) int i1;// (place holder) int exit_signal;// signal to send to parent thread on exit int task_cpu;// which CPU the task is scheduled on int rt_priority;// realtime priority int policy;// scheduling policy (man sched_setscheduler) int blkio_ticks;// time spent waiting for block IO int gtime;// guest time of the task in jiffies int cgtime;// guest time of the task children in jiffies } p ; int main() { char name[256]; char state[8]; FILE* f = fopen("/proc/self/stat", "r"); char hostname[1024]; gethostname(hostname, 1024); fscanf(f, "%d%s%s%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d", &p.pid, &p.tcomm, &p.state, &p.ppid, &p.pgrp, &p.sid, &p.tty_nr, &p.tty_pgrp, &p.flags, &p.min_flt, &p.cmin_flt, &p.maj_flt, &p.cmaj_flt, &p.utime, &p.stime, &p.cutime, &p.cstime, &p.priority, &p.nice, &p.num_threads, &p.it_real_value, &p.start_time, &p.vsize, &p.rss, &p.rsslim, &p.start_code, &p.end_code, &p.start_stack, &p.esp, &p.eip, &p.pending, &p.blocked, &p.sigign, &p.sigcatch, &p.wchan, &p.i0, &p.i1, &p.exit_signal, &p.task_cpu, &p.rt_priority, &p.policy, &p.blkio_ticks, &p.gtime, &p.cgtime); sleep (15); printf("HOST %s\n", hostname); printf("CPU %d\n", p.task_cpu); return 0; } When I run this script, I should receive an output like this: HOST clus04.hpc.local CPU 0 HOST clus04.hpc.local CPU 7 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 6 because system has scheduled my job in node "clus04.hpc.local" (a node with two 6-cores processors) and has reserved cores 0-1 of each processor. However, the real output I receive is HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 6 (and others outputs where there are one CPU repeated), so it seems SLURMs is reserving correctly four cores but, then, process is running two or more times in the same core. Slurmctld.log file in server shows this information: [...] [2024-01-25T09:38:42.867] DEBUG: JobId=206129 node clus04 vpus 1 cpus 4 [2024-01-25T09:38:42.867] ==================== [2024-01-25T09:38:42.867] JobId=206129 nhosts:1 ncpus:4 node_req:1 nodes=clus04 [2024-01-25T09:38:42.867] Node[0]: [2024-01-25T09:38:42.867] Mem(MB):0:0 Sockets:2 Cores:6 CPUs:4:0 [2024-01-25T09:38:42.867] Socket[0] Core[0] is allocated [2024-01-25T09:38:42.867] Socket[0] Core[1] is allocated [2024-01-25T09:38:42.867] Socket[1] Core[0] is allocated [2024-01-25T09:38:42.867] Socket[1] Core[1] is allocated [2024-01-25T09:38:42.867] -------------------- [2024-01-25T09:38:42.867] cpu_array_value[0]:4 reps:1 [2024-01-25T09:38:42.867] ==================== [2024-01-25T09:38:42.867] DEBUG: Dump job_resources: nhosts 1 cb 0-1,6-7 [...] [2024-01-25T09:38:58.213] _job_complete: JobId=206129 WEXITSTATUS 0 [2024-01-25T09:38:58.213] deallocate_nodes: JobId=206129 [2024-01-25T09:38:58.213] DEBUG: Dump job_resources: nhosts 1 cb 0-1,6-7 [2024-01-25T09:38:58.213] _job_complete: JobId=206129 done And slurmd.log file in node clus04 shows this information: [...] [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: step physical cores are '0-1,6-7' It seems that SLURM is reserving me correctly four cores for my job but, when inside my script I run four "srun", then each srun can freely take one of the four reserved cores, not exactly one to each task. What am I doing wrong? Thanks a lot!

1 0

Slurm version 23.11.3 is now available
by Tim McMullan 24 Jan '24

24 Jan '24

We are pleased to announce the availability of Slurm version 23.11.3. The 23.11.3 fixes a single issue related to the new Debian package support. For 23.11.2, the debian changelog was not updated correctly and would generate packages as 23.11.1. This release will correctly generate 23.11.3 packages, with no other changes. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.3 > ========================== > -- Fix debian/changelog file to reflect the correct version so the packages > are generated correctl. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 24 Jan '24

24 Jan '24

Hi, I just tested with 23.02.7-1 and the issue is gone. So it seems like the patch got released. Best, Tim On 1/24/24 16:55, Stefan Fleischmann wrote: > On Wed, 24 Jan 2024 12:37:04 -0300 Cristóbal Navarro > <cristobal.navarro.g(a)gmail.com> wrote: >> Many thanks >> One question? Do we have to apply this patch (and recompile slurm i >> guess) only on the compute-node with problems? >> Also, I noticed the patch now appears as "obsolete", is that ok? > We have Slurm installed on a NFS share, so what I did was to recompile > it and then I only replaced the library lib/slurm/cgroup_v2.so. Good > enough for now, I've been planning to update to 23.11 anyway soon. > > I suppose it's marked as obsolete because the patch went into a > release. According to the info in the bug report it should have been > included in 23.02.4. > > Cheers, > Stefan > >> On Wed, Jan 24, 2024 at 9:52 AM Stefan Fleischmann <sfle(a)kth.se> >> wrote: >> >>> Turns out I was wrong, this is not a problem in the kernel at all. >>> It's a known bug that is triggered by long bpf logs, see here >>> https://bugs.schedmd.com/show_bug.cgi?id=17210 >>> >>> There is a patch included there. >>> >>> Cheers, >>> Stefan >>> >>> On Tue, 23 Jan 2024 15:28:59 +0100 Stefan Fleischmann <sfle(a)kth.se> >>> wrote: >>>> I don't think there is much for SchedMD to do. As I said since it >>>> is working fine with newer kernels there doesn't seem to be any >>>> breaking change in cgroup2 in general, but only a regression >>>> introduced in one of the latest updates in 5.15. >>>> >>>> If Slurm was doing something wrong with cgroup2, and it >>>> accidentally worked until this recent change, then other kernel >>>> versions should show the same behavior. But as far as I can tell >>>> it still works just fine with newer kernels. >>>> >>>> Cheers, >>>> Stefan >>>> >>>> On Tue, 23 Jan 2024 15:20:56 +0100 >>>> Tim Schneider <tim.schneider1(a)tu-darmstadt.de> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have filed a bug report with SchedMD >>>>> (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the >>>>> support told me they cannot invest time in this issue since I >>>>> don't have a support contract. Maybe they will look into it >>>>> once it affects more people or someone important enough. >>>>> >>>>> So far, I have resorted to using 5.15.0-89-generic, but I am >>>>> also a bit concerned about the security aspect of this choice. >>>>> >>>>> Best, >>>>> >>>>> Tim >>>>> >>>>> On 23.01.24 14:59, Stefan Fleischmann wrote: >>>>>> Hi! >>>>>> >>>>>> I'm seeing the same in our environment. My conclusion is that >>>>>> it is a regression in the Ubuntu 5.15 kernel, introduced with >>>>>> 5.15.0-90-generic. Last working kernel version is >>>>>> 5.15.0-89-generic. I have filed a bug report here: >>>>>> https://bugs.launchpad.net/bugs/2050098 >>>>>> >>>>>> Please add yourself to the affected users in the bug report >>>>>> so it hopefully gets more attention. >>>>>> >>>>>> I've tested with newer kernels (6.5, 6.6 and 6.7) and the >>>>>> problem does not exist there. 6.5 is the latest hwe kernel >>>>>> for 22.04 and would be an option for now. Reverting back to >>>>>> 5.15.0-89 would work as well, but I haven't looked into the >>>>>> security aspects of that. >>>>>> >>>>>> Cheers, >>>>>> Stefan >>>>>> >>>>>> On Mon, 22 Jan 2024 13:31:15 -0300 >>>>>> cristobal.navarro.g at gmail.com wrote: >>>>>> >>>>>>> Hi Tim and community, >>>>>>> We are currently having the same issue (cgroups not working >>>>>>> it seems, showing all GPUs on jobs) on a GPU-compute node >>>>>>> (DGX A100) a couple of days ago after a full update (apt >>>>>>> upgrade). Now whenever we launch a job for that partition, >>>>>>> we get the error message mentioned by Tim. As a note, we >>>>>>> have another custom GPU-compute node with L40s, on a >>>>>>> different partition, and that one works fine. Before this >>>>>>> error, we always had small differences in kernel version >>>>>>> between nodes, so I am not sure if this can be the problem. >>>>>>> Nevertheless, here is the info of our nodes as well. >>>>>>> >>>>>>> *[Problem node]* The DGX A100 node has this kernel >>>>>>> cnavarro at nodeGPU01:~$ uname -a >>>>>>> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 >>>>>>> 20:28:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> *[Functioning node]* The Custom GPU node (L40s) has this >>>>>>> kernel cnavarro at nodeGPU02:~$ uname -a >>>>>>> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 >>>>>>> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> *And the login node *(slurmctld) >>>>>>> ? ~ uname -a >>>>>>> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue >>>>>>> Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> Any ideas what we should check? >>>>>>> >>>>>>> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 >>>>>>> at tu-darmstadt.de> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am using SLURM 22.05.9 on a small compute cluster. Since I >>>>>>>> reinstalled two of our nodes, I get the following error when >>>>>>>> launching a job: >>>>>>>> >>>>>>>> slurmstepd: error: load_ebpf_prog: BPF load error (No space >>>>>>>> left on device). Please check your system limits (MEMLOCK). >>>>>>>> >>>>>>>> Also the cgroups do not seem to work properly anymore, as I >>>>>>>> am able to see all GPUs even if I do not request them, >>>>>>>> which is not the case on the other nodes. >>>>>>>> >>>>>>>> One difference I found between the updated nodes and the >>>>>>>> original nodes (both are Ubuntu 22.04) is the kernel >>>>>>>> version, which is "5.15.0-89-generic #99-Ubuntu SMP" on the >>>>>>>> functioning nodes and "5.15.0-91-generic #101-Ubuntu SMP" >>>>>>>> on the updated nodes. I could not figure out how to install >>>>>>>> the exact first kernel version on the updated nodes, but I >>>>>>>> noticed that when I reinstall 5.15.0 with this tool: >>>>>>>> https://github.com/pimlie/ubuntu-mainline-kernel.sh, the >>>>>>>> error message disappears. However, once I do that, the >>>>>>>> network driver does not function properly anymore, so this >>>>>>>> does not seem to be a good solution. >>>>>>>> >>>>>>>> Has anyone seen this issue before or is there maybe >>>>>>>> something else I should take a look at? I am also happy to >>>>>>>> just find a workaround such that I can take these nodes >>>>>>>> back online. >>>>>>>> >>>>>>>> I appreciate any help! >>>>>>>> >>>>>>>> Thanks a lot in advance and best wishes, >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> >>>>>>>> >>>> >>> >>

2 1

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Cristóbal Navarro 24 Jan '24

24 Jan '24

Many thanks One question? Do we have to apply this patch (and recompile slurm i guess) only on the compute-node with problems? Also, I noticed the patch now appears as "obsolete", is that ok? On Wed, Jan 24, 2024 at 9:52 AM Stefan Fleischmann <sfle(a)kth.se> wrote: > Turns out I was wrong, this is not a problem in the kernel at all. It's > a known bug that is triggered by long bpf logs, see here > https://bugs.schedmd.com/show_bug.cgi?id=17210 > > There is a patch included there. > > Cheers, > Stefan > > On Tue, 23 Jan 2024 15:28:59 +0100 Stefan Fleischmann <sfle(a)kth.se> > wrote: > > I don't think there is much for SchedMD to do. As I said since it is > > working fine with newer kernels there doesn't seem to be any breaking > > change in cgroup2 in general, but only a regression introduced in one > > of the latest updates in 5.15. > > > > If Slurm was doing something wrong with cgroup2, and it accidentally > > worked until this recent change, then other kernel versions should > > show the same behavior. But as far as I can tell it still works just > > fine with newer kernels. > > > > Cheers, > > Stefan > > > > On Tue, 23 Jan 2024 15:20:56 +0100 > > Tim Schneider <tim.schneider1(a)tu-darmstadt.de> wrote: > > > > > Hi, > > > > > > I have filed a bug report with SchedMD > > > (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the support > > > told me they cannot invest time in this issue since I don't have a > > > support contract. Maybe they will look into it once it affects more > > > people or someone important enough. > > > > > > So far, I have resorted to using 5.15.0-89-generic, but I am also a > > > bit concerned about the security aspect of this choice. > > > > > > Best, > > > > > > Tim > > > > > > On 23.01.24 14:59, Stefan Fleischmann wrote: > > > > Hi! > > > > > > > > I'm seeing the same in our environment. My conclusion is that it > > > > is a regression in the Ubuntu 5.15 kernel, introduced with > > > > 5.15.0-90-generic. Last working kernel version is > > > > 5.15.0-89-generic. I have filed a bug report here: > > > > https://bugs.launchpad.net/bugs/2050098 > > > > > > > > Please add yourself to the affected users in the bug report so it > > > > hopefully gets more attention. > > > > > > > > I've tested with newer kernels (6.5, 6.6 and 6.7) and the problem > > > > does not exist there. 6.5 is the latest hwe kernel for 22.04 and > > > > would be an option for now. Reverting back to 5.15.0-89 would work > > > > as well, but I haven't looked into the security aspects of that. > > > > > > > > Cheers, > > > > Stefan > > > > > > > > On Mon, 22 Jan 2024 13:31:15 -0300 > > > > cristobal.navarro.g at gmail.com wrote: > > > > > > > >> Hi Tim and community, > > > >> We are currently having the same issue (cgroups not working it > > > >> seems, showing all GPUs on jobs) on a GPU-compute node (DGX A100) > > > >> a couple of days ago after a full update (apt upgrade). Now > > > >> whenever we launch a job for that partition, we get the error > > > >> message mentioned by Tim. As a note, we have another custom > > > >> GPU-compute node with L40s, on a different partition, and that > > > >> one works fine. Before this error, we always had small > > > >> differences in kernel version between nodes, so I am not sure if > > > >> this can be the problem. Nevertheless, here is the info of our > > > >> nodes as well. > > > >> > > > >> *[Problem node]* The DGX A100 node has this kernel > > > >> cnavarro at nodeGPU01:~$ uname -a > > > >> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 > > > >> 20:28:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> *[Functioning node]* The Custom GPU node (L40s) has this kernel > > > >> cnavarro at nodeGPU02:~$ uname -a > > > >> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 > > > >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> *And the login node *(slurmctld) > > > >> ? ~ uname -a > > > >> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 > > > >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> Any ideas what we should check? > > > >> > > > >> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 at > > > >> tu-darmstadt.de> wrote: > > > >> > > > >>> Hi, > > > >>> > > > >>> I am using SLURM 22.05.9 on a small compute cluster. Since I > > > >>> reinstalled two of our nodes, I get the following error when > > > >>> launching a job: > > > >>> > > > >>> slurmstepd: error: load_ebpf_prog: BPF load error (No space left > > > >>> on device). Please check your system limits (MEMLOCK). > > > >>> > > > >>> Also the cgroups do not seem to work properly anymore, as I am > > > >>> able to see all GPUs even if I do not request them, which is not > > > >>> the case on the other nodes. > > > >>> > > > >>> One difference I found between the updated nodes and the > > > >>> original nodes (both are Ubuntu 22.04) is the kernel version, > > > >>> which is "5.15.0-89-generic #99-Ubuntu SMP" on the functioning > > > >>> nodes and "5.15.0-91-generic #101-Ubuntu SMP" on the updated > > > >>> nodes. I could not figure out how to install the exact first > > > >>> kernel version on the updated nodes, but I noticed that when I > > > >>> reinstall 5.15.0 with this tool: > > > >>> https://github.com/pimlie/ubuntu-mainline-kernel.sh, the error > > > >>> message disappears. However, once I do that, the network driver > > > >>> does not function properly anymore, so this does not seem to be > > > >>> a good solution. > > > >>> > > > >>> Has anyone seen this issue before or is there maybe something > > > >>> else I should take a look at? I am also happy to just find a > > > >>> workaround such that I can take these nodes back online. > > > >>> > > > >>> I appreciate any help! > > > >>> > > > >>> Thanks a lot in advance and best wishes, > > > >>> > > > >>> Tim > > > >>> > > > >>> > > > >>> > > > -- Cristóbal A. Navarro

2 1

slurm-config on NFS-volume
by Werf, C.G. van der (Carel) 24 Jan '24

24 Jan '24

Hi, Among other clusters, I have a simple cluster with 2 nodes, running slurm. 1 node runs : mysqld, slurmdbd, slurmctld and slurmd. The other node, only runs slurmd. Slurm config is in node1: /etc/slurm. A copy of the config is in node2:/etc/slurm. This slurm configuration runs ok. But, as I am normally used to configure on a shared location, where there is only 1 copy of the config, I tried to change the current setup by moving the slurm-config to a shared (NFS) disk. So on both nodes : /etc/slurm -> nfs-server:/shared/etc/slurm But, I have tried several ways... The slurmctld and slurmd daemons will not start when I refer to this "nfs-shared" location. Any ideas what is missing here ? Authorizations seem ok. With Regards, | Carel van der Werf | | Developer/Administrator Linux | ICT-Bèta | Department of Science | Utrecht University |

3 3

error: Couldn't find the specified plugin name for cred/munge looking at all files
by Jesse Aiton 24 Jan '24

24 Jan '24

Hello Slurm Folks, I have a weird issue where on the same server, which acts as both a controller and a node, slurmctld can’t find cred_munge.so slurmctld: debug3: Trying to load plugin /app/slurm-24.0.8/lib/slurm/cred_munge.so slurmctld: debug4: /app/slurm-24.0.8/lib/slurm/cred_munge.so: Does not exist or not a regular file. slurmctld: error: Couldn't find the specified plugin name for cred/munge looking at all files slurmctld: error: cannot open plugin directory /app/slurm-24.0.8/lib/slurm slurmctld: error: cannot find cred plugin for cred/munge slurmctld: error: cannot create cred context for cred/munge slurmctld: fatal: failed to initialize cred plugin But slurmd can: slurmd: debug3: Trying to load plugin /app/slurm-24.0.8/lib/slurm/cred_munge.so slurmd: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x180800 slurmd: cred/munge: init: Munge credential signature plugin loaded slurmd: debug3: Success. This is on Ubuntu 20.04 and happens both with Slurm 20.11.09 and 24.0.8 Thank you, Jesse # slurm.conf file generated by configurator easy.html. # Put this file on all nodes of your cluster. # See the slurm.conf man page for more information. # ClusterName=prod-cluster SlurmctldHost=controller # #MailProg=/bin/mail #MpiDefault= #MpiParams=ports=#-# ProctrackType=proctrack/cgroup ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid #SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid #SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm #SlurmdUser=root StateSaveLocation=/var/spool/slurmctld #SwitchType= TaskPlugin=task/affinity,task/cgroup # # # TIMERS #KillWait=30 #MinJobAge=300 #SlurmctldTimeout=120 #SlurmdTimeout=300 # # # SCHEDULING SchedulerType=sched/backfill SelectType=select/cons_tres # # # LOGGING AND ACCOUNTING #AccountingStorageType= #JobAcctGatherFrequency=30 #JobAcctGatherType= #SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld.log #SlurmdDebug=info SlurmdLogFile=/var/log/slurmd.log # # # COMPUTE NODES NodeName=controller CPUs=1 State=UNKNOWN NodeName=node CPUs=1 State=UNKNOWN PartitionName=prod-part Nodes=ALL Default=YES MaxTime=INFINITE State=UP

3 5

Slurm version 23.11.2 is now available
by Tim McMullan 23 Jan '24

23 Jan '24

We are pleased to announce the availability of Slurm version 23.11.2. The 23.11.2 release includes a number of fixes to stability and various bug fixes. Some notable changes include several fixes to the new scontrol reconfigure method, including one that could result in jobs getting cancelled prematurely, a couple errors that resulted in the backup slurmctld stopping on fail-back, and an issue during upgrades with older MySQL versions with a small max_allowed_packet value for sites with a large number of associations. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.2 > ========================== > -- slurmrestd - Reject single http query with multiple path requests. > -- Fix launching Singularity v4.x containers with srun --container by setting > .process.terminal to true in generated config.json when step has > pseudoterminal (--pty) requested. > -- Fix loading in dyanmic/cloud node jobs after net_cred expired. > -- Fix cgroup null path error on slurmd/slurmstepd tear down. > -- data_parser/v0.0.40 - Prevent failure if accounting is disabled, instead > issue a warning if needed data from the database can not be retrieved. > -- openapi/slurmctld - Prevent failure if accounting is disabled. > -- Prevent slurmscriptd processing delays from blocking other threads in > slurmctld while trying to launch various scripts. This is additional work > for a fix in 23.02.6. > -- Fix memory leak when receiving alias addrs from controller. > -- scontrol - Accept `scontrol token lifespan=infinite` to create tokens that > effectively do not expire. > -- Avoid errors when Slurmdb accounting disabled when '--json' or '--yaml' is > invoked with CLI commands and slurmrestd. Add warnings when query would > have populated data from Slurmdb instead of errors. > -- Fix slurmctld memory leak when running job with --tres-per-task=gres:shard:# > -- Fix backfill trying to start jobs outside of backfill window. > -- Fix oversubscription on partitions with PreemptMode=OFF. > -- Preserve node reason on power up if the node is downed or drained. > -- data_parser/v0.0.40 - Avoid aborting when invoking a not implemented > parser. > -- data_parser/v0.0.40 - Fix how nice values are parsed for job submissions. > -- data_parser/v0.0.40 - Fix regression where parsing error did not result in > invalid request being rejected. > -- Fix segfault in front-end node registration. > -- Prevent jobs using none typed gpus from being killed by the controller after > a reconfig or restart. > -- Fix deadlock situation in the dbd when adding associations. > -- Update default values of text/blob columns when updating from old mysql > versions in more situations. This improves a previous fix to handle an > uncommon case when upgrading mysql/mariadb. > -- Fix rpmbuild in openSUSE/SLES due to incorrect mariadb dependency. > -- Fix compilation on RHEL 7. > -- When upgrading the slurmdbd to 23.11, avoid generating a query to update > the association table that is larger than max_allowed_packet which would > result in an upgrade failure. > -- Fix rare deadlock when a dynamic node registers at the same time that a > once per minute background task occurs. > -- Fix build issue on 32-bit systems. > -- data_parser/v0.0.40 - Fix enumerated strings in OpenAPI specification not > have type field specified. > -- Improve scontrol show job -d information of used shared gres (shard/mps) > topology. > -- Allow Slurm to compile without MUNGE if --without-munge is used as an > argument to configure. > -- accounting_storage/mysql - Fix usage query to use new lineage column > instead of lft/rgt. > -- slurmrestd - Improve handling of missing parsers when content plugins > expect parsers not loaded. > -- slurmrestd - Correct parsing of StepIds when querying jobs. > -- slurmrestd - Improve error from parsing failures of lists. > -- slurmrestd - Improve parsing of singular values for lists. > -- accounting_storage/mysql - Fix PrivateData=User when listing associations. > -- Disable sorting of dynamic nodes to avoid issues when restarting with > heterogenous jobs that cause jobs to abort on restart. > -- Don't allow deletion of non-dynamic nodes. > -- accounting_storage/mysql - Fix issue adding partition based associations. > -- Respect non-"slurm" settings for I_MPI_HYDRA_BOOTSTRAP and HYDRA_BOOTSTRAP > and avoid injecting the --external-launcher option which will cause > mpirun/mpiexec to fail with an unexpected argument error. > -- Fix bug where scontrol hold would change node count for jobs with > implicitly defined node counts. > -- data_parser/v0.0.40 - Fix regression of support for "hold" in > job description. > -- Avoid sending KILL RPCs to unresolvable POWERING_UP and POWERED_DOWN nodes. > -- data_parser/v0.0.38 - Fix several potential NULL dereferences that could > cause slurmrestd to crash. > -- Add --gres-flags=one-task-per-sharing. Do not allow different tasks in to be > allocated shared gres from the same sharing gres. > -- Add SelectTypeParameters=ENFORCE_BINDING_GRES and ONE_TASK_PER_SHARING_GRES. > This gives default behavior for a job's --gres-flags. > -- Alter the networking code to try connecting to the backup controllers > if the DNS lookup for the primary SlurmctldHost fails. > -- Alter the name resolution to only log at verbose() in client commands > on failures. This allows for HA setups where the DNS entries are withdrawn > for some SlurmctldHost entries without flooding the user with errors. > -- Prevent slurmscriptd PID leaks when running slurmctld in foreground mode. > -- Open all slurmctld listening ports at startup, and persist throughout. > This also changes the backup slurmctld process to open the SlurmctldPort > range, instead of only the first. > -- Fix backup slurmctld shutting down instead of resuming standby duty if it > took control. > -- Fix race condition that delayed the primary slurmctld resuming when taking > control from a backup controller. > -- srun - Ensure processed messages are meant for this job in case of a > rapidly-reused TCP port. > -- srun - Prevent step launch failure while waiting for step allocation if > a stray message is received. > -- Fix backup slurmctld to be able to respond to configless config file > requests correctly. > -- Fix slurmctld crashing when recovering from a failed reconfigure. > -- Fix slurmscriptd operation after recovering from a failed reconfigure. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

GPU devices mapping with job's cgroup in cgroups v2 using eBPF
by Mahendra Paipuri 23 Jan '24

23 Jan '24

Hello all, Happy new year! We have recently upgraded the cgroups on our SLURM cluster to v2. In cgroups v1, the interface `/devices.list` used to have the information of which device has been attached to that particular cgroup. From my understanding, cgroups v2 use eBPF to manage devices and so as SLURM to manage the GPUs. I was looking for a way to be able to programatically determine the job cgroups to device mapping and I came across this thread ( https://bugzilla.redhat.com/show_bug.cgi?id=1717396) which has a similar discussion in the context of VMs. So, I have used `bpftool` to inspect the job cgroups. An example output: ``` # /tmp/bpftool cgroup list /sys/fs/cgroup/system.slice/slurmstepd.scope/job_1956132 ID AttachType AttachFlags Name ``` When I add `effective` flag, I see the attached eBPF program ``` # /tmp/bpftool cgroup list /sys/fs/cgroup/system.slice/slurmstepd.scope/job_1956132 effective ID AttachType Name 4197 cgroup_device Slurm_Cgroup_v2 ``` >From my understand, `effective` flag shows the inherited eBPF programs as well. So, my question is at which level of cgroups the eBPF program is attached? I tried to inspect at various levels but all of them returned none. Then looking into translated byte code of eBPF program, I get the following ``` # /tmp/bpftool prog dump xlated id 4197 0: (61) r2 = *(u32 *)(r1 +0) 1: (54) w2 &= 65535 2: (61) r3 = *(u32 *)(r1 +0) 3: (74) w3 >>= 16 4: (61) r4 = *(u32 *)(r1 +4) 5: (61) r5 = *(u32 *)(r1 +8) 6: (55) if r2 != 0x2 goto pc+4 7: (55) if r4 != 0xc3 goto pc+3 8: (55) if r5 != 0x0 goto pc+2 9: (b7) r0 = 0 10: (95) exit 11: (55) if r2 != 0x2 goto pc+4 12: (55) if r4 != 0xc3 goto pc+3 13: (55) if r5 != 0x1 goto pc+2 14: (b7) r0 = 0 15: (95) exit 16: (55) if r2 != 0x2 goto pc+4 17: (55) if r4 != 0xc3 goto pc+3 18: (55) if r5 != 0x2 goto pc+2 19: (b7) r0 = 0 20: (95) exit 21: (55) if r2 != 0x2 goto pc+4 22: (55) if r4 != 0xc3 goto pc+3 23: (55) if r5 != 0x3 goto pc+2 24: (b7) r0 = 1 25: (95) exit 26: (b7) r0 = 1 27: (95) exit ``` >From the output, it is clear that GPU:3 (among 0,1,2,3) is the one that is attached to that job's cgroup. However, I was looking for a way to dump eBPF maps that can directly provide the major, minor numbers and permissions of device as discussed in the comment (https://bugzilla.redhat.com/show_bug.cgi?id=1717396#c5). When I inspect eBPF program, I dont see any maps associated. ``` # /tmp/bpftool prog list id 4197 4197: cgroup_device name Slurm_Cgroup_v2 tag 1a261c8a913ff67c gpl loaded_at 2024-01-02T08:19:56+0100 uid 0 xlated 224B jited 142B memlock 4096B ``` So, my second question is how can I get a similar information as `map dump` that can give us the device's major minor numbers directly instead of parsing the byte code from `prog dump`? I am still discovering the eBPF ecosystem so if I am missing something very obvious, please let me know. I would really appreciate that. Cheers! Regards Mahendra

2 1

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 23 Jan '24

23 Jan '24

Hi, I have filed a bug report with SchedMD (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the support told me they cannot invest time in this issue since I don't have a support contract. Maybe they will look into it once it affects more people or someone important enough. So far, I have resorted to using 5.15.0-89-generic, but I am also a bit concerned about the security aspect of this choice. Best, Tim On 23.01.24 14:59, Stefan Fleischmann wrote: > Hi! > > I'm seeing the same in our environment. My conclusion is that it is a > regression in the Ubuntu 5.15 kernel, introduced with 5.15.0-90-generic. > Last working kernel version is 5.15.0-89-generic. I have filed a bug > report here: https://bugs.launchpad.net/bugs/2050098 > > Please add yourself to the affected users in the bug report so it > hopefully gets more attention. > > I've tested with newer kernels (6.5, 6.6 and 6.7) and the problem does > not exist there. 6.5 is the latest hwe kernel for 22.04 and would be an > option for now. Reverting back to 5.15.0-89 would work as well, but I > haven't looked into the security aspects of that. > > Cheers, > Stefan > > On Mon, 22 Jan 2024 13:31:15 -0300 > cristobal.navarro.g at gmail.com wrote: > >> Hi Tim and community, >> We are currently having the same issue (cgroups not working it seems, >> showing all GPUs on jobs) on a GPU-compute node (DGX A100) a couple >> of days ago after a full update (apt upgrade). Now whenever we launch >> a job for that partition, we get the error message mentioned by Tim. >> As a note, we have another custom GPU-compute node with L40s, on a >> different partition, and that one works fine. >> Before this error, we always had small differences in kernel version >> between nodes, so I am not sure if this can be the problem. >> Nevertheless, here is the info of our nodes as well. >> >> *[Problem node]* The DGX A100 node has this kernel >> cnavarro at nodeGPU01:~$ uname -a >> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 20:28:30 >> UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> *[Functioning node]* The Custom GPU node (L40s) has this kernel >> cnavarro at nodeGPU02:~$ uname -a >> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 >> UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> *And the login node *(slurmctld) >> ? ~ uname -a >> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> Any ideas what we should check? >> >> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 at >> tu-darmstadt.de> wrote: >> >>> Hi, >>> >>> I am using SLURM 22.05.9 on a small compute cluster. Since I >>> reinstalled two of our nodes, I get the following error when >>> launching a job: >>> >>> slurmstepd: error: load_ebpf_prog: BPF load error (No space left on >>> device). Please check your system limits (MEMLOCK). >>> >>> Also the cgroups do not seem to work properly anymore, as I am able >>> to see all GPUs even if I do not request them, which is not the >>> case on the other nodes. >>> >>> One difference I found between the updated nodes and the original >>> nodes (both are Ubuntu 22.04) is the kernel version, which is >>> "5.15.0-89-generic #99-Ubuntu SMP" on the functioning nodes and >>> "5.15.0-91-generic #101-Ubuntu SMP" on the updated nodes. I could >>> not figure out how to install the exact first kernel version on the >>> updated nodes, but I noticed that when I reinstall 5.15.0 with this >>> tool: https://github.com/pimlie/ubuntu-mainline-kernel.sh, the >>> error message disappears. However, once I do that, the network >>> driver does not function properly anymore, so this does not seem to >>> be a good solution. >>> >>> Has anyone seen this issue before or is there maybe something else I >>> should take a look at? I am also happy to just find a workaround >>> such that I can take these nodes back online. >>> >>> I appreciate any help! >>> >>> Thanks a lot in advance and best wishes, >>> >>> Tim >>> >>> >>>

2 1

Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres)
by Kherfani, Hafedh (Professional Services, TC) 23 Jan '24

23 Jan '24

Hello Experts, I'm a new Slurm user (so please bare with me :) ...). Recently we've deployed Slurm version 23.11 on a very simple cluster, which consists of a Master node (acting as a Login & Slurmdbd node as well), a Compute Node which has a NVIDIA HGX A100-SXM4-40GB GPU, detected as 4 x GPU's: GPU [0-4], and a Storage Array presenting/sharing the NFS disk (where users' home directories will be created as well). The problem is that I've never been able to run a simple/dummy batch script in a parallel way using the 4 GPU's. In fact, running the same command "sbatch gpu-job.sh" multiple times shows that only one single job is running, while the other jobs are in a pending state: [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 214 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 215 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 216 [slurmtest@c-a100-master test-batch-scripts]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 216 gpu gpu-job slurmtest PD 0:00 1 (None) 215 gpu gpu-job slurmtest PD 0:00 1 (Priority) 214 gpu gpu-job slurmtest PD 0:00 1 (Priority) 213 gpu gpu-job slurmtest PD 0:00 1 (Priority) 212 gpu gpu-job slurmtest PD 0:00 1 (Resources) 211 gpu gpu-job slurmtest R 0:14 1 c-a100-cn01 PS: CPU jobs (i.e. using the default debug partition, without call the GPU Gres) can be run in parallel. The issue with running parallel jobs is only seen when using the GPU's as Gres. I've tried many combinations of settings in gres.conf and slurm.conf, many (if not most) of these combinations would result in error messages in slurmctld and slurmd logs. The current gres.conf and slurm.conf contents is shown below. Even though it doesn't give errors when restarting slurmctld and slurmd services (on master and compute nodes, resp.), but as I said, it doesn't allow jobs to be executed in parallel. Batch script contents shared below as well, in order to give more clarity on what I'm trying to do: [root@c-a100-master slurm]# cat gres.conf | grep -v "^#" NodeName=c-a100-cn01 AutoDetect=nvml Name=gpu Type=A100 File=/dev/nvidia[0-3] [root@c-a100-master slurm]# cat slurm.conf | grep -v "^#" | egrep -i "AccountingStorageTRES|GresTypes|NodeName|partition" GresTypes=gpu AccountingStorageTRES=gres/gpu NodeName=c-a100-cn01 Gres=gpu:A100:4 CPUs=64 Boards=1 SocketsPerBoard=1 CoresPerSocket=32 ThreadsPerCore=2 RealMemory=515181 State=UNKNOWN PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP PartitionName=gpu Nodes=ALL MaxTime=10:0:0 [slurmtest@c-a100-master test-batch-scripts]$ cat gpu-job.sh #!/bin/bash #SBATCH --job-name=gpu-job #SBATCH --partition=gpu #SBATCH --nodes=1 #SBATCH --gpus-per-node=4 #SBATCH --gres=gpu:4 #SBATCH --tasks-per-node=1 #SBATCH --output=gpu_job_output.%j # Output file name (replaces %j with job ID) #SBATCH --error=gpu_job_error.%j # Error file name (replaces %j with job ID) hostname date sleep 40 pwd Any help on which changes need to be made to the config files (mainly slurm.conf and gres.cong) and/or the batch script, so that multiple jobs can be in a "Running" state at the same time (in parallel) ? Thanks in advance for your help ! Best regards, Hafedh Kherfani

8 10

propose environment variables SLURM_STDOUT, SLURM_STDERR, SLURM_STDIN
by urbanjost 22 Jan '24

22 Jan '24

RE: Placing the full pathname of the job stdout in an environment variable Would others find it useful if new variables were added that contained the full pathnames of the standard input, error and input files of batch jobs? ## SYNOPSIS Proposed new environment variables SLURM_STDOUT,SLURM_STDERR, and SLURM_STDIN pointing to the full pathnames of the output, error output, and input files of a task would be a useful feature for batch jobs in particular. ## PROBLEM There are cases where it is desirable to have a job process the standard files (stdin, stdout, stderr) from a batch job from within the job instead of later via an epilog or via post-processing. Just a few examples where the job might want to reference the full pathname of stdout: * copy the file to a remote machine (via scp(1), for example) * move the file from local or scratch space to permanent globally accessible storage * remove the file depending on completion status * mail the output * archive it * post-process it Slurm commands like scontrol(1) and squeue(1) can help you locate the file names. But the names returned do not always expand all the macros allowed when specifying the names, and require calling a Slurm command (which might overload Slurm if millions of jobs are submitted or cause problems if Slurm is not responding for any reason(?)). For example: #!/bin/bash # Minimalist submit script for sbatch(1) #SBATCH --nodes 1-1 --ntasks=1 --time 0-0:1:00 --chdir=/tmp #SBATCH --output=stdout.%j #SBATCH --error=stderr.%A:%a:%J:%j:%N:%n:%s:%t:%u.out # query filenames via squeue(1) export SLURM_STDOUT=$(squeue --noheader --Format=STDOUT: --job=$SLURM_JOBID) export SLURM_STDERR=$(squeue --noheader --Format=STDERR: --job=$SLURM_JOBID) # PS: Current documentation for squeue(1) does not explicitly tell the # user a null size works. # query filenames via scontrol(1) declare -x $(scontrol show job=$SLURM_JOBID|grep StdOut) declare -x $(scontrol show job=$SLURM_JOBID|grep StdErr) cat <<EOF SLURM_STDOUT=$SLURM_STDOUT SLURM_STDERR=$SLURM_STDERR StdOut=$StdOut StdErr=$StdErr EOF ls stdout.* stderr.* The resulting output shows either none or some macros are expanded by the commands (and I am not sure all commands will always return a full pathname(?)) SLURM_STDOUT=/home/urbanjs/venus/V600/stdout.%j SLURM_STDERR=/home/urbanjs/venus/V600/stderr.%A:%a:%J:%j:%N:%n:%s:%t:%u.out StdOut=/home/urbanjs/venus/V600/stdout.96 StdErr=/home/urbanjs/venus/V600/stderr.96:4294967294:%J:96:%N:%n:%s:%t:urbanjs.out stderr.96:4294967294:96:96:mercury:0:4294967294:0:urbanjs.out stdout.96 One currently available work-around would be that the user just avoid the filename macros and always specify the filenames using a standard convention (which would obviously have to be specific to a specific platform). This is error-prone as it requires the user to strictly follow a protocol where the pathnames are specified that can be overridden by command-line options on sbatch(1) and so-on. Alternatively instead of the new environment variables Slurm could have a command option that returns the fully expanded names always expanding the macros. But if you query Slurm you have to worry about scaleability. Is it OK for a 100 000, jobs to query Slurm simulataneously via squeue(1) or scontrol(1)? What happens if for external reasons the Slurm daemons are down or not responding? As a current work-around using commands like realpath(1), stat(1), ls(1),find(1),and getting pids with fuser(1), pidof(1), and ps(1) become attractive. I find using the realpath(1) command at the top of the job and $SLURM_TASK_ID works the best. realpath(1) may not be available on all platforms(?) as well as the /proc/$PID/fd/1 file, in which case the other commands might be used to the same effect but it would be nice to have a simple standard way that does not require calling a command. The proposed environment variables seem the most obvious solution. # do this early in the job in case the user changes or closes # file descriptor 1 ... export SLURM_STDOUT=$(realpath /proc/$SLURM_TASK_PID/fd/1) # create softlink OUTPUT to stdout of job ln -s -f $SLURM_STDOUT OUTPUT # now user can do things like mailx(1) or scp(1) the OUTPUT file. Other reasons I might want to know the stdout pathname are to minimize network traffic and filespace on central servers. I might specify the output of a job goes to scratch or local storage or a memory-resident file system and move it all to a location on another system or a long-term archive instead of a central area that might be filled by other jobs or that I have a quota on, etc. So since the stdout might not be easily accessible from my other sessions also having something like "scontrol write batch_stdout=$SLURM_JOBID" that shows the stdout of the job would also be useful. Note the LSF job scheduler bpeek(1) command allows for a tail(1)-like interface to the job stdout to support the stdout not being accessible from all system access points, just as described. The current situation where in typical usage the output file is often in a globally mounted area and I can just access it from other nodes with grep(1), tail(1), and so on covers many users needs in simple cluster configurations; but there are several scenarios as described above primarily encountered by people running many thousands of single-node jobs where assigning output to a local scratch device and then selectively processing at job termination is preferable. As mentioned, those users can specify such a local name with "#SBATCH --output" and then, knowing it, process it as desired but having something like $SLURM_STDOUT being the full pathname to the file is much more generic. ## SUMMARY So adding the variables creates an ability for users to reliably access the standard files of a job with a low-load scalable method; and being a SLURM_* module it could be much more reliably accessed via prolog and epilog scripts and module(1) scripts as well. ## PS: Related to this would be instead of just --chdir a --mkdir and --scratch #SBATCH option would be useful that expanded file macros that would be similar to --chdir but would make sure the directory existed (using only user privelages) and a --scratch that would do the same as --mkdir but remove the directory at job termination. urbanjs@mercury:~/SLURM/proposals$ Sent with [Proton Mail](https://proton.me/) secure email.

3 2

slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 22 Jan '24

22 Jan '24

Hi, I am using SLURM 22.05.9 on a small compute cluster. Since I reinstalled two of our nodes, I get the following error when launching a job: slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK). Also the cgroups do not seem to work properly anymore, as I am able to see all GPUs even if I do not request them, which is not the case on the other nodes. One difference I found between the updated nodes and the original nodes (both are Ubuntu 22.04) is the kernel version, which is "5.15.0-89-generic #99-Ubuntu SMP" on the functioning nodes and "5.15.0-91-generic #101-Ubuntu SMP" on the updated nodes. I could not figure out how to install the exact first kernel version on the updated nodes, but I noticed that when I reinstall 5.15.0 with this tool: https://github.com/pimlie/ubuntu-mainline-kernel.sh, the error message disappears. However, once I do that, the network driver does not function properly anymore, so this does not seem to be a good solution. Has anyone seen this issue before or is there maybe something else I should take a look at? I am also happy to just find a workaround such that I can take these nodes back online. I appreciate any help! Thanks a lot in advance and best wishes, Tim

2 1

Tried setting up GANG scheduling for timeslicing, but jobs are not alternating
by Francisco José Letterio 22 Jan '24

22 Jan '24

I'm trying to set up GANG scheduling with Slurm on my single-node server so the people at the lab can run experiments without blocking each other (so if say someone has to run some code that takes days to finish, other jobs that take less have the chance to run alternated with it and so they don't have to wait days until they can run) I followed the GANG scheduling slurm.conf setup tutorial on the slurm website, and to check if it's working properly I launched a bunch of jobs that print the current time and then sleep for a while. But when I check squeue, the jobs never alternate, they run sequentially one after the other. Why is this happening? Here's my slurm.conf file: # See the slurm.conf man page for more information. ClusterName=localcluster SlurmctldHost=localhost ProctrackType=proctrack/linuxproc ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/lib/slurm/slurmd SlurmUser=slurm StateSaveLocation=/var/lib/slurm/slurmctld TaskPlugin=task/none # TIMERS InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 # SCHEDULING #Seteado para que cada laburo alterne cada 15 segundos SchedulerTimeSlice=15 SchedulerType=sched/builtin SelectType=select/linear SelectTypeParameters=CR_Memory PreemptMode=GANG # LOGGING AND ACCOUNTING JobCompType=jobcomp/none JobAcctGatherFrequency=30 SlurmctldDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurm/slurmd.log # COMPUTE NODES NodeName=lab04 CPUs=48 CoresPerSocket=12 ThreadsPerCore=2 State=UNKNOWN RealMemory=257249 PartitionName=LocalQ Nodes=ALL Default=YES MaxTime=INFINITE State=UP OverSubscribe=FORCE:6 DefMemPerNode=257249 MaxMemPerNode=257249 >From what I understand, SchedulerTimeSlice=15 means that jobs should alternate running every 15 seconds. This is the job I'm launching with sbatch (launching many copies of this job one after the other): #!/bin/bash #SBATCH -J test # Job name #SBATCH -o job.%j.out # Name of stdout output file (%j expands to %jobId) #SBATCH -N 1 # Total number of nodes requested echo "Test output from Slurm Testjob" date sleep 10 date sleep 10 date sleep 10 date sleep 10 I would expect jobs to print one or two dates, then the Slurm scheduler comes and lets another job run in the meantime, and then the final print(s) come with a delay way greater than 10secs However after launching many copies of this job this is what I see on squeue : Before the first job launched is done: JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 18 LocalQ test eiarussi PD 0:00 1 (Resources) 19 LocalQ test eiarussi PD 0:00 1 (Priority) 20 LocalQ test eiarussi PD 0:00 1 (Priority) 21 LocalQ test eiarussi PD 0:00 1 (Priority) 17 LocalQ test eiarussi R 0:31 1 lab04 After that job ends: JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 19 LocalQ test eiarussi PD 0:00 1 (Resources) 20 LocalQ test eiarussi PD 0:00 1 (Priority) 21 LocalQ test eiarussi PD 0:00 1 (Priority) 18 LocalQ test eiarussi R 0:02 1 lab04 Why did the first job run for 30+ seconds uninterrupted, instead of another job being allowed to run after 15 seconds? Am I misunderstanding how GANG scheduling works, or is the a problem in my conf file?

1 0

MIG-Slice: Unavailable GRES
by Dražen Jalšovec 22 Jan '24

22 Jan '24

Hi, We are testing the MIG deployment on our new slurm compute node with 4 x H100 GPUs. It looks like everything is configured correctly but we have a problem accessing mig devices. When I submit jobs requesting a mig gpu device #SBATCH --gres=gpu:H100_1g.10gb:1, the jobs get submitted to the node, but only 4 jobs get executed and all other jobs fail. I was able to "solve it" by adding code to submit to the mig device UUID. Am I missing something in the slurm/gres/cgroups configuration that would automatically assign correct CUDA_VISIBLE_DEVICES MIG UUID to the job? If I submit 10 jobs requesting --gres=gpu:H100_1g.10gb:1 8 jobs start running: #1 - CUDA_VISIBLE_DEVICES=0 - jobs 1-4 run on one of the mig devices configured on GPU-device ... #8 - CUDA_VISIBLE_DEVICES=7 - jobs 5-8 fail with the message that there are no available CUDA_VISIBLE_DEVICES ... #10 - CUDA_VISIBLE_DEVICES=? - after waiting for free mig resources jobs 9-10 runs or fails depending if GPU-device is free (migs are free) $ nvidia-smi --query-gpu=uuid --id=0 --format=csv,noheader GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3 $ nvidia-smi --query-gpu=uuid --id=7 --format=csv,noheader No devices were found The slurm assigns CUDA_VISIBLE_DEVICES numbers based on the requested resources: Request: gpu:H100_1g.10gb:1 - CUDA_VISIBLE_DEVICES=0 gpu:H100_1g.10gb:2 - CUDA_VISIBLE_DEVICES=0,1 ... Any job with H100_1g.10gb request is going to get device number between 0 and 7 gpu:H100_1g.10gb:8 - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 gpu:H100_2g.20gb:1 - CUDA_VISIBLE_DEVICES=8 Any job with H100_2g.20gb request is going to get device number between 8 and 10 ... gpu:H100_2g.10gb:3 - CUDA_VISIBLE_DEVICES=8,9,10 gpu:H100_3g.40gb:1 - CUDA_VISIBLE_DEVICES=11 gpu:H100_3g.40gb:2 - CUDA_VISIBLE_DEVICES=12 gpu:H100_7g.80gb:1 - CUDA_VISIBLE_DEVICES=13 I was able to "solve it" by adding this code to my slurm script: # Get the CUDA index using srun and awk CUDA_INDEX=$(srun env | grep CUDA_VISIBLE_DEVICES | awk -F '=' '{print $2}') ## GPU0 1g.10gb ## MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a ## MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664 ## MIG-d3ab0675-d318-5e53-b487-b50695cf2e00 ## MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0 ## MIG-85dd76da-c994-5830-adf0-467c66ae1b95 ## MIG-29a6d43b-882e-5b79-868a-15bb2c770b82 ## MIG-558debd2-dc13-5406-9256-73ef4f279737 ## GPU1 2g.20gb ## MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f ## MIG-0a804c0c-aa27-5993-97cb-cedb854735ce ## MIG-8af9af6d-8720-5763-81e2-83afc43eb42b ## GPU1 1g.10gb ## MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a ## GPU2 3g.40gb ## MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b ## MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc ## GPU3 7g.80gb ## MIG-ec043869-9176-577e-bac0-46c8411e4e37 # Define the list of UUIDs UUIDS=( "MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a" "MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664" "MIG-d3ab0675-d318-5e53-b487-b50695cf2e00" "MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0" "MIG-85dd76da-c994-5830-adf0-467c66ae1b95" "MIG-29a6d43b-882e-5b79-868a-15bb2c770b82" "MIG-558debd2-dc13-5406-9256-73ef4f279737" "MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a" "MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f" "MIG-0a804c0c-aa27-5993-97cb-cedb854735ce" "MIG-8af9af6d-8720-5763-81e2-83afc43eb42b" "MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b" "MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc" "MIG-ec043869-9176-577e-bac0-46c8411e4e37" ) # Assign the UUID based on the CUDA index SELECTED_UUID=${UUIDS[$CUDA_INDEX]} # Print the selected UUID echo $SELECTED_UUID export CUDA_VISIBLE_DEVICES=$SELECTED_UUID echo "CUDA_VISIBLE_DEVICES set to: "$SELECTED_UUID echo "Test: "$CUDA_VISIBLE_DEVICES CUDA_VISIBLE_DEVICES=$SELECTED_UUID python3.8 gpu_script.py Result: Script start MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a CUDA_VISIBLE_DEVICES set to: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a Matrix calculation on CUDA device completed successfully. Script end +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 7 0 28686 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 8 0 28744 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 9 0 28300 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 10 0 28512 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 11 0 28506 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 12 0 28516 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 13 0 28511 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 1 9 0 28552 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | +---------------------------------------------------------------------------------------+ This is the current setup: RockyLinux 9.2 slurm-22.05.9-1.el9.x86_64 CUDA Version: 12.1 Driver Version: 530.30.02 # nvidia-smi -L GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3) MIG 1g.10gb Device 0: (UUID: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a) MIG 1g.10gb Device 1: (UUID: MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664) MIG 1g.10gb Device 2: (UUID: MIG-d3ab0675-d318-5e53-b487-b50695cf2e00) MIG 1g.10gb Device 3: (UUID: MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0) MIG 1g.10gb Device 4: (UUID: MIG-85dd76da-c994-5830-adf0-467c66ae1b95) MIG 1g.10gb Device 5: (UUID: MIG-29a6d43b-882e-5b79-868a-15bb2c770b82) MIG 1g.10gb Device 6: (UUID: MIG-558debd2-dc13-5406-9256-73ef4f279737) GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1439a39c-948b-0657-98e9-aff8595a8729) MIG 2g.20gb Device 0: (UUID: MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f) MIG 2g.20gb Device 1: (UUID: MIG-0a804c0c-aa27-5993-97cb-cedb854735ce) MIG 2g.20gb Device 2: (UUID: MIG-8af9af6d-8720-5763-81e2-83afc43eb42b) MIG 1g.10gb Device 3: (UUID: MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a) GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-3b182fed-bfdf-bc11-d1bd-c41fd1468d2a) MIG 3g.40gb Device 0: (UUID: MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b) MIG 3g.40gb Device 1: (UUID: MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc) GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-7eba9d78-e908-3db8-2633-a269aeec395e) MIG 7g.80gb Device 0: (UUID: MIG-ec043869-9176-577e-bac0-46c8411e4e37) $ scontrol show node node16 NodeName=node16 Arch=x86_64 CoresPerSocket=1 CPUAlloc=1 CPUEfctv=112 CPUTot=112 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 NodeAddr=node16 NodeHostName=node16 Version=22.05.9 OS=Linux 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023 RealMemory=1030000 AllocMem=0 FreeMem=1011801 Sockets=112 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpu_H100 BootTime=2024-01-18T13:52:27 SlurmdStartTime=2024-01-19T11:25:44 LastBusyTime=2024-01-19T10:59:13 CfgTRES=cpu=112,mem=1030000M,billing=112,gres/gpu=14 AllocTRES=cpu=1 CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s $ scontrol show partition gpu_H100 PartitionName=gpu_H100 AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=node16 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=112 TotalNodes=1 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=112,mem=1030000M,node=1,billing=112,gres/gpu=14 slurm.conf: AccountingStorageTRES=gres/gpu,gres/gpu:A100,gres/gpu:H100,gres/gpu:H100_1g.10gb,gres/gpu:H100_2g.20gb,gres/gpu:H100_3g.40gb,gres/gpu:H100_7g.80gb GresTypes=gpu ... NodeName=node16 CPUs=112 RealMemory=1030000 Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 PartitionName=gpu_H100 Nodes=node16 Default=NO MaxTime=INFINITE State=UP gres.conf #AutoDetect=nvml # GPU 0 MIG 0 /proc/driver/nvidia/capabilities/gpu0/mig/gi7/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap66,/dev/nvidia-caps/nvidia-cap67 CPUs=0-55 # GPU 0 MIG 1 /proc/driver/nvidia/capabilities/gpu0/mig/gi8/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap75,/dev/nvidia-caps/nvidia-cap76 CPUs=0-55 # GPU 0 MIG 2 /proc/driver/nvidia/capabilities/gpu0/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap84,/dev/nvidia-caps/nvidia-cap85 CPUs=0-55 # GPU 0 MIG 3 /proc/driver/nvidia/capabilities/gpu0/mig/gi10/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap93,/dev/nvidia-caps/nvidia-cap94 CPUs=0-55 # GPU 0 MIG 4 /proc/driver/nvidia/capabilities/gpu0/mig/gi11/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap102,/dev/nvidia-caps/nvidia-cap103 CPUs=0-55 # GPU 0 MIG 5 /proc/driver/nvidia/capabilities/gpu0/mig/gi12/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap111,/dev/nvidia-caps/nvidia-cap112 CPUs=0-55 # GPU 0 MIG 6 /proc/driver/nvidia/capabilities/gpu0/mig/gi13/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap120,/dev/nvidia-caps/nvidia-cap121 CPUs=0-55 # GPU 1 MIG 0 /proc/driver/nvidia/capabilities/gpu1/mig/gi3/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap165,/dev/nvidia-caps/nvidia-cap166 CPUs=0-55 # GPU 1 MIG 1 /proc/driver/nvidia/capabilities/gpu1/mig/gi5/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap183,/dev/nvidia-caps/nvidia-cap184 CPUs=0-55 # GPU 1 MIG 2 /proc/driver/nvidia/capabilities/gpu1/mig/gi6/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap192,/dev/nvidia-caps/nvidia-cap193 CPUs=0-55 # GPU 1 MIG 3 /proc/driver/nvidia/capabilities/gpu1/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap219,/dev/nvidia-caps/nvidia-cap220 CPUs=0-55 # GPU 2 MIG 0 /proc/driver/nvidia/capabilities/gpu2/mig/gi1/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap282,/dev/nvidia-caps/nvidia-cap283 CPUs=0-55 # GPU 2 MIG 1 /proc/driver/nvidia/capabilities/gpu2/mig/gi2/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap291,/dev/nvidia-caps/nvidia-cap292 CPUs=0-55 # GPU 3 MIG 0 /proc/driver/nvidia/capabilities/gpu3/mig/gi0/access NodeName=node16 Name=gpu Type=H100_7g.80gb MultipleFiles=/dev/nvidia3,/dev/nvidia-caps/nvidia-cap408,/dev/nvidia-caps/nvidia-cap409 CPUs=0-55 [2024-01-19T14:31:13.701] debug: gres/gpu: init: loaded [2024-01-19T14:31:13.701] debug: gpu/generic: init: init: GPU Generic plugin loaded [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.702] Gres Name=gpu Type=H100_7g.80gb Count=1 Best, Drazen Jalsovec

1 0

Is there a way to map Azure AD users to NIS?
by Ben Wellborn 20 Jan '24

20 Jan '24

In the past, we would connect AD to NIS users using Centrify and adnisd, but I don't think adnisd can connect to Azure AD (HCI stack). Thanks, Ben

1 0

Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres)
by Jason Macklin 19 Jan '24

19 Jan '24

If you run "scontrol show jobid <jobid>" of your pending job with the "(Resources)" tag you may see more about what is unavailable to your job. Slurm default configs can cause an entire compute node of resources to be "allocated" to a running job regardless of whether it needs all of them or not so you may need to alter one or both of the following settings to allow more than one job to run on a single node at once. You'll find these in your slurm.conf. Don't forget to "scontrol reconf" and even potentially restart both "slurmctld" & "slurmd" on your nodes if you do end up making changes. SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory I hope this helps. Kind regards, Jason ---- Jason Macklin Manager Cyberinfrastructure, Research Cyberinfrastructure 860.837.2142 t | 860.202.7779 m jason.macklin(a)jax.org The Jackson Laboratory Maine | Connecticut | California | Shanghai www.jax.org<http://www.jax.org> The Jackson Laboratory: Leading the search for tomorrow's cures ________________________________ From: slurm-users <slurm-users-bounces(a)lists.schedmd.com> on behalf of slurm-users-request(a)lists.schedmd.com <slurm-users-request(a)lists.schedmd.com> Sent: Friday, January 19, 2024 9:24 AM To: slurm-users(a)lists.schedmd.com <slurm-users(a)lists.schedmd.com> Subject: [EXTERNAL]slurm-users Digest, Vol 75, Issue 31 Send slurm-users mailing list submissions to slurm-users(a)lists.schedmd.com To subscribe or unsubscribe via the World Wide Web, visit https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users or, via email, send a message with subject or body 'help' to slurm-users-request(a)lists.schedmd.com You can reach the person managing the list at slurm-users-owner(a)lists.schedmd.com When replying, please edit your Subject line so it is more specific than "Re: Contents of slurm-users digest..." Today's Topics: 1. Re: Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) (Marko Markoc) 2. Re: Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) (?mit Seren) ---------------------------------------------------------------------- Message: 1 Date: Fri, 19 Jan 2024 06:12:24 -0800 From: Marko Markoc <mmarkoc(a)pdx.edu> To: Slurm User Community List <slurm-users(a)lists.schedmd.com> Subject: Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) Message-ID: <CABnuMe4JTA0e6=VbO8D+To=8FGO+3Byv1dK_MC+OuRitzN5dXg(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" +1 on checking the memory allocation. Or add/check if you have any DefMemPerX set in your slurm.conf On Fri, Jan 19, 2024 at 12:33?AM mohammed shambakey <shambakey1(a)gmail.com> wrote: > Hi > > I'm not an expert, but is it possible that the currently running jobs is > consuming the whole node because it is allocated the whole memory of the > node (so the other 2 jobs had to wait until it finishes)? > Maybe if you try to restrict the required memory for each job? > > Regards > > On Thu, Jan 18, 2024 at 4:46?PM ?mit Seren <uemit.seren(a)gmail.com> wrote: > >> This line also has tobe changed: >> >> >> #SBATCH --gpus-per-node=4 ? #SBATCH --gpus-per-node=1 >> >> --gpus-per-node seems to be the new parameter that is replacing the --gres= >> one, so you can remove the ?gres line completely. >> >> >> >> Best >> >> ?mit >> >> >> >> *From: *slurm-users <slurm-users-bounces(a)lists.schedmd.com> on behalf of >> Kherfani, Hafedh (Professional Services, TC) <hafedh.kherfani(a)hpe.com> >> *Date: *Thursday, 18. January 2024 at 15:40 >> *To: *Slurm User Community List <slurm-users(a)lists.schedmd.com> >> *Subject: *Re: [slurm-users] Need help with running multiple >> instances/executions of a batch script in parallel (with NVIDIA HGX A100 >> GPU as a Gres) >> >> Hi Noam and Matthias, >> >> >> >> Thanks both for your answers. >> >> >> >> I changed the ?#SBATCH --gres=gpu:4? directive (in the batch script) with >> ?#SBATCH --gres=gpu:1? as you suggested, but it didn?t make a difference, >> as running this batch script 3 times will result in the first job to be in >> a running state, while the second and third jobs will still be in a pending >> state ? >> >> >> >> [slurmtest@c-a100-master test-batch-scripts]$ cat gpu-job.sh >> >> #!/bin/bash >> >> #SBATCH --job-name=gpu-job >> >> #SBATCH --partition=gpu >> >> #SBATCH --nodes=1 >> >> #SBATCH --gpus-per-node=4 >> >> #SBATCH --gres=gpu:1 # <<<< Changed from ?4? >> to ?1? >> >> #SBATCH --tasks-per-node=1 >> >> #SBATCH --output=gpu_job_output.%j >> >> #SBATCH --error=gpu_job_error.%j >> >> >> >> hostname >> >> date >> >> sleep 40 >> >> pwd >> >> >> >> [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh >> >> Submitted batch job *217* >> >> [slurmtest@c-a100-master test-batch-scripts]$ squeue >> >> JOBID PARTITION NAME USER ST TIME NODES >> NODELIST(REASON) >> >> 217 gpu gpu-job slurmtes R 0:02 1 >> c-a100-cn01 >> >> [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh >> >> Submitted batch job *218* >> >> [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh >> >> Submitted batch job *219* >> >> [slurmtest@c-a100-master test-batch-scripts]$ squeue >> >> JOBID PARTITION NAME USER ST TIME NODES >> NODELIST(REASON) >> >> 219 gpu gpu-job slurmtes *PD* 0:00 1 >> (Priority) >> >> 218 gpu gpu-job slurmtes *PD* 0:00 1 >> (Resources) >> >> 217 gpu gpu-job slurmtes *R* 0:07 1 >> c-a100-cn01 >> >> >> >> Basically I?m seeking for some help/hints on how to tell Slurm, from the >> batch script for example: ?I want only 1 or 2 GPUs to be used/consumed by >> the job?, and then I run the batch script/job a couple of times with sbatch >> command, and confirm that we can indeed have multiple jobs using a GPU and >> running in parallel, at the same time. >> >> >> >> Makes sense ? >> >> >> >> >> >> Best regards, >> >> >> >> *Hafedh * >> >> >> >> *From:* slurm-users <slurm-users-bounces(a)lists.schedmd.com> *On Behalf >> Of *Bernstein, Noam CIV USN NRL (6393) Washington DC (USA) >> *Sent:* jeudi 18 janvier 2024 2:30 PM >> *To:* Slurm User Community List <slurm-users(a)lists.schedmd.com> >> *Subject:* Re: [slurm-users] Need help with running multiple >> instances/executions of a batch script in parallel (with NVIDIA HGX A100 >> GPU as a Gres) >> >> >> >> On Jan 18, 2024, at 7:31 AM, Matthias Loose <m.loose(a)mindcode.de> wrote: >> >> >> >> Hi Hafedh, >> >> Im no expert in the GPU side of SLURM, but looking at you current >> configuration to me its working as intended at the moment. You have defined >> 4 GPUs and start multiple jobs each consuming 4 GPUs each. So the jobs wait >> for the ressource the be free again. >> >> I think what you need to look into is the MPS plugin, which seems to do >> what you are trying to achieve: >> https://slurm.schedmd.com/gres.html#MPS_Management >> >> >> >> I agree with the first paragraph. How many GPUs are you expecting each >> job to use? I'd have assumed, based on the original text, that each job is >> supposed to use 1 GPU, and the 4 jobs were supposed to be running >> side-by-side on the one node you have (with 4 GPUs). If so, you need to >> tell each job to request only 1 GPU, and currently each one is requesting 4. >> >> >> >> If your jobs are actually supposed to be using 4 GPUs each, I still don't >> see any advantage to MPS (at least in what is my usual GPU usage pattern): >> all the jobs will take longer to finish, because they are sharing the fixed >> resource. If they take turns, at least the first ones finish as fast as >> they can, and the last one will finish no later than it would have if they >> were all time-sharing the GPUs. I guess NVIDIA had something in mind when >> they developed MPS, so I guess our pattern may not be typical (or at least >> not universal), and in that case the MPS plugin may well be what you need. >> > > > -- > Mohammed >

1 0

slurmctld/slurmdbd (code=exited, status=217/USER)
by Miriam Olmi 19 Jan '24

19 Jan '24

Hi all, I am having some issue with the new version of slurm 23.11.0-1. I had already installed and configured slurm 23.02.3-1 on my cluster and all the services were active and running properly. After I install with the same procedure the new version of slurm I have that the slurmctld and slurmdbd daemons fail to start all with the same error: (code=exited, status=217/USER) And investigating the problem with the command journalctl -xe I find: slurmctld.service: Failed to determine user credentials: No such process slurmctld.service: Failed at step USER spawning /usr/sbin/slurmctld: No such process I had a look at the slurmctld.service file for both the slurm versions and I found the following differences in the [Service] section. >From the slurmctld.service file of slurm 23.02.3-1: [Service] Type=simple EnvironmentFile=-/etc/sysconfig/slurmctld EnvironmentFile=-/etc/default/slurmctld ExecStart=/usr/sbin/slurmctld -D -s $SLURMCTLD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID LimitNOFILE=65536 TasksMax=infinity >From the slurmctld.service file of slurm 23.11.0-1: [Service] Type=notify EnvironmentFile=-/etc/sysconfig/slurmctld EnvironmentFile=-/etc/default/slurmctld User=slurm Group=slurm ExecStart=/usr/sbin/slurmctld --systemd $SLURMCTLD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID LimitNOFILE=65536 TasksMax=infinity I think the presence of the new lines regarding the slurm user might be the problem but I am not sure and I have no idea how to solve it. Can anyone halp me? Thanks in advance, Miriam

2 1

Jobs exiting together
by Alexander Silva 19 Jan '24

19 Jan '24

Recently, i have built an hpc cluster with slurm as workload. The test jobs with quatum chemistry codes have worked fine. However, production jobs with lammps have shown an unexpected behavior when the first job completed, normally or not, cause the termination of the others in the same compute node. Initially, I thought that was due to mpi malfunction, but this behavior is algo observed for serial lammps code. The lammps group said to me that behavior could be generated by slurm. My question to you is about what parameter in slurm.conf could be responsible for the termination of the other jobs. I am using an epilogue script that work normally in another cluster. Thanks.

1 0

Running slurm job on requested nvidia mig device
by Dražen Jalšovec 19 Jan '24

19 Jan '24

Hi, We are testing the MIG deployment on our new slurm compute node with 4 x H100 GPUs. It looks like everything is configured correctly but we have a problem accessing mig devices. When I submit jobs requesting a mig gpu device #SBATCH --gres=gpu:H100_1g.10gb:1, the jobs get submitted to the node, but only 4 jobs get executed and all other jobs fail. I was able to "solve it" by adding code to submit to the mig device UUID. Am I missing something in the slurm/gres/cgroups configuration that would automatically assign correct CUDA_VISIBLE_DEVICES MIG UUID to the job? If I submit 10 jobs requesting --gres=gpu:H100_1g.10gb:1 8 jobs start running: #1 - CUDA_VISIBLE_DEVICES=0 - jobs 1-4 run on one of the mig devices configured on GPU-device ... #8 - CUDA_VISIBLE_DEVICES=7 - jobs 5-8 fail with the message that there are no available CUDA_VISIBLE_DEVICES ... #10 - CUDA_VISIBLE_DEVICES=? - after waiting for free mig resources jobs 9-10 runs or fails depending if GPU-device is free (migs are free) $ nvidia-smi --query-gpu=uuid --id=0 --format=csv,noheader GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3 $ nvidia-smi --query-gpu=uuid --id=7 --format=csv,noheader No devices were found The slurm assigns CUDA_VISIBLE_DEVICES numbers based on the requested resources: Request: gpu:H100_1g.10gb:1 - CUDA_VISIBLE_DEVICES=0 gpu:H100_1g.10gb:2 - CUDA_VISIBLE_DEVICES=0,1 ... Any job with H100_1g.10gb request is going to get device number between 0 and 7 gpu:H100_1g.10gb:8 - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 gpu:H100_2g.20gb:1 - CUDA_VISIBLE_DEVICES=8 Any job with H100_2g.20gb request is going to get device number between 8 and 10 ... gpu:H100_2g.10gb:3 - CUDA_VISIBLE_DEVICES=8,9,10 gpu:H100_3g.40gb:1 - CUDA_VISIBLE_DEVICES=11 gpu:H100_3g.40gb:2 - CUDA_VISIBLE_DEVICES=12 gpu:H100_7g.80gb:1 - CUDA_VISIBLE_DEVICES=13 I was able to "solve it" by adding this code to my slurm script: # Get the CUDA index using srun and awk CUDA_INDEX=$(srun env | grep CUDA_VISIBLE_DEVICES | awk -F '=' '{print $2}') ## GPU0 1g.10gb ## MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a ## MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664 ## MIG-d3ab0675-d318-5e53-b487-b50695cf2e00 ## MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0 ## MIG-85dd76da-c994-5830-adf0-467c66ae1b95 ## MIG-29a6d43b-882e-5b79-868a-15bb2c770b82 ## MIG-558debd2-dc13-5406-9256-73ef4f279737 ## GPU1 2g.20gb ## MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f ## MIG-0a804c0c-aa27-5993-97cb-cedb854735ce ## MIG-8af9af6d-8720-5763-81e2-83afc43eb42b ## GPU1 1g.10gb ## MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a ## GPU2 3g.40gb ## MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b ## MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc ## GPU3 7g.80gb ## MIG-ec043869-9176-577e-bac0-46c8411e4e37 # Define the list of UUIDs UUIDS=( "MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a" "MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664" "MIG-d3ab0675-d318-5e53-b487-b50695cf2e00" "MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0" "MIG-85dd76da-c994-5830-adf0-467c66ae1b95" "MIG-29a6d43b-882e-5b79-868a-15bb2c770b82" "MIG-558debd2-dc13-5406-9256-73ef4f279737" "MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a" "MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f" "MIG-0a804c0c-aa27-5993-97cb-cedb854735ce" "MIG-8af9af6d-8720-5763-81e2-83afc43eb42b" "MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b" "MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc" "MIG-ec043869-9176-577e-bac0-46c8411e4e37" ) # Assign the UUID based on the CUDA index SELECTED_UUID=${UUIDS[$CUDA_INDEX]} # Print the selected UUID echo $SELECTED_UUID export CUDA_VISIBLE_DEVICES=$SELECTED_UUID echo "CUDA_VISIBLE_DEVICES set to: "$SELECTED_UUID echo "Test: "$CUDA_VISIBLE_DEVICES CUDA_VISIBLE_DEVICES=$SELECTED_UUID python3.8 gpu_script.py Result: Script start MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a CUDA_VISIBLE_DEVICES set to: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a Matrix calculation on CUDA device completed successfully. Script end +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 7 0 28686 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 8 0 28744 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 9 0 28300 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 10 0 28512 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 11 0 28506 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 12 0 28516 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 13 0 28511 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 1 9 0 28552 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | +---------------------------------------------------------------------------------------+ This is the current setup: RockyLinux 9.2 slurm-22.05.9-1.el9.x86_64 CUDA Version: 12.1 Driver Version: 530.30.02 # nvidia-smi -L GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3) MIG 1g.10gb Device 0: (UUID: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a) MIG 1g.10gb Device 1: (UUID: MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664) MIG 1g.10gb Device 2: (UUID: MIG-d3ab0675-d318-5e53-b487-b50695cf2e00) MIG 1g.10gb Device 3: (UUID: MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0) MIG 1g.10gb Device 4: (UUID: MIG-85dd76da-c994-5830-adf0-467c66ae1b95) MIG 1g.10gb Device 5: (UUID: MIG-29a6d43b-882e-5b79-868a-15bb2c770b82) MIG 1g.10gb Device 6: (UUID: MIG-558debd2-dc13-5406-9256-73ef4f279737) GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1439a39c-948b-0657-98e9-aff8595a8729) MIG 2g.20gb Device 0: (UUID: MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f) MIG 2g.20gb Device 1: (UUID: MIG-0a804c0c-aa27-5993-97cb-cedb854735ce) MIG 2g.20gb Device 2: (UUID: MIG-8af9af6d-8720-5763-81e2-83afc43eb42b) MIG 1g.10gb Device 3: (UUID: MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a) GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-3b182fed-bfdf-bc11-d1bd-c41fd1468d2a) MIG 3g.40gb Device 0: (UUID: MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b) MIG 3g.40gb Device 1: (UUID: MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc) GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-7eba9d78-e908-3db8-2633-a269aeec395e) MIG 7g.80gb Device 0: (UUID: MIG-ec043869-9176-577e-bac0-46c8411e4e37) $ scontrol show node node16 NodeName=node16 Arch=x86_64 CoresPerSocket=1 CPUAlloc=1 CPUEfctv=112 CPUTot=112 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 NodeAddr=node16 NodeHostName=node16 Version=22.05.9 OS=Linux 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023 RealMemory=1030000 AllocMem=0 FreeMem=1011801 Sockets=112 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpu_H100 BootTime=2024-01-18T13:52:27 SlurmdStartTime=2024-01-19T11:25:44 LastBusyTime=2024-01-19T10:59:13 CfgTRES=cpu=112,mem=1030000M,billing=112,gres/gpu=14 AllocTRES=cpu=1 CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s $ scontrol show partition gpu_H100 PartitionName=gpu_H100 AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=node16 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=112 TotalNodes=1 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=112,mem=1030000M,node=1,billing=112,gres/gpu=14 slurm.conf: AccountingStorageTRES=gres/gpu,gres/gpu:A100,gres/gpu:H100,gres/gpu:H100_1g.10gb,gres/gpu:H100_2g.20gb,gres/gpu:H100_3g.40gb,gres/gpu:H100_7g.80gb GresTypes=gpu ... NodeName=node16 CPUs=112 RealMemory=1030000 Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 PartitionName=gpu_H100 Nodes=node16 Default=NO MaxTime=INFINITE State=UP gres.conf #AutoDetect=nvml # GPU 0 MIG 0 /proc/driver/nvidia/capabilities/gpu0/mig/gi7/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap66,/dev/nvidia-caps/nvidia-cap67 CPUs=0-55 # GPU 0 MIG 1 /proc/driver/nvidia/capabilities/gpu0/mig/gi8/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap75,/dev/nvidia-caps/nvidia-cap76 CPUs=0-55 # GPU 0 MIG 2 /proc/driver/nvidia/capabilities/gpu0/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap84,/dev/nvidia-caps/nvidia-cap85 CPUs=0-55 # GPU 0 MIG 3 /proc/driver/nvidia/capabilities/gpu0/mig/gi10/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap93,/dev/nvidia-caps/nvidia-cap94 CPUs=0-55 # GPU 0 MIG 4 /proc/driver/nvidia/capabilities/gpu0/mig/gi11/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap102,/dev/nvidia-caps/nvidia-cap103 CPUs=0-55 # GPU 0 MIG 5 /proc/driver/nvidia/capabilities/gpu0/mig/gi12/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap111,/dev/nvidia-caps/nvidia-cap112 CPUs=0-55 # GPU 0 MIG 6 /proc/driver/nvidia/capabilities/gpu0/mig/gi13/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap120,/dev/nvidia-caps/nvidia-cap121 CPUs=0-55 # GPU 1 MIG 0 /proc/driver/nvidia/capabilities/gpu1/mig/gi3/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap165,/dev/nvidia-caps/nvidia-cap166 CPUs=0-55 # GPU 1 MIG 1 /proc/driver/nvidia/capabilities/gpu1/mig/gi5/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap183,/dev/nvidia-caps/nvidia-cap184 CPUs=0-55 # GPU 1 MIG 2 /proc/driver/nvidia/capabilities/gpu1/mig/gi6/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap192,/dev/nvidia-caps/nvidia-cap193 CPUs=0-55 # GPU 1 MIG 3 /proc/driver/nvidia/capabilities/gpu1/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap219,/dev/nvidia-caps/nvidia-cap220 CPUs=0-55 # GPU 2 MIG 0 /proc/driver/nvidia/capabilities/gpu2/mig/gi1/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap282,/dev/nvidia-caps/nvidia-cap283 CPUs=0-55 # GPU 2 MIG 1 /proc/driver/nvidia/capabilities/gpu2/mig/gi2/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap291,/dev/nvidia-caps/nvidia-cap292 CPUs=0-55 # GPU 3 MIG 0 /proc/driver/nvidia/capabilities/gpu3/mig/gi0/access NodeName=node16 Name=gpu Type=H100_7g.80gb MultipleFiles=/dev/nvidia3,/dev/nvidia-caps/nvidia-cap408,/dev/nvidia-caps/nvidia-cap409 CPUs=0-55 [2024-01-19T14:31:13.701] debug: gres/gpu: init: loaded [2024-01-19T14:31:13.701] debug: gpu/generic: init: init: GPU Generic plugin loaded [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.702] Gres Name=gpu Type=H100_7g.80gb Count=1 Best, Drazen Jalsovec

1 0

Potential Side Effects of larger MessageTimeout value
by Herc Silverstein 18 Jan '24

18 Jan '24

Hi, What are potential bad side effects of using a large/larger MessageTimeout? And is there a value at which this setting is too large (long)? Thanks, Herc

1 0

2025

2024

slurm-users ----- 2025 ----- April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users