January 2024 - slurm-users - lists.schedmd.com

job_container/tmpfs and srun.
by Phill Harvey-Smith 17 Jan '25

17 Jan '25

Hi all, On our setup we are using job_container/tmpfs to give each job it's own temp space. Since our compute nodes have reasonably sized disks for tasks that do a lot of disk I/O on user's data we have asked users to copy their data to the local disk at the beginning of the task and (if needed) copy it back at the end. This saves lots of NFS thrashing slowing down both the task and the NFS servers. However some of our users are having problems with this, their initial sbatch script will create a temp directory in their private /tmp copy their data to it and then try to srun a program. The srun will fall over as it doesn't seem to have have access to the copied data. I suspect this is because the srun task is getting it's own private /tmp. So my question is, is there a way to have the srun task inherit the /tmp of the initial sbatch? I'll include a sample of the script our user is using below. If any further information is required please feel free to ask. Cheers. Phill. #!/usr/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:00:10 #SBATCH --mem-per-cpu=3999 #SBATCH --output=script_out.log #SBATCH --error=script_error.log # The above options puts the STDOUT and STDERR of sbatch in # log files prefixed with 'script_'. # Create a randomly-named directory under /tmp jobtmpdir=$(mktemp -d) # Register a function to try and cleanup in case of job failure cleanup_handler() { echo "Cleaning up ${jobtmpdir}" rm -rf ${jobtmpdir} } trap 'cleanup_handler' SIGTERM EXIT # Change working directory to this directory cd ${jobtmpdir} # Copy the executable and input files from # where the job was submitted to the temporary directory. cp ${SLURM_SUBMIT_DIR}/a.out . cp ${SLURM_SUBMIT_DIR}/input.txt . # Run the executable, handling the collection of stdout # and stderr ourselves by redirecting to file srun ./a.out 2> task_error.log > task_out.log # Copy output data back to the submit directory. cp output.txt ${SLURM_SUBMIT_DIR} cp task_out.log ${SLURM_SUBMIT_DIR} cp task_error.log ${SLURM_SUBMIT_DIR} # Cleanup cd ${SLURM_SUBMIT_DIR} cleanup_handler

2 1

Issues with Slurm 23.11.1
by Fokke Dijkstra 15 Mar '24

15 Mar '24

Dear all, Since the upgrade from Slurm 22.05 to 23.11.1 we are having problems with the communication between the slurmctld and slurmd processes. We are running a cluster with 183 nodes and almost 19000 cores. Unfortunately some nodes are in a different network preventing full internode communication. A network topology and setting TopologyParam RouteTree have been used to make sure no slurmd communication happens between nodes on different networks. In the new Slurm version we see the following issues, which did not appear in 22.05: 1. slurmd processes acquire many network connections in CLOSE-WAIT (or CLOSE_WAIT depending on the tool used) causing the processes to hang, when trying to restart slurmd. When checking for CLOSE-WAIT processes we see the following behaviour: Recv-Q Send-Q Local Address:Port Peer Address:Port Process 1 0 10.5.2.40:6818 10.5.0.43:58572 users:(("slurmd",pid=1930095,fd=72),("slurmd",pid=1930067,fd=72)) 1 0 10.5.2.40:6818 10.5.0.43:58284 users:(("slurmd",pid=1930095,fd=8),("slurmd",pid=1930067,fd=8)) 1 0 10.5.2.40:6818 10.5.0.43:58186 users:(("slurmd",pid=1930095,fd=22),("slurmd",pid=1930067,fd=22)) 1 0 10.5.2.40:6818 10.5.0.43:58592 users:(("slurmd",pid=1930095,fd=76),("slurmd",pid=1930067,fd=76)) 1 0 10.5.2.40:6818 10.5.0.43:58338 users:(("slurmd",pid=1930095,fd=19),("slurmd",pid=1930067,fd=19)) 1 0 10.5.2.40:6818 10.5.0.43:58568 users:(("slurmd",pid=1930095,fd=68),("slurmd",pid=1930067,fd=68)) 1 0 10.5.2.40:6818 10.5.0.43:58472 users:(("slurmd",pid=1930095,fd=69),("slurmd",pid=1930067,fd=69)) 1 0 10.5.2.40:6818 10.5.0.43:58486 users:(("slurmd",pid=1930095,fd=38),("slurmd",pid=1930067,fd=38)) 1 0 10.5.2.40:6818 10.5.0.43:58316 users:(("slurmd",pid=1930095,fd=29),("slurmd",pid=1930067,fd=29)) The first IP address is that of the compute node, the second that of the node running slurmctld. The nodes can communicate using these IP addresses just fine. 2. slurmd cannot be properly restarted [2024-01-18T10:45:26.589] slurmd version 23.11.1 started [2024-01-18T10:45:26.593] error: Error binding slurm stream socket: Address already in use [2024-01-18T10:45:26.593] fatal: Unable to bind listen port (6818): Address already in use This is probably because of the processes being in CLOSE-WAIT, which can only be killed using signal -9. 3. We see jobs stuck in completing CG state, probably due to communication issues between slurmctld and slurmd. The slurmctld sends repeated kill requests but those do not seem to be acknowledged by the client. This happens more often in large job arrays, or generally when many jobs start at the same time. However, this could be just a biased observation (i.e., it is more noticeable on large job arrays because there are more jobs to fail in the first place). 4. Since the new version we also see messages like: [2024-01-17T09:58:48.589] error: Failed to kill program loading user environment [2024-01-17T09:58:48.590] error: Failed to load current user environment variables [2024-01-17T09:58:48.590] error: _get_user_env: Unable to get user's local environment, running only with passed environment The effect of this is that the users run with the wrong environment and can’t load the modules for the software that is needed by their jobs. This leads to many job failures. The issue appears to be somewhat similar to the one described at: https://bugs.schedmd.com/show_bug.cgi?id=18561 In that case the site downgraded the slurmd clients to 22.05 which got rid of the problems. We’ve now downgraded the slurmd on the compute nodes to 23.02.7 which also seems to be a workaround for the issue. Does anyone know of a better solution? Kind regards, Fokke Dijkstra -- Fokke Dijkstra <f.dijkstra(a)rug.nl> <f.dijkstra(a)rug.nl> Team High Performance Computing Center for Information Technology, University of Groningen Postbus 11044, 9700 CA Groningen, The Netherlands

2 3

slurmctld: slurm_bufs_sendto(msg_type=SRUN_STEP_SIGNAL) failed: Connection reset by peer
by Rike-Benjamin Schuppner 16 Feb '24

16 Feb '24

Hi, I am getting the following error in the logs whenever I run a few srun jobs in a batch. Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: debug: _send_timeout: Socket POLLERR: Connection reset by peer Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: error: slurm_send_node_msg: [socket:[921897]] slurm_bufs_sendto(msg_type=SRUN_STEP_SIGNAL) failed: Connection reset by peer Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: debug: laying out the 1 tasks on 1 hosts compute2 dist 1 The slurm version is 23.11.3 and an example sbatch file is: #!/bin/bash #SBATCH --job-name=slurm_test #SBATCH --mem=1gb #SBATCH --time=00:05:00 #SBATCH --output=slurm_test_%j.log pwd; hostname; date srun --exclusive -c1 -N1 -n1 bash -c "hostname ; sleep 10" & srun --exclusive -c1 -N1 -n1 bash -c "hostname ; sleep 10" & wait The more sruns I have in the script (&-backgrounded or not), the more the error shows up. Is there anything I could do to fix this error? Best /rike

2 1

Compilation question
by Sylvain MARET 10 Feb '24

10 Feb '24

Hello everyone ! I'm trying to compile slurm 22.05.11 on Rocky linux 8.7 with freeipmi support I've seen the documentation so I've done the configure step : ./configure --with-pmix=$PMIXHOME --with-ucx=$UCXHOME --with-nvml=$NVMLHOME --prefix=$SLURMHOME --with-freeipmi=/usr but when I run make I end up with the following error : /bin/sh ../../../../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -L/usr/lib64 -lhdf5_hl -lhdf5 -lsz -lz -ldl -lm -o sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -L../../../../../src/api/.libs -lslurmfull -ldl ../libhdf5_api.la -lpthread -lm -lresolv libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o .libs/sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -Wl,--export-dynamic -L/usr/lib64 -L../../../../../src/api/.libs /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so ../.libs/libhdf5_api.a -lhdf5_hl -lhdf5 -lsz -lz -ldl -lpthread -lm -lresolv -pthread -Wl,-rpath -Wl,/softs/batch/slurm/22.05.11/lib/slurm sh5util.o:(.init_array+0x0): undefined reference to `__nv_init_env' sh5util.o:(.init_array+0x8): undefined reference to `__flushz' sh5util.o:(.init_array+0x10): undefined reference to `__daz' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_transfer_unique' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_sort_key_pairs' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrchr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_unsetenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_sort' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_for_each' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin__pgi_isnanld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_extra_conf_path' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_ctzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_running_in_slurmctld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mcopy1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_clzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_count' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin_va_gparg1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_destroy_config_key_pair' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xfree_ptr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_getenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_free_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_log_level' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mset8' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup_printf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_delete_first' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_append' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_error' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_init_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_log_var' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_parse_file' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_fatal' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_popcountll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrcmp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_destroy' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_setenvpf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_destroy' collect2: error: ld returned 1 exit status What can I do to resolve these undefined reference error ? Regards, Sylvain Maret

3 2

after upgrade to 23.11.1 nodes stuck in completion state
by Paul Raines 01 Feb '24

01 Feb '24

I finally had downtime on our cluster running 20.11.3 and decided to upgrade SLURM. All daemons were stopped on nodes and master. Rocky 8 Linux OS was updated but not changed configuration-wise in anyway. On the master, when I first installed 23.11.1 and tried to run slurmdbd -D -vvv at the command line, it balked as the more than 2 major version jump. I then installed 22.05.6 and run slurmdbd -D -vvv which took a long time but completed fine. I reinstalled 23.11.1 and did slurmdbd -D --vvv which went quick. I stopped it then ran slurmdbd normally with systemctl. When I ran slurmctld it complained about incompatable state files. I didn't care about loosing state so ran with the -i option to ignore. Seemed happy. I then started up slurmd on my various GPU and non-GPU nodes. I ran a test interactive job just to see if I would get a shell with expected SLURM environment on the nodes and it worked. Except on the GPU nodes I got errors about not being able to get/set GPU frequences. I removed GpuFreqDef=medium from slurm.conf and that went away. I had my people start running jobs last night. This morning I come in and see a lot of nodes are stuck in completing state. All these are GPU nodes. For example on node 'sevilla' squeue reports JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 3679433 lcna100 sjob_123 sg1526 CG 8:29 1 sevilla 3679432 lcna100 sjob_123 sg1526 CG 8:35 1 sevilla 3679431 lcna100 sjob_123 sg1526 CG 8:42 1 sevilla In the slurmd.log on sevilla I see every minute the lines: [2024-01-28T09:37:52.003] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug: _rpc_terminate_job: uid = 1150 JobId=3679432 [2024-01-28T09:37:52.004] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug: _rpc_terminate_job: uid = 1150 JobId=3679431 [2024-01-28T09:37:52.004] debug2: Finish processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug: _rpc_terminate_job: uid = 1150 JobId=3679433 [2024-01-28T09:37:52.004] debug2: Finish processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Finish processing RPC: REQUEST_TERMINATE_JOB Back at the time 3679433 was cancelled by the user I see [2024-01-28T00:27:48.658] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T00:27:48.658] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T00:27:48.658] debug: _rpc_terminate_job: uid = 1150 JobId=3679433 [2024-01-28T00:27:48.658] debug: credential for job 3679433 revoked [2024-01-28T00:27:48.658] debug: _rpc_terminate_job: sent SUCCESS for 3679433, waiting for prolog to finish [2024-01-28T00:27:48.658] debug: Waiting for job 3679433's prolog to complete On sevilla itself there are no processes currently running for user sg1526 so no sign of these jobs. There ARE processes running for user mu40 though [root@sevilla ~]# pgrep -f slurm_script | xargs -n1 pstree -p | grep ^slurm slurm_script(30691)---starter-suid(30764)-+-python3(30784)-+-{python3}(30817) slurm_script(30861)---starter-suid(30873)-+-python3(30888)-+-{python3}(30915) slurm_script(30963)---starter-suid(30975)-+-python3(30992)-+-{python3}(31020) [root@sevilla ~]# strings /proc/30691/cmdline /bin/sh /var/slurm/spool/d/job3679220/slurm_script /usr/bin/sacct -p --jobs=3679220 --format=JobID,User,ReqTRES,NodeList,Start,End,Elapsed,CPUTime,State,ExitCode JobID|User|ReqTRES|NodeList|Start|End|Elapsed|CPUTime|State|ExitCode| 3679220|mu40|billing=6,cpu=3,gres/gpu=1,mem=32G,node=1|sevilla|2024-01-27T22:33:27|2024-01-27T23:26:39|00:53:12|02:39:36|NODE_FAIL|1:0| 3679220.batch|||sevilla|2024-01-27T22:33:27|2024-01-27T23:26:39|00:53:12|02:39:36|CANCELLED|| 3679220.extern|||sevilla|2024-01-27T22:33:27|2024-01-27T23:26:39|00:53:12|02:39:36|CANCELLED|| The slurmd.log has at the time it started lines like: [2024-01-27T22:33:28.679] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30691, probably hasn't started yet or has already finished [2024-01-27T22:33:58.092] [3679220.extern] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30650, probably hasn't started yet or has already finished [2024-01-27T22:33:58.166] [3679220.batch] debug2: profile signaling type Task [2024-01-27T22:33:58.168] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30691, probably hasn't started yet or has already finished [2024-01-27T22:33:58.170] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30764, probably hasn't started yet or has already finished [2024-01-27T22:33:58.172] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30784, probably hasn't started yet or has already finished ... [2024-01-27T22:46:39.018] debug: _step_connect: connect() failed for /var/slurm/spool/d/sevilla_3679220.4294967292: Connection refused [2024-01-27T22:46:39.018] debug: Cleaned up stray socket /var/slurm/spool/d/sevilla_3679220.4294967292 [2024-01-27T22:46:39.018] debug: _step_connect: connect() failed for /var/slurm/spool/d/sevilla_3679220.4294967291: Connection refused [2024-01-27T22:46:39.018] debug: Cleaned up stray socket /var/slurm/spool/d/sevilla_3679220.4294967291 [2024-01-27T22:46:39.018] _handle_stray_script: Purging vestigial job script /var/slurm/spool/d/job3679220/slurm_script then later at the time of termination [2024-01-27T23:26:39.042] debug: _rpc_terminate_job: uid = 1150 JobId=3679220 [2024-01-27T23:26:39.042] debug: credential for job 3679220 revoked [2024-01-27T23:26:39.042] debug2: No steps in jobid 3679220 to send signal 998 [2024-01-27T23:26:39.043] debug2: No steps in jobid 3679220 to send signal 18 [2024-01-27T23:26:39.043] debug2: No steps in jobid 3679220 to send signal 15 [2024-01-27T23:26:39.044] debug2: set revoke expiration for jobid 3679220 to 1706416119 UTS [2024-01-27T23:26:39.044] debug: Waiting for job 3679220's prolog to complete [2024-01-27T23:26:39.044] debug: Finished wait for job 3679220's prolog to complete [2024-01-27T23:26:39.068] debug: completed epilog for jobid 3679220 [2024-01-27T23:26:39.071] debug: JobId=3679220: sent epilog complete msg: rc = 0 So I have a node stuck in 'comp' state that claims to have jobs completing that are definite NOT running on the box BUT there are jobs running on the box that SLURM thinks are done --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

4 7

Why is Slurm 20 the latest RPM in RHEL 8/Fedora repo?
by Robert Kudyba 31 Jan '24

31 Jan '24

According to these links: https://rpmfind.net/linux/rpm2html/search.php?query=slurm https://src.fedoraproject.org/rpms/slurm Why doesn't RHEL 8 get a newer version? Can someone update the repo maintainer Philip Kovacs <https://src.fedoraproject.org/user/pkfed> < pkfed(a)fedoraproject.org>? There was a ticket at https://bugzilla.redhat.com/show_bug.cgi?id=1912491 but no movement on RHEL 8.

5 4

Mailing list upgrade - slurm-users list paused
by Tim Wickberg 30 Jan '24

30 Jan '24

Hey folks - The mailing list will be offline for about an hour as we upgrade the host, upgrade the mailing list software, and change the mail configuration around. As part of these changes, the "From: " field will no longer be the original sender, but instead use the mailing list ID itself. This is to comply with DMARC sending options, and allow us to start DKIM signing messages to ensure deliverability once Google and Yahoo impose new policy changes in February. This is the last post on the current (mailman2) list. I'll send a welcome message on the upgraded (mailman3) list once finished, and when the list is open to new traffic again. - Tim -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 1

Testing to SchedMD folks
by Tim Wickberg 30 Jan '24

30 Jan '24

Hey - this is hopefully a final test before the list is re-enabled for everyone. - Tim

1 0

Socket timed out - tuning
by Reed Dier 29 Jan '24

29 Jan '24

Hoping someone can help point me towards some tweaks to help prevent denial-of-service issues. > sbatch: error: Batch job submission failed: Socket timed out on send/recv operation Root cause is understood, issues with shared storage for the slurmctld’s was impacted, leading to an increase in write latency to the StateSaveLocation. Then with a large enough avalanche of job submissions, things the RPC’s would stack up and stop responding. I’ve been running well with some tweaks sourced from the “high-throughput” guide <https://slurm.schedmd.com/high_throughput.html>. > SchedulerParameters=max_rpc_cnt=400,\ > sched_min_interval=50000,\ > sched_max_job_start=300,\ > batch_sched_delay=6 > KillWait=30 > MessageTimeout=30 I’m assuming that I was running into batch_sched_delay because looking at sdiag after the fact, it was averaging .2s, and total time is 5.5h out of 16h8m18s at the time of the sdiag sample. > ******************************************************* > sdiag output at Thu Jan 25 11:08:18 2024 (1706198898) > Data since Wed Jan 24 19:00:00 2024 (1706140800) > ******************************************************* > REQUEST_SUBMIT_BATCH_JOB ( 4003) count:98400 ave_time:201442 total_time:19821991013 Currently on 22.05.8, but hoping to get to 23.02.7 soon™, and I think this could possible resolve the issue well enough if I’m reading it correctly from the release notes <https://slurm.schedmd.com/archive/slurm-23.02-latest/news.html>? > HIGHLIGHTS > ========== > -- slurmctld - Add new RPC rate limiting feature. This is enabled through > SlurmctldParameters=rl_enable, otherwise disabled by default. > rl_enable <https://slurm.schedmd.com/slurm.conf.html#OPT_rl_enable>Enable per-user RPC rate-limiting support. Client-commands will be told to back off and sleep for a second once the limit has been reached. This is implemented as a "token bucket", which permits a certain degree of "bursty" RPC load from an individual user before holding them to a steady-state RPC load established by the refill period and rate. But given that the hardware seems to be well over provisioned, CPU never drops below 5% idle, it feels like there is more room to squeeze some optimization out of here that I’m missing in the interim, and hoping to get a better overall understanding in the process. I scrape the DBD Agent queue size from sdiag every 30s and the largest value I saw was 115, which is much higher than normal, but should be well below MaxDBDMsgs, where the minimum value is 10000. I would really hope that I didn’t potentially hit a 30s MessageTimeout value, but I guess thats on the table all well, not knowing if that would potentially trigger the sbatch submission failure like that. Just moving the max_rpc_cnt value up seems like an easy button, but also seems like it could have some adverse effects for backfill scheduling, and may be diminishing returns for actually keeping RPCs flowing? > Setting max_rpc_cnt to more than 256 will be only useful to let backfill continue scheduling work after locks have been yielded (i.e. each 2 seconds) if there are a maximum of MAX(max_rpc_cnt/10, 20) RPCs in the queue. i.e. max_rpc_cnt=1000, the scheduler will be allowed to continue after yielding locks only when there are less than or equal to 100 pending RPCs. Obviously, fix the storage is the real solution, but hoping that there may be more goodness to unlock, even if it is as simple as “upgrade to 23.02”. Appreciate any insight, Reed

1 0

Two jobs each with a different partition running on same node?
by Loris Bennett 29 Jan '24

29 Jan '24

Hi, I seem to remember that in the past, if a node was configured to be in two partitions, the actual partition of the node was determined by the partition associated with the jobs running on it. Moreover, at any instance where the node was running one or more jobs, the node could only actually be in a single partition. Was this indeed the case and is it still the case with version Slurm 23.02.7? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT (ex-ZEDAT), Freie Universität Berlin

2 1

srun only runs one job on a node
by Rike-Benjamin Schuppner 27 Jan '24

27 Jan '24

Hi, Not sure if I do not understand the scheduling algorithm correctly of if this is a bug. The bug seems similar to the one described and fixed here https://github.com/SchedMD/slurm/commit/6a2c99edbf96e50463cc2f16d8e5eb955c8… Given the node definition: NodeName=cluster01 CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=64283 NodeName=cluster02 CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=128814 PartitionName=DEFAULT MaxTime=2-0 MaxNodes=2 PartitionName=mini Nodes=cluster01,cluster02 State=UP and the batch script: for i in {1..6} do srun --exclusive --mem=100 --threads-per-core=1 -N1 -c1 -n1 bash -c 'echo $(hostname) $(date); sleep 10 ; echo done $(hostname) $(date)' & done wait When run with: time sbatch --wait --partition=mini --ntasks=10 --cpus-per-task=1 --mem=1000 mini.slurm I get the following log after 2m 18s: cluster01 Sat Jan 27 02:05:12 PM CET 2024 cluster01 Sat Jan 27 02:05:12 PM CET 2024 cluster01 Sat Jan 27 02:05:13 PM CET 2024 cluster01 Sat Jan 27 02:05:13 PM CET 2024 cluster02 Sat Jan 27 02:05:13 PM CET 2024 done cluster01 Sat Jan 27 02:05:22 PM CET 2024 done cluster01 Sat Jan 27 02:05:22 PM CET 2024 done cluster01 Sat Jan 27 02:05:23 PM CET 2024 done cluster01 Sat Jan 27 02:05:23 PM CET 2024 done cluster02 Sat Jan 27 02:05:23 PM CET 2024 srun: Job 650 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=650.6 cluster01 Sat Jan 27 02:07:14 PM CET 2024 done cluster01 Sat Jan 27 02:07:24 PM CET 2024 So apparently I am getting 4 sruns on cluster01 (why not 6 when I have 8 CPUs?), but only 1 is scheduled on cluster02. Then it is exhausted and waits endlessly long (almost 2 minutes?) before scheduling again. The step log tells me: slurmctld: debug: laying out the 1 tasks on 1 hosts cluster01 dist 1 slurmctld: STEPS: _step_alloc_lps: JobId=650 StepId=4 node 0 (cluster01) gres_cpus_alloc=0 tasks=1 cpus_per_task=1 slurmctld: STEPS: _pick_step_cores: step JobId=650 StepId=4 requires 1 cores on node 0 with cpus_per_core=1, available cpus from job: 8 slurmctld: STEPS: _pick_step_core: alloc Node:0 Socket:0 Core:3 slurmctld: STEPS: step alloc on job node 0 (cluster01) used 4 of 8 CPUs slurmctld: STEPS: _slurm_rpc_job_step_create: JobId=650 StepId=4 cluster01 usec=5502 slurmctld: STEPS: _pick_step_nodes: JobId=650 Currently running steps use 4 of allocated 8 CPUs on node cluster01 slurmctld: STEPS: _pick_step_nodes: JobId=650 Currently running steps use 1 of allocated 2 CPUs on node cluster02 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=0 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=1 has nodes cluster02 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=2 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=3 has nodes cluster01 slurmctld: STEPS: _mark_busy_nodes: JobId=650 StepId=4 has nodes cluster01 slurmctld: STEPS: _pick_step_nodes: step pick 1-1 nodes, avail:cluster[01-02] idle: picked:NONE slurmctld: STEPS: _pick_step_nodes: step picked 0 of 1 nodes slurmctld: STEPS: Picked nodes cluster01 when accumulating from cluster01 slurmctld: debug: laying out the 1 tasks on 1 hosts cluster01 dist 1 slurmctld: STEPS: _step_alloc_lps: JobId=650 StepId=5 node 0 (cluster01) gres_cpus_alloc=0 tasks=1 cpus_per_task=1 slurmctld: STEPS: _pick_step_cores: step JobId=650 StepId=5 requires 1 cores on node 0 with cpus_per_core=1, available cpus from job: 8 slurmctld: STEPS: unable to pick step cores for job node 0 (cluster01): Requested nodes are busy slurmctld: STEPS: Deallocating 100MB of memory on node 0 (cluster01) now used: 400 of 1000 slurmctld: STEPS: step dealloc on job node 0 (cluster01) used: 4 of 8 CPUs slurmctld: STEPS: _slurm_rpc_job_step_create for JobId=650: Requested nodes are busy So why is it unable to allocate on cluster02? And why does its need to when it is only using 4 of 8 CPUs on cluster01 anyway? Now if I change to --ntasks=8, the algorithm doesn’t even schedule cluster02 at all: cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 cluster01 Sat Jan 27 02:15:30 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 done cluster01 Sat Jan 27 02:15:40 PM CET 2024 srun: Job 651 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=651.6 cluster01 Sat Jan 27 02:17:35 PM CET 2024 srun: Job 651 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Step created for StepId=651.7 cluster01 Sat Jan 27 02:17:36 PM CET 2024 done cluster01 Sat Jan 27 02:17:45 PM CET 2024 done cluster01 Sat Jan 27 02:17:46 PM CET 2024 Thanks for your help. Best /rike

1 0

LuaSQLite3 with slurm-lua-spank
by Jackson, Gary L. 26 Jan '24

26 Jan '24

What do I need to do to use the LuaSQLite3 library with Lua SPANK? For what it’s worth, I don’t know anything about Lua. -- Gary

1 0

sinfo: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host
by Michael Lewis 26 Jan '24

26 Jan '24

Hi All, I’m trying to get slurm-23.11.3 running on Ubuntu 20.04 and running on a stand alone system. I’m running into an issue I can not find the answer to. After compiling and installing when I fire up slurmctld and slurmd I get an error from sinfo: sinfo: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host sinfo: error: fetch_config: DNS SRV lookup failed sinfo: error: _establish_config_source: failed to fetch config sinfo: fatal: Could not establish a configuration source I looks like a DNS issue but the system has no issue resolving to its hostname or localhost. The slurm.conf file is also being read properly as I have the logs directed to a place convenient to me. I see lots have had these same issues but cannot find a clear resolution. I have slurm running on a stand alone system in another lab with and identical setup without issue. Any advice would be greatly appreciated. Thanks, Mike

2 2

Database cluster
by Daniel L'Hommedieu 26 Jan '24

26 Jan '24

Community: What do you do to ensure database reliability in your SLURM environment? We can have multiple controllers and multiple slurmdbds, but my understanding is that slurmdbd can be configured with a single MySQL server, so what do you do? Do you have that “single MySQL server” be a cluster, such as Percona XtraDB? Do you use MySQL replication, then manually switch to slurmdbd to a replication slave if the master goes down? Do you do something else? Thanks. Daniel

6 7

Problem using Podman with scrun on SLURM 23.11.3
by Marcus Lauer 25 Jan '24

25 Jan '24

I am getting an unusual error when trying to run Podman containers using scrun on SLURM 23.11.3 (and 23.11.1 previously). In short, Podman works when not configured to use scrun, but when configured to use scrun it fails. Podman gives this error: scrun: fatal: Unable to request job allocation: Job cannot be submitted without the current working directory specified. The slurmctld logs show this error: _slurm_rpc_allocate_resources: Job cannot be submitted without the current working directory specified. The scrun command does not appear to have a --chdir option so the working directory must be detected automatically, and this appears to be failing. Perhaps someone has encountered this or a similar error before? If anyone has seen this before I would love to hear about your experiences. A bit more detail in case anyone is interested. This is a two-node test system with one head node and one compute node, both running Rocky Linux 8.9. I have created /etc/containers/storage.conf and containers.conf files which are identical to the ones in the SLURM Containers Guide except that I set the rootless_storage_path to the same value as the graphroot in storage.conf. When I get rid of the /etc/containers/containers.conf file I can run containers on the head node. The backing store is ext4, though I have tried NFS (which had other problems). I have tried the vfs and overlay storage drivers but get the same results. On the SLURM side I created an oci.conf which is the same as the "oci.conf example for crun using run (suggested)" from the SLURM Container Guide. I'm using crun because I have forced cgroup v2 on both systems ( systemd.unified_cgroup_hierarchy=1 on the kernel cmdline) and crun seems to support cgroup v2 better. I am using an scrun.lua which is similar to the one in the scrun manual page with some paths modified for my setup. I should also mention that sbatch jobs run just fine and srun works as well. the test cluster seems to be working fine in general. -- Marcus Lauer Systems Administrator CETS Group, Research Support

1 0

Question about CPUs and cores
by Gestió Servidors 25 Jan '24

25 Jan '24

Hi, I want to run a simple test that uses one node and four cores. Also, in my script, I execute a binary that reports me in what core is running one of the four tasks. These are my files: * submit script: #!/bin/bash #SBATCH --job-name=test_jobs # Job name #SBATCH --output=test_job_%j.out # Output file (using array index) #SBATCH --error=test_job_%j.err # Error file (using array index) #SBATCH --ntasks=4 #SBATCH --ntasks-per-core=1 #SBATCH --cpus-per-task=1 # Number of CPU cores per task echo "START : $(date)" srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & srun -n1 -c1 check_cores >> output_small.txt & # Wait for all jobs to finish before exiting the job submission script wait echo "END : $(date)" * check_cores.c file: #include <stdio.h> #include<unistd.h> struct pstat { int pid; //process id char tcomm[256];//filename of the executable char state[2]; //state (R is running, S is sleeping, D is sleeping in an //uninterruptible wait, Z is zombie, T is traced or stopped) int ppid;// process id of the parent process int pgrp;// pgrp of the process int sid;// session id int tty_nr;// tty the process uses int tty_pgrp;// pgrp of the tty int flags;// task flags int min_flt;// number of minor faults int cmin_flt;// number of minor faults with child's int maj_flt;// number of major faults int cmaj_flt;// number of major faults with child's int utime;// user mode jiffies int stime;// kernel mode jiffies int cutime;// user mode jiffies with child's int cstime;// kernel mode jiffies with child's int priority;// priority level int nice;// nice level int num_threads;// number of threads int it_real_value;// (obsolete, always 0) int start_time;// time the process started after system boot int vsize;// virtual memory size int rss;// resident set memory size int rsslim;// current limit in bytes on the rss int start_code;// address above which program text can run int end_code;// address below which program text can run int start_stack;// address of the start of the stack int esp;// current value of ESP int eip;// current value of EIP int pending;// bitmap of pending signals int blocked;// bitmap of blocked signals int sigign;// bitmap of ignored signals int sigcatch;// bitmap of catched signals int wchan;// address where process went to sleep int i0;// (place holder) int i1;// (place holder) int exit_signal;// signal to send to parent thread on exit int task_cpu;// which CPU the task is scheduled on int rt_priority;// realtime priority int policy;// scheduling policy (man sched_setscheduler) int blkio_ticks;// time spent waiting for block IO int gtime;// guest time of the task in jiffies int cgtime;// guest time of the task children in jiffies } p ; int main() { char name[256]; char state[8]; FILE* f = fopen("/proc/self/stat", "r"); char hostname[1024]; gethostname(hostname, 1024); fscanf(f, "%d%s%s%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d%d", &p.pid, &p.tcomm, &p.state, &p.ppid, &p.pgrp, &p.sid, &p.tty_nr, &p.tty_pgrp, &p.flags, &p.min_flt, &p.cmin_flt, &p.maj_flt, &p.cmaj_flt, &p.utime, &p.stime, &p.cutime, &p.cstime, &p.priority, &p.nice, &p.num_threads, &p.it_real_value, &p.start_time, &p.vsize, &p.rss, &p.rsslim, &p.start_code, &p.end_code, &p.start_stack, &p.esp, &p.eip, &p.pending, &p.blocked, &p.sigign, &p.sigcatch, &p.wchan, &p.i0, &p.i1, &p.exit_signal, &p.task_cpu, &p.rt_priority, &p.policy, &p.blkio_ticks, &p.gtime, &p.cgtime); sleep (15); printf("HOST %s\n", hostname); printf("CPU %d\n", p.task_cpu); return 0; } When I run this script, I should receive an output like this: HOST clus04.hpc.local CPU 0 HOST clus04.hpc.local CPU 7 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 6 because system has scheduled my job in node "clus04.hpc.local" (a node with two 6-cores processors) and has reserved cores 0-1 of each processor. However, the real output I receive is HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 1 HOST clus04.hpc.local CPU 6 (and others outputs where there are one CPU repeated), so it seems SLURMs is reserving correctly four cores but, then, process is running two or more times in the same core. Slurmctld.log file in server shows this information: [...] [2024-01-25T09:38:42.867] DEBUG: JobId=206129 node clus04 vpus 1 cpus 4 [2024-01-25T09:38:42.867] ==================== [2024-01-25T09:38:42.867] JobId=206129 nhosts:1 ncpus:4 node_req:1 nodes=clus04 [2024-01-25T09:38:42.867] Node[0]: [2024-01-25T09:38:42.867] Mem(MB):0:0 Sockets:2 Cores:6 CPUs:4:0 [2024-01-25T09:38:42.867] Socket[0] Core[0] is allocated [2024-01-25T09:38:42.867] Socket[0] Core[1] is allocated [2024-01-25T09:38:42.867] Socket[1] Core[0] is allocated [2024-01-25T09:38:42.867] Socket[1] Core[1] is allocated [2024-01-25T09:38:42.867] -------------------- [2024-01-25T09:38:42.867] cpu_array_value[0]:4 reps:1 [2024-01-25T09:38:42.867] ==================== [2024-01-25T09:38:42.867] DEBUG: Dump job_resources: nhosts 1 cb 0-1,6-7 [...] [2024-01-25T09:38:58.213] _job_complete: JobId=206129 WEXITSTATUS 0 [2024-01-25T09:38:58.213] deallocate_nodes: JobId=206129 [2024-01-25T09:38:58.213] DEBUG: Dump job_resources: nhosts 1 cb 0-1,6-7 [2024-01-25T09:38:58.213] _job_complete: JobId=206129 done And slurmd.log file in node clus04 shows this information: [...] [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.685] [206129.batch] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.883] [206129.1] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.911] [206129.0] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.912] [206129.2] debug: task/cgroup: step physical cores are '0-1,6-7' [...] [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: job abstract cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: step abstract cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: job physical cores are '0-1,6-7' [2024-01-25T09:38:30.922] [206129.3] debug: task/cgroup: step physical cores are '0-1,6-7' It seems that SLURM is reserving me correctly four cores for my job but, when inside my script I run four "srun", then each srun can freely take one of the four reserved cores, not exactly one to each task. What am I doing wrong? Thanks a lot!

1 0

Slurm version 23.11.3 is now available
by Tim McMullan 24 Jan '24

24 Jan '24

We are pleased to announce the availability of Slurm version 23.11.3. The 23.11.3 fixes a single issue related to the new Debian package support. For 23.11.2, the debian changelog was not updated correctly and would generate packages as 23.11.1. This release will correctly generate 23.11.3 packages, with no other changes. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.3 > ========================== > -- Fix debian/changelog file to reflect the correct version so the packages > are generated correctl. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 24 Jan '24

24 Jan '24

Hi, I just tested with 23.02.7-1 and the issue is gone. So it seems like the patch got released. Best, Tim On 1/24/24 16:55, Stefan Fleischmann wrote: > On Wed, 24 Jan 2024 12:37:04 -0300 Cristóbal Navarro > <cristobal.navarro.g(a)gmail.com> wrote: >> Many thanks >> One question? Do we have to apply this patch (and recompile slurm i >> guess) only on the compute-node with problems? >> Also, I noticed the patch now appears as "obsolete", is that ok? > We have Slurm installed on a NFS share, so what I did was to recompile > it and then I only replaced the library lib/slurm/cgroup_v2.so. Good > enough for now, I've been planning to update to 23.11 anyway soon. > > I suppose it's marked as obsolete because the patch went into a > release. According to the info in the bug report it should have been > included in 23.02.4. > > Cheers, > Stefan > >> On Wed, Jan 24, 2024 at 9:52 AM Stefan Fleischmann <sfle(a)kth.se> >> wrote: >> >>> Turns out I was wrong, this is not a problem in the kernel at all. >>> It's a known bug that is triggered by long bpf logs, see here >>> https://bugs.schedmd.com/show_bug.cgi?id=17210 >>> >>> There is a patch included there. >>> >>> Cheers, >>> Stefan >>> >>> On Tue, 23 Jan 2024 15:28:59 +0100 Stefan Fleischmann <sfle(a)kth.se> >>> wrote: >>>> I don't think there is much for SchedMD to do. As I said since it >>>> is working fine with newer kernels there doesn't seem to be any >>>> breaking change in cgroup2 in general, but only a regression >>>> introduced in one of the latest updates in 5.15. >>>> >>>> If Slurm was doing something wrong with cgroup2, and it >>>> accidentally worked until this recent change, then other kernel >>>> versions should show the same behavior. But as far as I can tell >>>> it still works just fine with newer kernels. >>>> >>>> Cheers, >>>> Stefan >>>> >>>> On Tue, 23 Jan 2024 15:20:56 +0100 >>>> Tim Schneider <tim.schneider1(a)tu-darmstadt.de> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have filed a bug report with SchedMD >>>>> (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the >>>>> support told me they cannot invest time in this issue since I >>>>> don't have a support contract. Maybe they will look into it >>>>> once it affects more people or someone important enough. >>>>> >>>>> So far, I have resorted to using 5.15.0-89-generic, but I am >>>>> also a bit concerned about the security aspect of this choice. >>>>> >>>>> Best, >>>>> >>>>> Tim >>>>> >>>>> On 23.01.24 14:59, Stefan Fleischmann wrote: >>>>>> Hi! >>>>>> >>>>>> I'm seeing the same in our environment. My conclusion is that >>>>>> it is a regression in the Ubuntu 5.15 kernel, introduced with >>>>>> 5.15.0-90-generic. Last working kernel version is >>>>>> 5.15.0-89-generic. I have filed a bug report here: >>>>>> https://bugs.launchpad.net/bugs/2050098 >>>>>> >>>>>> Please add yourself to the affected users in the bug report >>>>>> so it hopefully gets more attention. >>>>>> >>>>>> I've tested with newer kernels (6.5, 6.6 and 6.7) and the >>>>>> problem does not exist there. 6.5 is the latest hwe kernel >>>>>> for 22.04 and would be an option for now. Reverting back to >>>>>> 5.15.0-89 would work as well, but I haven't looked into the >>>>>> security aspects of that. >>>>>> >>>>>> Cheers, >>>>>> Stefan >>>>>> >>>>>> On Mon, 22 Jan 2024 13:31:15 -0300 >>>>>> cristobal.navarro.g at gmail.com wrote: >>>>>> >>>>>>> Hi Tim and community, >>>>>>> We are currently having the same issue (cgroups not working >>>>>>> it seems, showing all GPUs on jobs) on a GPU-compute node >>>>>>> (DGX A100) a couple of days ago after a full update (apt >>>>>>> upgrade). Now whenever we launch a job for that partition, >>>>>>> we get the error message mentioned by Tim. As a note, we >>>>>>> have another custom GPU-compute node with L40s, on a >>>>>>> different partition, and that one works fine. Before this >>>>>>> error, we always had small differences in kernel version >>>>>>> between nodes, so I am not sure if this can be the problem. >>>>>>> Nevertheless, here is the info of our nodes as well. >>>>>>> >>>>>>> *[Problem node]* The DGX A100 node has this kernel >>>>>>> cnavarro at nodeGPU01:~$ uname -a >>>>>>> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 >>>>>>> 20:28:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> *[Functioning node]* The Custom GPU node (L40s) has this >>>>>>> kernel cnavarro at nodeGPU02:~$ uname -a >>>>>>> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 >>>>>>> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> *And the login node *(slurmctld) >>>>>>> ? ~ uname -a >>>>>>> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue >>>>>>> Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >>>>>>> >>>>>>> Any ideas what we should check? >>>>>>> >>>>>>> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 >>>>>>> at tu-darmstadt.de> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am using SLURM 22.05.9 on a small compute cluster. Since I >>>>>>>> reinstalled two of our nodes, I get the following error when >>>>>>>> launching a job: >>>>>>>> >>>>>>>> slurmstepd: error: load_ebpf_prog: BPF load error (No space >>>>>>>> left on device). Please check your system limits (MEMLOCK). >>>>>>>> >>>>>>>> Also the cgroups do not seem to work properly anymore, as I >>>>>>>> am able to see all GPUs even if I do not request them, >>>>>>>> which is not the case on the other nodes. >>>>>>>> >>>>>>>> One difference I found between the updated nodes and the >>>>>>>> original nodes (both are Ubuntu 22.04) is the kernel >>>>>>>> version, which is "5.15.0-89-generic #99-Ubuntu SMP" on the >>>>>>>> functioning nodes and "5.15.0-91-generic #101-Ubuntu SMP" >>>>>>>> on the updated nodes. I could not figure out how to install >>>>>>>> the exact first kernel version on the updated nodes, but I >>>>>>>> noticed that when I reinstall 5.15.0 with this tool: >>>>>>>> https://github.com/pimlie/ubuntu-mainline-kernel.sh, the >>>>>>>> error message disappears. However, once I do that, the >>>>>>>> network driver does not function properly anymore, so this >>>>>>>> does not seem to be a good solution. >>>>>>>> >>>>>>>> Has anyone seen this issue before or is there maybe >>>>>>>> something else I should take a look at? I am also happy to >>>>>>>> just find a workaround such that I can take these nodes >>>>>>>> back online. >>>>>>>> >>>>>>>> I appreciate any help! >>>>>>>> >>>>>>>> Thanks a lot in advance and best wishes, >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> >>>>>>>> >>>> >>> >>

2 1

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Cristóbal Navarro 24 Jan '24

24 Jan '24

Many thanks One question? Do we have to apply this patch (and recompile slurm i guess) only on the compute-node with problems? Also, I noticed the patch now appears as "obsolete", is that ok? On Wed, Jan 24, 2024 at 9:52 AM Stefan Fleischmann <sfle(a)kth.se> wrote: > Turns out I was wrong, this is not a problem in the kernel at all. It's > a known bug that is triggered by long bpf logs, see here > https://bugs.schedmd.com/show_bug.cgi?id=17210 > > There is a patch included there. > > Cheers, > Stefan > > On Tue, 23 Jan 2024 15:28:59 +0100 Stefan Fleischmann <sfle(a)kth.se> > wrote: > > I don't think there is much for SchedMD to do. As I said since it is > > working fine with newer kernels there doesn't seem to be any breaking > > change in cgroup2 in general, but only a regression introduced in one > > of the latest updates in 5.15. > > > > If Slurm was doing something wrong with cgroup2, and it accidentally > > worked until this recent change, then other kernel versions should > > show the same behavior. But as far as I can tell it still works just > > fine with newer kernels. > > > > Cheers, > > Stefan > > > > On Tue, 23 Jan 2024 15:20:56 +0100 > > Tim Schneider <tim.schneider1(a)tu-darmstadt.de> wrote: > > > > > Hi, > > > > > > I have filed a bug report with SchedMD > > > (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the support > > > told me they cannot invest time in this issue since I don't have a > > > support contract. Maybe they will look into it once it affects more > > > people or someone important enough. > > > > > > So far, I have resorted to using 5.15.0-89-generic, but I am also a > > > bit concerned about the security aspect of this choice. > > > > > > Best, > > > > > > Tim > > > > > > On 23.01.24 14:59, Stefan Fleischmann wrote: > > > > Hi! > > > > > > > > I'm seeing the same in our environment. My conclusion is that it > > > > is a regression in the Ubuntu 5.15 kernel, introduced with > > > > 5.15.0-90-generic. Last working kernel version is > > > > 5.15.0-89-generic. I have filed a bug report here: > > > > https://bugs.launchpad.net/bugs/2050098 > > > > > > > > Please add yourself to the affected users in the bug report so it > > > > hopefully gets more attention. > > > > > > > > I've tested with newer kernels (6.5, 6.6 and 6.7) and the problem > > > > does not exist there. 6.5 is the latest hwe kernel for 22.04 and > > > > would be an option for now. Reverting back to 5.15.0-89 would work > > > > as well, but I haven't looked into the security aspects of that. > > > > > > > > Cheers, > > > > Stefan > > > > > > > > On Mon, 22 Jan 2024 13:31:15 -0300 > > > > cristobal.navarro.g at gmail.com wrote: > > > > > > > >> Hi Tim and community, > > > >> We are currently having the same issue (cgroups not working it > > > >> seems, showing all GPUs on jobs) on a GPU-compute node (DGX A100) > > > >> a couple of days ago after a full update (apt upgrade). Now > > > >> whenever we launch a job for that partition, we get the error > > > >> message mentioned by Tim. As a note, we have another custom > > > >> GPU-compute node with L40s, on a different partition, and that > > > >> one works fine. Before this error, we always had small > > > >> differences in kernel version between nodes, so I am not sure if > > > >> this can be the problem. Nevertheless, here is the info of our > > > >> nodes as well. > > > >> > > > >> *[Problem node]* The DGX A100 node has this kernel > > > >> cnavarro at nodeGPU01:~$ uname -a > > > >> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 > > > >> 20:28:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> *[Functioning node]* The Custom GPU node (L40s) has this kernel > > > >> cnavarro at nodeGPU02:~$ uname -a > > > >> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 > > > >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> *And the login node *(slurmctld) > > > >> ? ~ uname -a > > > >> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 > > > >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > > >> > > > >> Any ideas what we should check? > > > >> > > > >> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 at > > > >> tu-darmstadt.de> wrote: > > > >> > > > >>> Hi, > > > >>> > > > >>> I am using SLURM 22.05.9 on a small compute cluster. Since I > > > >>> reinstalled two of our nodes, I get the following error when > > > >>> launching a job: > > > >>> > > > >>> slurmstepd: error: load_ebpf_prog: BPF load error (No space left > > > >>> on device). Please check your system limits (MEMLOCK). > > > >>> > > > >>> Also the cgroups do not seem to work properly anymore, as I am > > > >>> able to see all GPUs even if I do not request them, which is not > > > >>> the case on the other nodes. > > > >>> > > > >>> One difference I found between the updated nodes and the > > > >>> original nodes (both are Ubuntu 22.04) is the kernel version, > > > >>> which is "5.15.0-89-generic #99-Ubuntu SMP" on the functioning > > > >>> nodes and "5.15.0-91-generic #101-Ubuntu SMP" on the updated > > > >>> nodes. I could not figure out how to install the exact first > > > >>> kernel version on the updated nodes, but I noticed that when I > > > >>> reinstall 5.15.0 with this tool: > > > >>> https://github.com/pimlie/ubuntu-mainline-kernel.sh, the error > > > >>> message disappears. However, once I do that, the network driver > > > >>> does not function properly anymore, so this does not seem to be > > > >>> a good solution. > > > >>> > > > >>> Has anyone seen this issue before or is there maybe something > > > >>> else I should take a look at? I am also happy to just find a > > > >>> workaround such that I can take these nodes back online. > > > >>> > > > >>> I appreciate any help! > > > >>> > > > >>> Thanks a lot in advance and best wishes, > > > >>> > > > >>> Tim > > > >>> > > > >>> > > > >>> > > > -- Cristóbal A. Navarro

2 1

slurm-config on NFS-volume
by Werf, C.G. van der (Carel) 24 Jan '24

24 Jan '24

Hi, Among other clusters, I have a simple cluster with 2 nodes, running slurm. 1 node runs : mysqld, slurmdbd, slurmctld and slurmd. The other node, only runs slurmd. Slurm config is in node1: /etc/slurm. A copy of the config is in node2:/etc/slurm. This slurm configuration runs ok. But, as I am normally used to configure on a shared location, where there is only 1 copy of the config, I tried to change the current setup by moving the slurm-config to a shared (NFS) disk. So on both nodes : /etc/slurm -> nfs-server:/shared/etc/slurm But, I have tried several ways... The slurmctld and slurmd daemons will not start when I refer to this "nfs-shared" location. Any ideas what is missing here ? Authorizations seem ok. With Regards, | Carel van der Werf | | Developer/Administrator Linux | ICT-Bèta | Department of Science | Utrecht University |

3 3

error: Couldn't find the specified plugin name for cred/munge looking at all files
by Jesse Aiton 24 Jan '24

24 Jan '24

Hello Slurm Folks, I have a weird issue where on the same server, which acts as both a controller and a node, slurmctld can’t find cred_munge.so slurmctld: debug3: Trying to load plugin /app/slurm-24.0.8/lib/slurm/cred_munge.so slurmctld: debug4: /app/slurm-24.0.8/lib/slurm/cred_munge.so: Does not exist or not a regular file. slurmctld: error: Couldn't find the specified plugin name for cred/munge looking at all files slurmctld: error: cannot open plugin directory /app/slurm-24.0.8/lib/slurm slurmctld: error: cannot find cred plugin for cred/munge slurmctld: error: cannot create cred context for cred/munge slurmctld: fatal: failed to initialize cred plugin But slurmd can: slurmd: debug3: Trying to load plugin /app/slurm-24.0.8/lib/slurm/cred_munge.so slurmd: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x180800 slurmd: cred/munge: init: Munge credential signature plugin loaded slurmd: debug3: Success. This is on Ubuntu 20.04 and happens both with Slurm 20.11.09 and 24.0.8 Thank you, Jesse # slurm.conf file generated by configurator easy.html. # Put this file on all nodes of your cluster. # See the slurm.conf man page for more information. # ClusterName=prod-cluster SlurmctldHost=controller # #MailProg=/bin/mail #MpiDefault= #MpiParams=ports=#-# ProctrackType=proctrack/cgroup ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid #SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid #SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm #SlurmdUser=root StateSaveLocation=/var/spool/slurmctld #SwitchType= TaskPlugin=task/affinity,task/cgroup # # # TIMERS #KillWait=30 #MinJobAge=300 #SlurmctldTimeout=120 #SlurmdTimeout=300 # # # SCHEDULING SchedulerType=sched/backfill SelectType=select/cons_tres # # # LOGGING AND ACCOUNTING #AccountingStorageType= #JobAcctGatherFrequency=30 #JobAcctGatherType= #SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld.log #SlurmdDebug=info SlurmdLogFile=/var/log/slurmd.log # # # COMPUTE NODES NodeName=controller CPUs=1 State=UNKNOWN NodeName=node CPUs=1 State=UNKNOWN PartitionName=prod-part Nodes=ALL Default=YES MaxTime=INFINITE State=UP

3 5

Slurm version 23.11.2 is now available
by Tim McMullan 23 Jan '24

23 Jan '24

We are pleased to announce the availability of Slurm version 23.11.2. The 23.11.2 release includes a number of fixes to stability and various bug fixes. Some notable changes include several fixes to the new scontrol reconfigure method, including one that could result in jobs getting cancelled prematurely, a couple errors that resulted in the backup slurmctld stopping on fail-back, and an issue during upgrades with older MySQL versions with a small max_allowed_packet value for sites with a large number of associations. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Tim > * Changes in Slurm 23.11.2 > ========================== > -- slurmrestd - Reject single http query with multiple path requests. > -- Fix launching Singularity v4.x containers with srun --container by setting > .process.terminal to true in generated config.json when step has > pseudoterminal (--pty) requested. > -- Fix loading in dyanmic/cloud node jobs after net_cred expired. > -- Fix cgroup null path error on slurmd/slurmstepd tear down. > -- data_parser/v0.0.40 - Prevent failure if accounting is disabled, instead > issue a warning if needed data from the database can not be retrieved. > -- openapi/slurmctld - Prevent failure if accounting is disabled. > -- Prevent slurmscriptd processing delays from blocking other threads in > slurmctld while trying to launch various scripts. This is additional work > for a fix in 23.02.6. > -- Fix memory leak when receiving alias addrs from controller. > -- scontrol - Accept `scontrol token lifespan=infinite` to create tokens that > effectively do not expire. > -- Avoid errors when Slurmdb accounting disabled when '--json' or '--yaml' is > invoked with CLI commands and slurmrestd. Add warnings when query would > have populated data from Slurmdb instead of errors. > -- Fix slurmctld memory leak when running job with --tres-per-task=gres:shard:# > -- Fix backfill trying to start jobs outside of backfill window. > -- Fix oversubscription on partitions with PreemptMode=OFF. > -- Preserve node reason on power up if the node is downed or drained. > -- data_parser/v0.0.40 - Avoid aborting when invoking a not implemented > parser. > -- data_parser/v0.0.40 - Fix how nice values are parsed for job submissions. > -- data_parser/v0.0.40 - Fix regression where parsing error did not result in > invalid request being rejected. > -- Fix segfault in front-end node registration. > -- Prevent jobs using none typed gpus from being killed by the controller after > a reconfig or restart. > -- Fix deadlock situation in the dbd when adding associations. > -- Update default values of text/blob columns when updating from old mysql > versions in more situations. This improves a previous fix to handle an > uncommon case when upgrading mysql/mariadb. > -- Fix rpmbuild in openSUSE/SLES due to incorrect mariadb dependency. > -- Fix compilation on RHEL 7. > -- When upgrading the slurmdbd to 23.11, avoid generating a query to update > the association table that is larger than max_allowed_packet which would > result in an upgrade failure. > -- Fix rare deadlock when a dynamic node registers at the same time that a > once per minute background task occurs. > -- Fix build issue on 32-bit systems. > -- data_parser/v0.0.40 - Fix enumerated strings in OpenAPI specification not > have type field specified. > -- Improve scontrol show job -d information of used shared gres (shard/mps) > topology. > -- Allow Slurm to compile without MUNGE if --without-munge is used as an > argument to configure. > -- accounting_storage/mysql - Fix usage query to use new lineage column > instead of lft/rgt. > -- slurmrestd - Improve handling of missing parsers when content plugins > expect parsers not loaded. > -- slurmrestd - Correct parsing of StepIds when querying jobs. > -- slurmrestd - Improve error from parsing failures of lists. > -- slurmrestd - Improve parsing of singular values for lists. > -- accounting_storage/mysql - Fix PrivateData=User when listing associations. > -- Disable sorting of dynamic nodes to avoid issues when restarting with > heterogenous jobs that cause jobs to abort on restart. > -- Don't allow deletion of non-dynamic nodes. > -- accounting_storage/mysql - Fix issue adding partition based associations. > -- Respect non-"slurm" settings for I_MPI_HYDRA_BOOTSTRAP and HYDRA_BOOTSTRAP > and avoid injecting the --external-launcher option which will cause > mpirun/mpiexec to fail with an unexpected argument error. > -- Fix bug where scontrol hold would change node count for jobs with > implicitly defined node counts. > -- data_parser/v0.0.40 - Fix regression of support for "hold" in > job description. > -- Avoid sending KILL RPCs to unresolvable POWERING_UP and POWERED_DOWN nodes. > -- data_parser/v0.0.38 - Fix several potential NULL dereferences that could > cause slurmrestd to crash. > -- Add --gres-flags=one-task-per-sharing. Do not allow different tasks in to be > allocated shared gres from the same sharing gres. > -- Add SelectTypeParameters=ENFORCE_BINDING_GRES and ONE_TASK_PER_SHARING_GRES. > This gives default behavior for a job's --gres-flags. > -- Alter the networking code to try connecting to the backup controllers > if the DNS lookup for the primary SlurmctldHost fails. > -- Alter the name resolution to only log at verbose() in client commands > on failures. This allows for HA setups where the DNS entries are withdrawn > for some SlurmctldHost entries without flooding the user with errors. > -- Prevent slurmscriptd PID leaks when running slurmctld in foreground mode. > -- Open all slurmctld listening ports at startup, and persist throughout. > This also changes the backup slurmctld process to open the SlurmctldPort > range, instead of only the first. > -- Fix backup slurmctld shutting down instead of resuming standby duty if it > took control. > -- Fix race condition that delayed the primary slurmctld resuming when taking > control from a backup controller. > -- srun - Ensure processed messages are meant for this job in case of a > rapidly-reused TCP port. > -- srun - Prevent step launch failure while waiting for step allocation if > a stray message is received. > -- Fix backup slurmctld to be able to respond to configless config file > requests correctly. > -- Fix slurmctld crashing when recovering from a failed reconfigure. > -- Fix slurmscriptd operation after recovering from a failed reconfigure. -- Tim McMullan Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

GPU devices mapping with job's cgroup in cgroups v2 using eBPF
by Mahendra Paipuri 23 Jan '24

23 Jan '24

Hello all, Happy new year! We have recently upgraded the cgroups on our SLURM cluster to v2. In cgroups v1, the interface `/devices.list` used to have the information of which device has been attached to that particular cgroup. From my understanding, cgroups v2 use eBPF to manage devices and so as SLURM to manage the GPUs. I was looking for a way to be able to programatically determine the job cgroups to device mapping and I came across this thread ( https://bugzilla.redhat.com/show_bug.cgi?id=1717396) which has a similar discussion in the context of VMs. So, I have used `bpftool` to inspect the job cgroups. An example output: ``` # /tmp/bpftool cgroup list /sys/fs/cgroup/system.slice/slurmstepd.scope/job_1956132 ID AttachType AttachFlags Name ``` When I add `effective` flag, I see the attached eBPF program ``` # /tmp/bpftool cgroup list /sys/fs/cgroup/system.slice/slurmstepd.scope/job_1956132 effective ID AttachType Name 4197 cgroup_device Slurm_Cgroup_v2 ``` >From my understand, `effective` flag shows the inherited eBPF programs as well. So, my question is at which level of cgroups the eBPF program is attached? I tried to inspect at various levels but all of them returned none. Then looking into translated byte code of eBPF program, I get the following ``` # /tmp/bpftool prog dump xlated id 4197 0: (61) r2 = *(u32 *)(r1 +0) 1: (54) w2 &= 65535 2: (61) r3 = *(u32 *)(r1 +0) 3: (74) w3 >>= 16 4: (61) r4 = *(u32 *)(r1 +4) 5: (61) r5 = *(u32 *)(r1 +8) 6: (55) if r2 != 0x2 goto pc+4 7: (55) if r4 != 0xc3 goto pc+3 8: (55) if r5 != 0x0 goto pc+2 9: (b7) r0 = 0 10: (95) exit 11: (55) if r2 != 0x2 goto pc+4 12: (55) if r4 != 0xc3 goto pc+3 13: (55) if r5 != 0x1 goto pc+2 14: (b7) r0 = 0 15: (95) exit 16: (55) if r2 != 0x2 goto pc+4 17: (55) if r4 != 0xc3 goto pc+3 18: (55) if r5 != 0x2 goto pc+2 19: (b7) r0 = 0 20: (95) exit 21: (55) if r2 != 0x2 goto pc+4 22: (55) if r4 != 0xc3 goto pc+3 23: (55) if r5 != 0x3 goto pc+2 24: (b7) r0 = 1 25: (95) exit 26: (b7) r0 = 1 27: (95) exit ``` >From the output, it is clear that GPU:3 (among 0,1,2,3) is the one that is attached to that job's cgroup. However, I was looking for a way to dump eBPF maps that can directly provide the major, minor numbers and permissions of device as discussed in the comment (https://bugzilla.redhat.com/show_bug.cgi?id=1717396#c5). When I inspect eBPF program, I dont see any maps associated. ``` # /tmp/bpftool prog list id 4197 4197: cgroup_device name Slurm_Cgroup_v2 tag 1a261c8a913ff67c gpl loaded_at 2024-01-02T08:19:56+0100 uid 0 xlated 224B jited 142B memlock 4096B ``` So, my second question is how can I get a similar information as `map dump` that can give us the device's major minor numbers directly instead of parsing the byte code from `prog dump`? I am still discovering the eBPF ecosystem so if I am missing something very obvious, please let me know. I would really appreciate that. Cheers! Regards Mahendra

2 1

Re: [slurm-users] slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 23 Jan '24

23 Jan '24

Hi, I have filed a bug report with SchedMD (https://bugs.schedmd.com/show_bug.cgi?id=18623), but the support told me they cannot invest time in this issue since I don't have a support contract. Maybe they will look into it once it affects more people or someone important enough. So far, I have resorted to using 5.15.0-89-generic, but I am also a bit concerned about the security aspect of this choice. Best, Tim On 23.01.24 14:59, Stefan Fleischmann wrote: > Hi! > > I'm seeing the same in our environment. My conclusion is that it is a > regression in the Ubuntu 5.15 kernel, introduced with 5.15.0-90-generic. > Last working kernel version is 5.15.0-89-generic. I have filed a bug > report here: https://bugs.launchpad.net/bugs/2050098 > > Please add yourself to the affected users in the bug report so it > hopefully gets more attention. > > I've tested with newer kernels (6.5, 6.6 and 6.7) and the problem does > not exist there. 6.5 is the latest hwe kernel for 22.04 and would be an > option for now. Reverting back to 5.15.0-89 would work as well, but I > haven't looked into the security aspects of that. > > Cheers, > Stefan > > On Mon, 22 Jan 2024 13:31:15 -0300 > cristobal.navarro.g at gmail.com wrote: > >> Hi Tim and community, >> We are currently having the same issue (cgroups not working it seems, >> showing all GPUs on jobs) on a GPU-compute node (DGX A100) a couple >> of days ago after a full update (apt upgrade). Now whenever we launch >> a job for that partition, we get the error message mentioned by Tim. >> As a note, we have another custom GPU-compute node with L40s, on a >> different partition, and that one works fine. >> Before this error, we always had small differences in kernel version >> between nodes, so I am not sure if this can be the problem. >> Nevertheless, here is the info of our nodes as well. >> >> *[Problem node]* The DGX A100 node has this kernel >> cnavarro at nodeGPU01:~$ uname -a >> Linux nodeGPU01 5.15.0-1042-nvidia #42-Ubuntu SMP Wed Nov 15 20:28:30 >> UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> *[Functioning node]* The Custom GPU node (L40s) has this kernel >> cnavarro at nodeGPU02:~$ uname -a >> Linux nodeGPU02 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 >> UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> *And the login node *(slurmctld) >> ? ~ uname -a >> Linux patagon-master 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 >> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux >> >> Any ideas what we should check? >> >> On Thu, Jan 4, 2024 at 3:03?PM Tim Schneider <tim.schneider1 at >> tu-darmstadt.de> wrote: >> >>> Hi, >>> >>> I am using SLURM 22.05.9 on a small compute cluster. Since I >>> reinstalled two of our nodes, I get the following error when >>> launching a job: >>> >>> slurmstepd: error: load_ebpf_prog: BPF load error (No space left on >>> device). Please check your system limits (MEMLOCK). >>> >>> Also the cgroups do not seem to work properly anymore, as I am able >>> to see all GPUs even if I do not request them, which is not the >>> case on the other nodes. >>> >>> One difference I found between the updated nodes and the original >>> nodes (both are Ubuntu 22.04) is the kernel version, which is >>> "5.15.0-89-generic #99-Ubuntu SMP" on the functioning nodes and >>> "5.15.0-91-generic #101-Ubuntu SMP" on the updated nodes. I could >>> not figure out how to install the exact first kernel version on the >>> updated nodes, but I noticed that when I reinstall 5.15.0 with this >>> tool: https://github.com/pimlie/ubuntu-mainline-kernel.sh, the >>> error message disappears. However, once I do that, the network >>> driver does not function properly anymore, so this does not seem to >>> be a good solution. >>> >>> Has anyone seen this issue before or is there maybe something else I >>> should take a look at? I am also happy to just find a workaround >>> such that I can take these nodes back online. >>> >>> I appreciate any help! >>> >>> Thanks a lot in advance and best wishes, >>> >>> Tim >>> >>> >>>

2 1

Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres)
by Kherfani, Hafedh (Professional Services, TC) 23 Jan '24

23 Jan '24

Hello Experts, I'm a new Slurm user (so please bare with me :) ...). Recently we've deployed Slurm version 23.11 on a very simple cluster, which consists of a Master node (acting as a Login & Slurmdbd node as well), a Compute Node which has a NVIDIA HGX A100-SXM4-40GB GPU, detected as 4 x GPU's: GPU [0-4], and a Storage Array presenting/sharing the NFS disk (where users' home directories will be created as well). The problem is that I've never been able to run a simple/dummy batch script in a parallel way using the 4 GPU's. In fact, running the same command "sbatch gpu-job.sh" multiple times shows that only one single job is running, while the other jobs are in a pending state: [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 214 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 215 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 216 [slurmtest@c-a100-master test-batch-scripts]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 216 gpu gpu-job slurmtest PD 0:00 1 (None) 215 gpu gpu-job slurmtest PD 0:00 1 (Priority) 214 gpu gpu-job slurmtest PD 0:00 1 (Priority) 213 gpu gpu-job slurmtest PD 0:00 1 (Priority) 212 gpu gpu-job slurmtest PD 0:00 1 (Resources) 211 gpu gpu-job slurmtest R 0:14 1 c-a100-cn01 PS: CPU jobs (i.e. using the default debug partition, without call the GPU Gres) can be run in parallel. The issue with running parallel jobs is only seen when using the GPU's as Gres. I've tried many combinations of settings in gres.conf and slurm.conf, many (if not most) of these combinations would result in error messages in slurmctld and slurmd logs. The current gres.conf and slurm.conf contents is shown below. Even though it doesn't give errors when restarting slurmctld and slurmd services (on master and compute nodes, resp.), but as I said, it doesn't allow jobs to be executed in parallel. Batch script contents shared below as well, in order to give more clarity on what I'm trying to do: [root@c-a100-master slurm]# cat gres.conf | grep -v "^#" NodeName=c-a100-cn01 AutoDetect=nvml Name=gpu Type=A100 File=/dev/nvidia[0-3] [root@c-a100-master slurm]# cat slurm.conf | grep -v "^#" | egrep -i "AccountingStorageTRES|GresTypes|NodeName|partition" GresTypes=gpu AccountingStorageTRES=gres/gpu NodeName=c-a100-cn01 Gres=gpu:A100:4 CPUs=64 Boards=1 SocketsPerBoard=1 CoresPerSocket=32 ThreadsPerCore=2 RealMemory=515181 State=UNKNOWN PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP PartitionName=gpu Nodes=ALL MaxTime=10:0:0 [slurmtest@c-a100-master test-batch-scripts]$ cat gpu-job.sh #!/bin/bash #SBATCH --job-name=gpu-job #SBATCH --partition=gpu #SBATCH --nodes=1 #SBATCH --gpus-per-node=4 #SBATCH --gres=gpu:4 #SBATCH --tasks-per-node=1 #SBATCH --output=gpu_job_output.%j # Output file name (replaces %j with job ID) #SBATCH --error=gpu_job_error.%j # Error file name (replaces %j with job ID) hostname date sleep 40 pwd Any help on which changes need to be made to the config files (mainly slurm.conf and gres.cong) and/or the batch script, so that multiple jobs can be in a "Running" state at the same time (in parallel) ? Thanks in advance for your help ! Best regards, Hafedh Kherfani

8 10

propose environment variables SLURM_STDOUT, SLURM_STDERR, SLURM_STDIN
by urbanjost 22 Jan '24

22 Jan '24

RE: Placing the full pathname of the job stdout in an environment variable Would others find it useful if new variables were added that contained the full pathnames of the standard input, error and input files of batch jobs? ## SYNOPSIS Proposed new environment variables SLURM_STDOUT,SLURM_STDERR, and SLURM_STDIN pointing to the full pathnames of the output, error output, and input files of a task would be a useful feature for batch jobs in particular. ## PROBLEM There are cases where it is desirable to have a job process the standard files (stdin, stdout, stderr) from a batch job from within the job instead of later via an epilog or via post-processing. Just a few examples where the job might want to reference the full pathname of stdout: * copy the file to a remote machine (via scp(1), for example) * move the file from local or scratch space to permanent globally accessible storage * remove the file depending on completion status * mail the output * archive it * post-process it Slurm commands like scontrol(1) and squeue(1) can help you locate the file names. But the names returned do not always expand all the macros allowed when specifying the names, and require calling a Slurm command (which might overload Slurm if millions of jobs are submitted or cause problems if Slurm is not responding for any reason(?)). For example: #!/bin/bash # Minimalist submit script for sbatch(1) #SBATCH --nodes 1-1 --ntasks=1 --time 0-0:1:00 --chdir=/tmp #SBATCH --output=stdout.%j #SBATCH --error=stderr.%A:%a:%J:%j:%N:%n:%s:%t:%u.out # query filenames via squeue(1) export SLURM_STDOUT=$(squeue --noheader --Format=STDOUT: --job=$SLURM_JOBID) export SLURM_STDERR=$(squeue --noheader --Format=STDERR: --job=$SLURM_JOBID) # PS: Current documentation for squeue(1) does not explicitly tell the # user a null size works. # query filenames via scontrol(1) declare -x $(scontrol show job=$SLURM_JOBID|grep StdOut) declare -x $(scontrol show job=$SLURM_JOBID|grep StdErr) cat <<EOF SLURM_STDOUT=$SLURM_STDOUT SLURM_STDERR=$SLURM_STDERR StdOut=$StdOut StdErr=$StdErr EOF ls stdout.* stderr.* The resulting output shows either none or some macros are expanded by the commands (and I am not sure all commands will always return a full pathname(?)) SLURM_STDOUT=/home/urbanjs/venus/V600/stdout.%j SLURM_STDERR=/home/urbanjs/venus/V600/stderr.%A:%a:%J:%j:%N:%n:%s:%t:%u.out StdOut=/home/urbanjs/venus/V600/stdout.96 StdErr=/home/urbanjs/venus/V600/stderr.96:4294967294:%J:96:%N:%n:%s:%t:urbanjs.out stderr.96:4294967294:96:96:mercury:0:4294967294:0:urbanjs.out stdout.96 One currently available work-around would be that the user just avoid the filename macros and always specify the filenames using a standard convention (which would obviously have to be specific to a specific platform). This is error-prone as it requires the user to strictly follow a protocol where the pathnames are specified that can be overridden by command-line options on sbatch(1) and so-on. Alternatively instead of the new environment variables Slurm could have a command option that returns the fully expanded names always expanding the macros. But if you query Slurm you have to worry about scaleability. Is it OK for a 100 000, jobs to query Slurm simulataneously via squeue(1) or scontrol(1)? What happens if for external reasons the Slurm daemons are down or not responding? As a current work-around using commands like realpath(1), stat(1), ls(1),find(1),and getting pids with fuser(1), pidof(1), and ps(1) become attractive. I find using the realpath(1) command at the top of the job and $SLURM_TASK_ID works the best. realpath(1) may not be available on all platforms(?) as well as the /proc/$PID/fd/1 file, in which case the other commands might be used to the same effect but it would be nice to have a simple standard way that does not require calling a command. The proposed environment variables seem the most obvious solution. # do this early in the job in case the user changes or closes # file descriptor 1 ... export SLURM_STDOUT=$(realpath /proc/$SLURM_TASK_PID/fd/1) # create softlink OUTPUT to stdout of job ln -s -f $SLURM_STDOUT OUTPUT # now user can do things like mailx(1) or scp(1) the OUTPUT file. Other reasons I might want to know the stdout pathname are to minimize network traffic and filespace on central servers. I might specify the output of a job goes to scratch or local storage or a memory-resident file system and move it all to a location on another system or a long-term archive instead of a central area that might be filled by other jobs or that I have a quota on, etc. So since the stdout might not be easily accessible from my other sessions also having something like "scontrol write batch_stdout=$SLURM_JOBID" that shows the stdout of the job would also be useful. Note the LSF job scheduler bpeek(1) command allows for a tail(1)-like interface to the job stdout to support the stdout not being accessible from all system access points, just as described. The current situation where in typical usage the output file is often in a globally mounted area and I can just access it from other nodes with grep(1), tail(1), and so on covers many users needs in simple cluster configurations; but there are several scenarios as described above primarily encountered by people running many thousands of single-node jobs where assigning output to a local scratch device and then selectively processing at job termination is preferable. As mentioned, those users can specify such a local name with "#SBATCH --output" and then, knowing it, process it as desired but having something like $SLURM_STDOUT being the full pathname to the file is much more generic. ## SUMMARY So adding the variables creates an ability for users to reliably access the standard files of a job with a low-load scalable method; and being a SLURM_* module it could be much more reliably accessed via prolog and epilog scripts and module(1) scripts as well. ## PS: Related to this would be instead of just --chdir a --mkdir and --scratch #SBATCH option would be useful that expanded file macros that would be similar to --chdir but would make sure the directory existed (using only user privelages) and a --scratch that would do the same as --mkdir but remove the directory at job termination. urbanjs@mercury:~/SLURM/proposals$ Sent with [Proton Mail](https://proton.me/) secure email.

3 2

slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK).
by Tim Schneider 22 Jan '24

22 Jan '24

Hi, I am using SLURM 22.05.9 on a small compute cluster. Since I reinstalled two of our nodes, I get the following error when launching a job: slurmstepd: error: load_ebpf_prog: BPF load error (No space left on device). Please check your system limits (MEMLOCK). Also the cgroups do not seem to work properly anymore, as I am able to see all GPUs even if I do not request them, which is not the case on the other nodes. One difference I found between the updated nodes and the original nodes (both are Ubuntu 22.04) is the kernel version, which is "5.15.0-89-generic #99-Ubuntu SMP" on the functioning nodes and "5.15.0-91-generic #101-Ubuntu SMP" on the updated nodes. I could not figure out how to install the exact first kernel version on the updated nodes, but I noticed that when I reinstall 5.15.0 with this tool: https://github.com/pimlie/ubuntu-mainline-kernel.sh, the error message disappears. However, once I do that, the network driver does not function properly anymore, so this does not seem to be a good solution. Has anyone seen this issue before or is there maybe something else I should take a look at? I am also happy to just find a workaround such that I can take these nodes back online. I appreciate any help! Thanks a lot in advance and best wishes, Tim

2 1

Tried setting up GANG scheduling for timeslicing, but jobs are not alternating
by Francisco José Letterio 22 Jan '24

22 Jan '24

I'm trying to set up GANG scheduling with Slurm on my single-node server so the people at the lab can run experiments without blocking each other (so if say someone has to run some code that takes days to finish, other jobs that take less have the chance to run alternated with it and so they don't have to wait days until they can run) I followed the GANG scheduling slurm.conf setup tutorial on the slurm website, and to check if it's working properly I launched a bunch of jobs that print the current time and then sleep for a while. But when I check squeue, the jobs never alternate, they run sequentially one after the other. Why is this happening? Here's my slurm.conf file: # See the slurm.conf man page for more information. ClusterName=localcluster SlurmctldHost=localhost ProctrackType=proctrack/linuxproc ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/lib/slurm/slurmd SlurmUser=slurm StateSaveLocation=/var/lib/slurm/slurmctld TaskPlugin=task/none # TIMERS InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 # SCHEDULING #Seteado para que cada laburo alterne cada 15 segundos SchedulerTimeSlice=15 SchedulerType=sched/builtin SelectType=select/linear SelectTypeParameters=CR_Memory PreemptMode=GANG # LOGGING AND ACCOUNTING JobCompType=jobcomp/none JobAcctGatherFrequency=30 SlurmctldDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurm/slurmd.log # COMPUTE NODES NodeName=lab04 CPUs=48 CoresPerSocket=12 ThreadsPerCore=2 State=UNKNOWN RealMemory=257249 PartitionName=LocalQ Nodes=ALL Default=YES MaxTime=INFINITE State=UP OverSubscribe=FORCE:6 DefMemPerNode=257249 MaxMemPerNode=257249 >From what I understand, SchedulerTimeSlice=15 means that jobs should alternate running every 15 seconds. This is the job I'm launching with sbatch (launching many copies of this job one after the other): #!/bin/bash #SBATCH -J test # Job name #SBATCH -o job.%j.out # Name of stdout output file (%j expands to %jobId) #SBATCH -N 1 # Total number of nodes requested echo "Test output from Slurm Testjob" date sleep 10 date sleep 10 date sleep 10 date sleep 10 I would expect jobs to print one or two dates, then the Slurm scheduler comes and lets another job run in the meantime, and then the final print(s) come with a delay way greater than 10secs However after launching many copies of this job this is what I see on squeue : Before the first job launched is done: JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 18 LocalQ test eiarussi PD 0:00 1 (Resources) 19 LocalQ test eiarussi PD 0:00 1 (Priority) 20 LocalQ test eiarussi PD 0:00 1 (Priority) 21 LocalQ test eiarussi PD 0:00 1 (Priority) 17 LocalQ test eiarussi R 0:31 1 lab04 After that job ends: JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 19 LocalQ test eiarussi PD 0:00 1 (Resources) 20 LocalQ test eiarussi PD 0:00 1 (Priority) 21 LocalQ test eiarussi PD 0:00 1 (Priority) 18 LocalQ test eiarussi R 0:02 1 lab04 Why did the first job run for 30+ seconds uninterrupted, instead of another job being allowed to run after 15 seconds? Am I misunderstanding how GANG scheduling works, or is the a problem in my conf file?

1 0

MIG-Slice: Unavailable GRES
by Dražen Jalšovec 22 Jan '24

22 Jan '24

Hi, We are testing the MIG deployment on our new slurm compute node with 4 x H100 GPUs. It looks like everything is configured correctly but we have a problem accessing mig devices. When I submit jobs requesting a mig gpu device #SBATCH --gres=gpu:H100_1g.10gb:1, the jobs get submitted to the node, but only 4 jobs get executed and all other jobs fail. I was able to "solve it" by adding code to submit to the mig device UUID. Am I missing something in the slurm/gres/cgroups configuration that would automatically assign correct CUDA_VISIBLE_DEVICES MIG UUID to the job? If I submit 10 jobs requesting --gres=gpu:H100_1g.10gb:1 8 jobs start running: #1 - CUDA_VISIBLE_DEVICES=0 - jobs 1-4 run on one of the mig devices configured on GPU-device ... #8 - CUDA_VISIBLE_DEVICES=7 - jobs 5-8 fail with the message that there are no available CUDA_VISIBLE_DEVICES ... #10 - CUDA_VISIBLE_DEVICES=? - after waiting for free mig resources jobs 9-10 runs or fails depending if GPU-device is free (migs are free) $ nvidia-smi --query-gpu=uuid --id=0 --format=csv,noheader GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3 $ nvidia-smi --query-gpu=uuid --id=7 --format=csv,noheader No devices were found The slurm assigns CUDA_VISIBLE_DEVICES numbers based on the requested resources: Request: gpu:H100_1g.10gb:1 - CUDA_VISIBLE_DEVICES=0 gpu:H100_1g.10gb:2 - CUDA_VISIBLE_DEVICES=0,1 ... Any job with H100_1g.10gb request is going to get device number between 0 and 7 gpu:H100_1g.10gb:8 - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 gpu:H100_2g.20gb:1 - CUDA_VISIBLE_DEVICES=8 Any job with H100_2g.20gb request is going to get device number between 8 and 10 ... gpu:H100_2g.10gb:3 - CUDA_VISIBLE_DEVICES=8,9,10 gpu:H100_3g.40gb:1 - CUDA_VISIBLE_DEVICES=11 gpu:H100_3g.40gb:2 - CUDA_VISIBLE_DEVICES=12 gpu:H100_7g.80gb:1 - CUDA_VISIBLE_DEVICES=13 I was able to "solve it" by adding this code to my slurm script: # Get the CUDA index using srun and awk CUDA_INDEX=$(srun env | grep CUDA_VISIBLE_DEVICES | awk -F '=' '{print $2}') ## GPU0 1g.10gb ## MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a ## MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664 ## MIG-d3ab0675-d318-5e53-b487-b50695cf2e00 ## MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0 ## MIG-85dd76da-c994-5830-adf0-467c66ae1b95 ## MIG-29a6d43b-882e-5b79-868a-15bb2c770b82 ## MIG-558debd2-dc13-5406-9256-73ef4f279737 ## GPU1 2g.20gb ## MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f ## MIG-0a804c0c-aa27-5993-97cb-cedb854735ce ## MIG-8af9af6d-8720-5763-81e2-83afc43eb42b ## GPU1 1g.10gb ## MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a ## GPU2 3g.40gb ## MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b ## MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc ## GPU3 7g.80gb ## MIG-ec043869-9176-577e-bac0-46c8411e4e37 # Define the list of UUIDs UUIDS=( "MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a" "MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664" "MIG-d3ab0675-d318-5e53-b487-b50695cf2e00" "MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0" "MIG-85dd76da-c994-5830-adf0-467c66ae1b95" "MIG-29a6d43b-882e-5b79-868a-15bb2c770b82" "MIG-558debd2-dc13-5406-9256-73ef4f279737" "MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a" "MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f" "MIG-0a804c0c-aa27-5993-97cb-cedb854735ce" "MIG-8af9af6d-8720-5763-81e2-83afc43eb42b" "MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b" "MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc" "MIG-ec043869-9176-577e-bac0-46c8411e4e37" ) # Assign the UUID based on the CUDA index SELECTED_UUID=${UUIDS[$CUDA_INDEX]} # Print the selected UUID echo $SELECTED_UUID export CUDA_VISIBLE_DEVICES=$SELECTED_UUID echo "CUDA_VISIBLE_DEVICES set to: "$SELECTED_UUID echo "Test: "$CUDA_VISIBLE_DEVICES CUDA_VISIBLE_DEVICES=$SELECTED_UUID python3.8 gpu_script.py Result: Script start MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a CUDA_VISIBLE_DEVICES set to: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a Matrix calculation on CUDA device completed successfully. Script end +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 7 0 28686 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 8 0 28744 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 9 0 28300 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 10 0 28512 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 11 0 28506 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 12 0 28516 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 13 0 28511 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 1 9 0 28552 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | +---------------------------------------------------------------------------------------+ This is the current setup: RockyLinux 9.2 slurm-22.05.9-1.el9.x86_64 CUDA Version: 12.1 Driver Version: 530.30.02 # nvidia-smi -L GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3) MIG 1g.10gb Device 0: (UUID: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a) MIG 1g.10gb Device 1: (UUID: MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664) MIG 1g.10gb Device 2: (UUID: MIG-d3ab0675-d318-5e53-b487-b50695cf2e00) MIG 1g.10gb Device 3: (UUID: MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0) MIG 1g.10gb Device 4: (UUID: MIG-85dd76da-c994-5830-adf0-467c66ae1b95) MIG 1g.10gb Device 5: (UUID: MIG-29a6d43b-882e-5b79-868a-15bb2c770b82) MIG 1g.10gb Device 6: (UUID: MIG-558debd2-dc13-5406-9256-73ef4f279737) GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1439a39c-948b-0657-98e9-aff8595a8729) MIG 2g.20gb Device 0: (UUID: MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f) MIG 2g.20gb Device 1: (UUID: MIG-0a804c0c-aa27-5993-97cb-cedb854735ce) MIG 2g.20gb Device 2: (UUID: MIG-8af9af6d-8720-5763-81e2-83afc43eb42b) MIG 1g.10gb Device 3: (UUID: MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a) GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-3b182fed-bfdf-bc11-d1bd-c41fd1468d2a) MIG 3g.40gb Device 0: (UUID: MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b) MIG 3g.40gb Device 1: (UUID: MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc) GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-7eba9d78-e908-3db8-2633-a269aeec395e) MIG 7g.80gb Device 0: (UUID: MIG-ec043869-9176-577e-bac0-46c8411e4e37) $ scontrol show node node16 NodeName=node16 Arch=x86_64 CoresPerSocket=1 CPUAlloc=1 CPUEfctv=112 CPUTot=112 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 NodeAddr=node16 NodeHostName=node16 Version=22.05.9 OS=Linux 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023 RealMemory=1030000 AllocMem=0 FreeMem=1011801 Sockets=112 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpu_H100 BootTime=2024-01-18T13:52:27 SlurmdStartTime=2024-01-19T11:25:44 LastBusyTime=2024-01-19T10:59:13 CfgTRES=cpu=112,mem=1030000M,billing=112,gres/gpu=14 AllocTRES=cpu=1 CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s $ scontrol show partition gpu_H100 PartitionName=gpu_H100 AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=node16 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=112 TotalNodes=1 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=112,mem=1030000M,node=1,billing=112,gres/gpu=14 slurm.conf: AccountingStorageTRES=gres/gpu,gres/gpu:A100,gres/gpu:H100,gres/gpu:H100_1g.10gb,gres/gpu:H100_2g.20gb,gres/gpu:H100_3g.40gb,gres/gpu:H100_7g.80gb GresTypes=gpu ... NodeName=node16 CPUs=112 RealMemory=1030000 Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 PartitionName=gpu_H100 Nodes=node16 Default=NO MaxTime=INFINITE State=UP gres.conf #AutoDetect=nvml # GPU 0 MIG 0 /proc/driver/nvidia/capabilities/gpu0/mig/gi7/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap66,/dev/nvidia-caps/nvidia-cap67 CPUs=0-55 # GPU 0 MIG 1 /proc/driver/nvidia/capabilities/gpu0/mig/gi8/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap75,/dev/nvidia-caps/nvidia-cap76 CPUs=0-55 # GPU 0 MIG 2 /proc/driver/nvidia/capabilities/gpu0/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap84,/dev/nvidia-caps/nvidia-cap85 CPUs=0-55 # GPU 0 MIG 3 /proc/driver/nvidia/capabilities/gpu0/mig/gi10/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap93,/dev/nvidia-caps/nvidia-cap94 CPUs=0-55 # GPU 0 MIG 4 /proc/driver/nvidia/capabilities/gpu0/mig/gi11/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap102,/dev/nvidia-caps/nvidia-cap103 CPUs=0-55 # GPU 0 MIG 5 /proc/driver/nvidia/capabilities/gpu0/mig/gi12/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap111,/dev/nvidia-caps/nvidia-cap112 CPUs=0-55 # GPU 0 MIG 6 /proc/driver/nvidia/capabilities/gpu0/mig/gi13/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap120,/dev/nvidia-caps/nvidia-cap121 CPUs=0-55 # GPU 1 MIG 0 /proc/driver/nvidia/capabilities/gpu1/mig/gi3/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap165,/dev/nvidia-caps/nvidia-cap166 CPUs=0-55 # GPU 1 MIG 1 /proc/driver/nvidia/capabilities/gpu1/mig/gi5/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap183,/dev/nvidia-caps/nvidia-cap184 CPUs=0-55 # GPU 1 MIG 2 /proc/driver/nvidia/capabilities/gpu1/mig/gi6/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap192,/dev/nvidia-caps/nvidia-cap193 CPUs=0-55 # GPU 1 MIG 3 /proc/driver/nvidia/capabilities/gpu1/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap219,/dev/nvidia-caps/nvidia-cap220 CPUs=0-55 # GPU 2 MIG 0 /proc/driver/nvidia/capabilities/gpu2/mig/gi1/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap282,/dev/nvidia-caps/nvidia-cap283 CPUs=0-55 # GPU 2 MIG 1 /proc/driver/nvidia/capabilities/gpu2/mig/gi2/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap291,/dev/nvidia-caps/nvidia-cap292 CPUs=0-55 # GPU 3 MIG 0 /proc/driver/nvidia/capabilities/gpu3/mig/gi0/access NodeName=node16 Name=gpu Type=H100_7g.80gb MultipleFiles=/dev/nvidia3,/dev/nvidia-caps/nvidia-cap408,/dev/nvidia-caps/nvidia-cap409 CPUs=0-55 [2024-01-19T14:31:13.701] debug: gres/gpu: init: loaded [2024-01-19T14:31:13.701] debug: gpu/generic: init: init: GPU Generic plugin loaded [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.702] Gres Name=gpu Type=H100_7g.80gb Count=1 Best, Drazen Jalsovec

1 0

Is there a way to map Azure AD users to NIS?
by Ben Wellborn 20 Jan '24

20 Jan '24

In the past, we would connect AD to NIS users using Centrify and adnisd, but I don't think adnisd can connect to Azure AD (HCI stack). Thanks, Ben

1 0

Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres)
by Jason Macklin 19 Jan '24

19 Jan '24

If you run "scontrol show jobid <jobid>" of your pending job with the "(Resources)" tag you may see more about what is unavailable to your job. Slurm default configs can cause an entire compute node of resources to be "allocated" to a running job regardless of whether it needs all of them or not so you may need to alter one or both of the following settings to allow more than one job to run on a single node at once. You'll find these in your slurm.conf. Don't forget to "scontrol reconf" and even potentially restart both "slurmctld" & "slurmd" on your nodes if you do end up making changes. SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory I hope this helps. Kind regards, Jason ---- Jason Macklin Manager Cyberinfrastructure, Research Cyberinfrastructure 860.837.2142 t | 860.202.7779 m jason.macklin(a)jax.org The Jackson Laboratory Maine | Connecticut | California | Shanghai www.jax.org<http://www.jax.org> The Jackson Laboratory: Leading the search for tomorrow's cures ________________________________ From: slurm-users <slurm-users-bounces(a)lists.schedmd.com> on behalf of slurm-users-request(a)lists.schedmd.com <slurm-users-request(a)lists.schedmd.com> Sent: Friday, January 19, 2024 9:24 AM To: slurm-users(a)lists.schedmd.com <slurm-users(a)lists.schedmd.com> Subject: [EXTERNAL]slurm-users Digest, Vol 75, Issue 31 Send slurm-users mailing list submissions to slurm-users(a)lists.schedmd.com To subscribe or unsubscribe via the World Wide Web, visit https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users or, via email, send a message with subject or body 'help' to slurm-users-request(a)lists.schedmd.com You can reach the person managing the list at slurm-users-owner(a)lists.schedmd.com When replying, please edit your Subject line so it is more specific than "Re: Contents of slurm-users digest..." Today's Topics: 1. Re: Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) (Marko Markoc) 2. Re: Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) (?mit Seren) ---------------------------------------------------------------------- Message: 1 Date: Fri, 19 Jan 2024 06:12:24 -0800 From: Marko Markoc <mmarkoc(a)pdx.edu> To: Slurm User Community List <slurm-users(a)lists.schedmd.com> Subject: Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) Message-ID: <CABnuMe4JTA0e6=VbO8D+To=8FGO+3Byv1dK_MC+OuRitzN5dXg(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" +1 on checking the memory allocation. Or add/check if you have any DefMemPerX set in your slurm.conf On Fri, Jan 19, 2024 at 12:33?AM mohammed shambakey <shambakey1(a)gmail.com> wrote: > Hi > > I'm not an expert, but is it possible that the currently running jobs is > consuming the whole node because it is allocated the whole memory of the > node (so the other 2 jobs had to wait until it finishes)? > Maybe if you try to restrict the required memory for each job? > > Regards > > On Thu, Jan 18, 2024 at 4:46?PM ?mit Seren <uemit.seren(a)gmail.com> wrote: > >> This line also has tobe changed: >> >> >> #SBATCH --gpus-per-node=4 ? #SBATCH --gpus-per-node=1 >> >> --gpus-per-node seems to be the new parameter that is replacing the --gres= >> one, so you can remove the ?gres line completely. >> >> >> >> Best >> >> ?mit >> >> >> >> *From: *slurm-users <slurm-users-bounces(a)lists.schedmd.com> on behalf of >> Kherfani, Hafedh (Professional Services, TC) <hafedh.kherfani(a)hpe.com> >> *Date: *Thursday, 18. January 2024 at 15:40 >> *To: *Slurm User Community List <slurm-users(a)lists.schedmd.com> >> *Subject: *Re: [slurm-users] Need help with running multiple >> instances/executions of a batch script in parallel (with NVIDIA HGX A100 >> GPU as a Gres) >> >> Hi Noam and Matthias, >> >> >> >> Thanks both for your answers. >> >> >> >> I changed the ?#SBATCH --gres=gpu:4? directive (in the batch script) with >> ?#SBATCH --gres=gpu:1? as you suggested, but it didn?t make a difference, >> as running this batch script 3 times will result in the first job to be in >> a running state, while the second and third jobs will still be in a pending >> state ? >> >> >> >> [slurmtest@c-a100-master test-batch-scripts]$ cat gpu-job.sh >> >> #!/bin/bash >> >> #SBATCH --job-name=gpu-job >> >> #SBATCH --partition=gpu >> >> #SBATCH --nodes=1 >> >> #SBATCH --gpus-per-node=4 >> >> #SBATCH --gres=gpu:1 # <<<< Changed from ?4? >> to ?1? >> >> #SBATCH --tasks-per-node=1 >> >> #SBATCH --output=gpu_job_output.%j >> >> #SBATCH --error=gpu_job_error.%j >> >> >> >> hostname >> >> date >> >> sleep 40 >> >> pwd >> >> >> >> [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh >> >> Submitted batch job *217* >> >> [slurmtest@c-a100-master test-batch-scripts]$ squeue >> >> JOBID PARTITION NAME USER ST TIME NODES >> NODELIST(REASON) >> >> 217 gpu gpu-job slurmtes R 0:02 1 >> c-a100-cn01 >> >> [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh >> >> Submitted batch job *218* >> >> [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh >> >> Submitted batch job *219* >> >> [slurmtest@c-a100-master test-batch-scripts]$ squeue >> >> JOBID PARTITION NAME USER ST TIME NODES >> NODELIST(REASON) >> >> 219 gpu gpu-job slurmtes *PD* 0:00 1 >> (Priority) >> >> 218 gpu gpu-job slurmtes *PD* 0:00 1 >> (Resources) >> >> 217 gpu gpu-job slurmtes *R* 0:07 1 >> c-a100-cn01 >> >> >> >> Basically I?m seeking for some help/hints on how to tell Slurm, from the >> batch script for example: ?I want only 1 or 2 GPUs to be used/consumed by >> the job?, and then I run the batch script/job a couple of times with sbatch >> command, and confirm that we can indeed have multiple jobs using a GPU and >> running in parallel, at the same time. >> >> >> >> Makes sense ? >> >> >> >> >> >> Best regards, >> >> >> >> *Hafedh * >> >> >> >> *From:* slurm-users <slurm-users-bounces(a)lists.schedmd.com> *On Behalf >> Of *Bernstein, Noam CIV USN NRL (6393) Washington DC (USA) >> *Sent:* jeudi 18 janvier 2024 2:30 PM >> *To:* Slurm User Community List <slurm-users(a)lists.schedmd.com> >> *Subject:* Re: [slurm-users] Need help with running multiple >> instances/executions of a batch script in parallel (with NVIDIA HGX A100 >> GPU as a Gres) >> >> >> >> On Jan 18, 2024, at 7:31 AM, Matthias Loose <m.loose(a)mindcode.de> wrote: >> >> >> >> Hi Hafedh, >> >> Im no expert in the GPU side of SLURM, but looking at you current >> configuration to me its working as intended at the moment. You have defined >> 4 GPUs and start multiple jobs each consuming 4 GPUs each. So the jobs wait >> for the ressource the be free again. >> >> I think what you need to look into is the MPS plugin, which seems to do >> what you are trying to achieve: >> https://slurm.schedmd.com/gres.html#MPS_Management >> >> >> >> I agree with the first paragraph. How many GPUs are you expecting each >> job to use? I'd have assumed, based on the original text, that each job is >> supposed to use 1 GPU, and the 4 jobs were supposed to be running >> side-by-side on the one node you have (with 4 GPUs). If so, you need to >> tell each job to request only 1 GPU, and currently each one is requesting 4. >> >> >> >> If your jobs are actually supposed to be using 4 GPUs each, I still don't >> see any advantage to MPS (at least in what is my usual GPU usage pattern): >> all the jobs will take longer to finish, because they are sharing the fixed >> resource. If they take turns, at least the first ones finish as fast as >> they can, and the last one will finish no later than it would have if they >> were all time-sharing the GPUs. I guess NVIDIA had something in mind when >> they developed MPS, so I guess our pattern may not be typical (or at least >> not universal), and in that case the MPS plugin may well be what you need. >> > > > -- > Mohammed >

1 0

slurmctld/slurmdbd (code=exited, status=217/USER)
by Miriam Olmi 19 Jan '24

19 Jan '24

Hi all, I am having some issue with the new version of slurm 23.11.0-1. I had already installed and configured slurm 23.02.3-1 on my cluster and all the services were active and running properly. After I install with the same procedure the new version of slurm I have that the slurmctld and slurmdbd daemons fail to start all with the same error: (code=exited, status=217/USER) And investigating the problem with the command journalctl -xe I find: slurmctld.service: Failed to determine user credentials: No such process slurmctld.service: Failed at step USER spawning /usr/sbin/slurmctld: No such process I had a look at the slurmctld.service file for both the slurm versions and I found the following differences in the [Service] section. >From the slurmctld.service file of slurm 23.02.3-1: [Service] Type=simple EnvironmentFile=-/etc/sysconfig/slurmctld EnvironmentFile=-/etc/default/slurmctld ExecStart=/usr/sbin/slurmctld -D -s $SLURMCTLD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID LimitNOFILE=65536 TasksMax=infinity >From the slurmctld.service file of slurm 23.11.0-1: [Service] Type=notify EnvironmentFile=-/etc/sysconfig/slurmctld EnvironmentFile=-/etc/default/slurmctld User=slurm Group=slurm ExecStart=/usr/sbin/slurmctld --systemd $SLURMCTLD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID LimitNOFILE=65536 TasksMax=infinity I think the presence of the new lines regarding the slurm user might be the problem but I am not sure and I have no idea how to solve it. Can anyone halp me? Thanks in advance, Miriam

2 1

Jobs exiting together
by Alexander Silva 19 Jan '24

19 Jan '24

Recently, i have built an hpc cluster with slurm as workload. The test jobs with quatum chemistry codes have worked fine. However, production jobs with lammps have shown an unexpected behavior when the first job completed, normally or not, cause the termination of the others in the same compute node. Initially, I thought that was due to mpi malfunction, but this behavior is algo observed for serial lammps code. The lammps group said to me that behavior could be generated by slurm. My question to you is about what parameter in slurm.conf could be responsible for the termination of the other jobs. I am using an epilogue script that work normally in another cluster. Thanks.

1 0

Running slurm job on requested nvidia mig device
by Dražen Jalšovec 19 Jan '24

19 Jan '24

Hi, We are testing the MIG deployment on our new slurm compute node with 4 x H100 GPUs. It looks like everything is configured correctly but we have a problem accessing mig devices. When I submit jobs requesting a mig gpu device #SBATCH --gres=gpu:H100_1g.10gb:1, the jobs get submitted to the node, but only 4 jobs get executed and all other jobs fail. I was able to "solve it" by adding code to submit to the mig device UUID. Am I missing something in the slurm/gres/cgroups configuration that would automatically assign correct CUDA_VISIBLE_DEVICES MIG UUID to the job? If I submit 10 jobs requesting --gres=gpu:H100_1g.10gb:1 8 jobs start running: #1 - CUDA_VISIBLE_DEVICES=0 - jobs 1-4 run on one of the mig devices configured on GPU-device ... #8 - CUDA_VISIBLE_DEVICES=7 - jobs 5-8 fail with the message that there are no available CUDA_VISIBLE_DEVICES ... #10 - CUDA_VISIBLE_DEVICES=? - after waiting for free mig resources jobs 9-10 runs or fails depending if GPU-device is free (migs are free) $ nvidia-smi --query-gpu=uuid --id=0 --format=csv,noheader GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3 $ nvidia-smi --query-gpu=uuid --id=7 --format=csv,noheader No devices were found The slurm assigns CUDA_VISIBLE_DEVICES numbers based on the requested resources: Request: gpu:H100_1g.10gb:1 - CUDA_VISIBLE_DEVICES=0 gpu:H100_1g.10gb:2 - CUDA_VISIBLE_DEVICES=0,1 ... Any job with H100_1g.10gb request is going to get device number between 0 and 7 gpu:H100_1g.10gb:8 - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 gpu:H100_2g.20gb:1 - CUDA_VISIBLE_DEVICES=8 Any job with H100_2g.20gb request is going to get device number between 8 and 10 ... gpu:H100_2g.10gb:3 - CUDA_VISIBLE_DEVICES=8,9,10 gpu:H100_3g.40gb:1 - CUDA_VISIBLE_DEVICES=11 gpu:H100_3g.40gb:2 - CUDA_VISIBLE_DEVICES=12 gpu:H100_7g.80gb:1 - CUDA_VISIBLE_DEVICES=13 I was able to "solve it" by adding this code to my slurm script: # Get the CUDA index using srun and awk CUDA_INDEX=$(srun env | grep CUDA_VISIBLE_DEVICES | awk -F '=' '{print $2}') ## GPU0 1g.10gb ## MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a ## MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664 ## MIG-d3ab0675-d318-5e53-b487-b50695cf2e00 ## MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0 ## MIG-85dd76da-c994-5830-adf0-467c66ae1b95 ## MIG-29a6d43b-882e-5b79-868a-15bb2c770b82 ## MIG-558debd2-dc13-5406-9256-73ef4f279737 ## GPU1 2g.20gb ## MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f ## MIG-0a804c0c-aa27-5993-97cb-cedb854735ce ## MIG-8af9af6d-8720-5763-81e2-83afc43eb42b ## GPU1 1g.10gb ## MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a ## GPU2 3g.40gb ## MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b ## MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc ## GPU3 7g.80gb ## MIG-ec043869-9176-577e-bac0-46c8411e4e37 # Define the list of UUIDs UUIDS=( "MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a" "MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664" "MIG-d3ab0675-d318-5e53-b487-b50695cf2e00" "MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0" "MIG-85dd76da-c994-5830-adf0-467c66ae1b95" "MIG-29a6d43b-882e-5b79-868a-15bb2c770b82" "MIG-558debd2-dc13-5406-9256-73ef4f279737" "MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a" "MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f" "MIG-0a804c0c-aa27-5993-97cb-cedb854735ce" "MIG-8af9af6d-8720-5763-81e2-83afc43eb42b" "MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b" "MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc" "MIG-ec043869-9176-577e-bac0-46c8411e4e37" ) # Assign the UUID based on the CUDA index SELECTED_UUID=${UUIDS[$CUDA_INDEX]} # Print the selected UUID echo $SELECTED_UUID export CUDA_VISIBLE_DEVICES=$SELECTED_UUID echo "CUDA_VISIBLE_DEVICES set to: "$SELECTED_UUID echo "Test: "$CUDA_VISIBLE_DEVICES CUDA_VISIBLE_DEVICES=$SELECTED_UUID python3.8 gpu_script.py Result: Script start MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a CUDA_VISIBLE_DEVICES set to: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a Matrix calculation on CUDA device completed successfully. Script end +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 7 0 28686 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 8 0 28744 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 9 0 28300 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 10 0 28512 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 11 0 28506 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 12 0 28516 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 13 0 28511 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 1 9 0 28552 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | +---------------------------------------------------------------------------------------+ This is the current setup: RockyLinux 9.2 slurm-22.05.9-1.el9.x86_64 CUDA Version: 12.1 Driver Version: 530.30.02 # nvidia-smi -L GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3) MIG 1g.10gb Device 0: (UUID: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a) MIG 1g.10gb Device 1: (UUID: MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664) MIG 1g.10gb Device 2: (UUID: MIG-d3ab0675-d318-5e53-b487-b50695cf2e00) MIG 1g.10gb Device 3: (UUID: MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0) MIG 1g.10gb Device 4: (UUID: MIG-85dd76da-c994-5830-adf0-467c66ae1b95) MIG 1g.10gb Device 5: (UUID: MIG-29a6d43b-882e-5b79-868a-15bb2c770b82) MIG 1g.10gb Device 6: (UUID: MIG-558debd2-dc13-5406-9256-73ef4f279737) GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1439a39c-948b-0657-98e9-aff8595a8729) MIG 2g.20gb Device 0: (UUID: MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f) MIG 2g.20gb Device 1: (UUID: MIG-0a804c0c-aa27-5993-97cb-cedb854735ce) MIG 2g.20gb Device 2: (UUID: MIG-8af9af6d-8720-5763-81e2-83afc43eb42b) MIG 1g.10gb Device 3: (UUID: MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a) GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-3b182fed-bfdf-bc11-d1bd-c41fd1468d2a) MIG 3g.40gb Device 0: (UUID: MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b) MIG 3g.40gb Device 1: (UUID: MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc) GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-7eba9d78-e908-3db8-2633-a269aeec395e) MIG 7g.80gb Device 0: (UUID: MIG-ec043869-9176-577e-bac0-46c8411e4e37) $ scontrol show node node16 NodeName=node16 Arch=x86_64 CoresPerSocket=1 CPUAlloc=1 CPUEfctv=112 CPUTot=112 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 NodeAddr=node16 NodeHostName=node16 Version=22.05.9 OS=Linux 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023 RealMemory=1030000 AllocMem=0 FreeMem=1011801 Sockets=112 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpu_H100 BootTime=2024-01-18T13:52:27 SlurmdStartTime=2024-01-19T11:25:44 LastBusyTime=2024-01-19T10:59:13 CfgTRES=cpu=112,mem=1030000M,billing=112,gres/gpu=14 AllocTRES=cpu=1 CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s $ scontrol show partition gpu_H100 PartitionName=gpu_H100 AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=node16 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=112 TotalNodes=1 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=112,mem=1030000M,node=1,billing=112,gres/gpu=14 slurm.conf: AccountingStorageTRES=gres/gpu,gres/gpu:A100,gres/gpu:H100,gres/gpu:H100_1g.10gb,gres/gpu:H100_2g.20gb,gres/gpu:H100_3g.40gb,gres/gpu:H100_7g.80gb GresTypes=gpu ... NodeName=node16 CPUs=112 RealMemory=1030000 Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 PartitionName=gpu_H100 Nodes=node16 Default=NO MaxTime=INFINITE State=UP gres.conf #AutoDetect=nvml # GPU 0 MIG 0 /proc/driver/nvidia/capabilities/gpu0/mig/gi7/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap66,/dev/nvidia-caps/nvidia-cap67 CPUs=0-55 # GPU 0 MIG 1 /proc/driver/nvidia/capabilities/gpu0/mig/gi8/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap75,/dev/nvidia-caps/nvidia-cap76 CPUs=0-55 # GPU 0 MIG 2 /proc/driver/nvidia/capabilities/gpu0/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap84,/dev/nvidia-caps/nvidia-cap85 CPUs=0-55 # GPU 0 MIG 3 /proc/driver/nvidia/capabilities/gpu0/mig/gi10/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap93,/dev/nvidia-caps/nvidia-cap94 CPUs=0-55 # GPU 0 MIG 4 /proc/driver/nvidia/capabilities/gpu0/mig/gi11/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap102,/dev/nvidia-caps/nvidia-cap103 CPUs=0-55 # GPU 0 MIG 5 /proc/driver/nvidia/capabilities/gpu0/mig/gi12/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap111,/dev/nvidia-caps/nvidia-cap112 CPUs=0-55 # GPU 0 MIG 6 /proc/driver/nvidia/capabilities/gpu0/mig/gi13/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap120,/dev/nvidia-caps/nvidia-cap121 CPUs=0-55 # GPU 1 MIG 0 /proc/driver/nvidia/capabilities/gpu1/mig/gi3/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap165,/dev/nvidia-caps/nvidia-cap166 CPUs=0-55 # GPU 1 MIG 1 /proc/driver/nvidia/capabilities/gpu1/mig/gi5/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap183,/dev/nvidia-caps/nvidia-cap184 CPUs=0-55 # GPU 1 MIG 2 /proc/driver/nvidia/capabilities/gpu1/mig/gi6/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap192,/dev/nvidia-caps/nvidia-cap193 CPUs=0-55 # GPU 1 MIG 3 /proc/driver/nvidia/capabilities/gpu1/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap219,/dev/nvidia-caps/nvidia-cap220 CPUs=0-55 # GPU 2 MIG 0 /proc/driver/nvidia/capabilities/gpu2/mig/gi1/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap282,/dev/nvidia-caps/nvidia-cap283 CPUs=0-55 # GPU 2 MIG 1 /proc/driver/nvidia/capabilities/gpu2/mig/gi2/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap291,/dev/nvidia-caps/nvidia-cap292 CPUs=0-55 # GPU 3 MIG 0 /proc/driver/nvidia/capabilities/gpu3/mig/gi0/access NodeName=node16 Name=gpu Type=H100_7g.80gb MultipleFiles=/dev/nvidia3,/dev/nvidia-caps/nvidia-cap408,/dev/nvidia-caps/nvidia-cap409 CPUs=0-55 [2024-01-19T14:31:13.701] debug: gres/gpu: init: loaded [2024-01-19T14:31:13.701] debug: gpu/generic: init: init: GPU Generic plugin loaded [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.702] Gres Name=gpu Type=H100_7g.80gb Count=1 Best, Drazen Jalsovec

1 0

Potential Side Effects of larger MessageTimeout value
by Herc Silverstein 18 Jan '24

18 Jan '24

Hi, What are potential bad side effects of using a large/larger MessageTimeout? And is there a value at which this setting is too large (long)? Thanks, Herc

1 0

error
by Felix 18 Jan '24

18 Jan '24

Hello I started a new AMD node, and the error is as follows: "CPU frequency setting not configured for this node" extended looks like this: [2024-01-18T18:28:06.682] CPU frequency setting not configured for this node [2024-01-18T18:28:06.691] slurmd started on Thu, 18 Jan 2024 18:28:06 +0200 [2024-01-18T18:28:06.691] CPUs=128 Boards=1 Sockets=1 Cores=64 Threads=2 Memory=256786 TmpDisk=875797 Uptime=4569 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) In the configuration file I have the following: NodeName=awn-1[04] NodeAddr=192.168.4.[111] CPUs=128 RealMemory=256000 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 Feature=HyperThread Could you please help me? Thank you Felix -- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

2 1

Re: [slurm-users] [BULK] slurm-users Digest, Vol 75, Issue 26
by Jason Macklin 18 Jan '24

18 Jan '24

If you run "scontrol show jobid <jobid>" of your pending job with the "(Resources)" tag you may see more about what is unavailable to your job. Slurm default configs can cause an entire compute node of resources to be "allocated" to a running job regardless of whether it needs all of them or not so you may need to alter one or both of the following settings to allow more than one job to run on a single node at once. You'll find these in your slurm.conf. Don't forget to "scontrol reconf" and even potentially restart both "slurmctld" & "slurmd" on your nodes if you do end up making changes. SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory I hope this helps. Kind regards, Jason ---- Jason Macklin Manager Cyberinfrastructure, Research Cyberinfrastructure 860.837.2142 t | 860.202.7779 m jason.macklin(a)jax.org The Jackson Laboratory Maine | Connecticut | California | Shanghai www.jax.org<http://www.jax.org> The Jackson Laboratory: Leading the search for tomorrow's cures ________________________________ From: slurm-users <slurm-users-bounces(a)lists.schedmd.com> on behalf of slurm-users-request(a)lists.schedmd.com <slurm-users-request(a)lists.schedmd.com> Sent: Thursday, January 18, 2024 9:46 AM To: slurm-users(a)lists.schedmd.com <slurm-users(a)lists.schedmd.com> Subject: [BULK] slurm-users Digest, Vol 75, Issue 26 Send slurm-users mailing list submissions to slurm-users(a)lists.schedmd.com To subscribe or unsubscribe via the World Wide Web, visit https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users or, via email, send a message with subject or body 'help' to slurm-users-request(a)lists.schedmd.com You can reach the person managing the list at slurm-users-owner(a)lists.schedmd.com When replying, please edit your Subject line so it is more specific than "Re: Contents of slurm-users digest..." Today's Topics: 1. Re: Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) (Baer, Troy) ---------------------------------------------------------------------- Message: 1 Date: Thu, 18 Jan 2024 14:46:48 +0000 From: "Baer, Troy" <troy(a)osc.edu> To: Slurm User Community List <slurm-users(a)lists.schedmd.com> Subject: Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) Message-ID: <CH0PR01MB6924127AF471DED69151805BCF712(a)CH0PR01MB6924.prod.exchangelabs.com> Content-Type: text/plain; charset="utf-8" Hi Hafedh, Your job script has the sbatch directive ??gpus-per-node=4? set. I suspect that if you look at what?s allocated to the running job by doing ?scontrol show job <jobid>? and looking at the TRES field, it?s been allocated 4 GPUs instead of one. Regards, --Troy From: slurm-users <slurm-users-bounces(a)lists.schedmd.com> On Behalf Of Kherfani, Hafedh (Professional Services, TC) Sent: Thursday, January 18, 2024 9:38 AM To: Slurm User Community List <slurm-users(a)lists.schedmd.com> Subject: Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) Hi Noam and Matthias, Thanks both for your answers. I changed the ?#SBATCH --gres=gpu:?4? directive (in the batch script) with ?#SBATCH --gres=gpu:?1? as you suggested, but it didn?t make a difference, as running Hi Noam and Matthias, Thanks both for your answers. I changed the ?#SBATCH --gres=gpu:4? directive (in the batch script) with ?#SBATCH --gres=gpu:1? as you suggested, but it didn?t make a difference, as running this batch script 3 times will result in the first job to be in a running state, while the second and third jobs will still be in a pending state ? [slurmtest@c-a100-master test-batch-scripts]$ cat gpu-job.sh #!/bin/bash #SBATCH --job-name=gpu-job #SBATCH --partition=gpu #SBATCH --nodes=1 #SBATCH --gpus-per-node=4 #SBATCH --gres=gpu:1 # <<<< Changed from ?4? to ?1? #SBATCH --tasks-per-node=1 #SBATCH --output=gpu_job_output.%j #SBATCH --error=gpu_job_error.%j hostname date sleep 40 pwd [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 217 [slurmtest@c-a100-master test-batch-scripts]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 217 gpu gpu-job slurmtes R 0:02 1 c-a100-cn01 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 218 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 219 [slurmtest@c-a100-master test-batch-scripts]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 219 gpu gpu-job slurmtes PD 0:00 1 (Priority) 218 gpu gpu-job slurmtes PD 0:00 1 (Resources) 217 gpu gpu-job slurmtes R 0:07 1 c-a100-cn01 Basically I?m seeking for some help/hints on how to tell Slurm, from the batch script for example: ?I want only 1 or 2 GPUs to be used/consumed by the job?, and then I run the batch script/job a couple of times with sbatch command, and confirm that we can indeed have multiple jobs using a GPU and running in parallel, at the same time. Makes sense ? Best regards, Hafedh From: slurm-users <slurm-users-bounces(a)lists.schedmd.com<mailto:slurm-users-bounces@lists.schedmd.com>> On Behalf Of Bernstein, Noam CIV USN NRL (6393) Washington DC (USA) Sent: jeudi 18 janvier 2024 2:30 PM To: Slurm User Community List <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Subject: Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) On Jan 18, 2024, at 7:31 AM, Matthias Loose <m.loose(a)mindcode.de<mailto:m.loose@mindcode.de>> wrote: Hi Hafedh, Im no expert in the GPU side of SLURM, but looking at you current configuration to me its working as intended at the moment. You have defined 4 GPUs and start multiple jobs each consuming 4 GPUs each. So the jobs wait for the ressource the be free again. I think what you need to look into is the MPS plugin, which seems to do what you are trying to achieve: https://slurm.schedmd.com/gres.html#MPS_Management<https://urldefense.com/v3/__https:/slurm.schedmd.com/gres.html*MPS_Manageme…> I agree with the first paragraph. How many GPUs are you expecting each job to use? I'd have assumed, based on the original text, that each job is supposed to use 1 GPU, and the 4 jobs were supposed to be running side-by-side on the one node you have (with 4 GPUs). If so, you need to tell each job to request only 1 GPU, and currently each one is requesting 4. If your jobs are actually supposed to be using 4 GPUs each, I still don't see any advantage to MPS (at least in what is my usual GPU usage pattern): all the jobs will take longer to finish, because they are sharing the fixed resource. If they take turns, at least the first ones finish as fast as they can, and the last one will finish no later than it would have if they were all time-sharing the GPUs. I guess NVIDIA had something in mind when they developed MPS, so I guess our pattern may not be typical (or at least not universal), and in that case the MPS plugin may well be what you need.

1 0

slurm.conf
by LEROY Christine 208562 18 Jan '24

18 Jan '24

Hello all, Is there an env variable in SLURM to tell where the slurm.conf is? We would like to have on the same client node, 2 type of possible submissions to address 2 different cluster. Thanks in advance, Christine

4 3

What happens if GPU GRES exceeding number of GPUs per node
by Purwanto, Wirawan 18 Jan '24

18 Jan '24

Hi, In my HPC center, I found a SLURM job that was submitted with --gres=gpu:6 whereas the cluster has only four GPUs per node each. It is a parallel job. Here are some relevant field printout: AllocCPUS 30 AllocGRES gpu:6 AllocTRES billing=30,cpu=30,gres/gpu=6,node=3 CPUTime 1-01:23:00 CPUTimeRAW 91380 Elapsed 00:50:46 JobID 20073 JobIDRaw 20073 JobName simple_cuda NCPUS 30 NGPUS 6.0 What happened in this case? This job was asking for 3 nodes, 10 core per node. When the user specified “--gres=gpu:6”, does this mean six GPUs for the entire job, or six GPUs per node? Per the description in https://slurm.schedmd.com/gres.html#Running_Jobs, it says: gres is “Generic resources required per node”. So it is illogical to request six GPUs per node. So what happened? Did SLURM quietly ignore the request and grant just one, or grant the max number (4)? Because apparently the job ran without error. Wirawan Purwanto Computational Scientist, HPC Group Information Technology Services Old Dominion University Norfolk, VA 23529

2 1

Strict GrpTRESMins limit
by Kamil Wilczek 17 Jan '24

17 Jan '24

Dear All, I tried to implement a strict limit on the GrpTRESMins for each user. The effect I'm trying to achieve is that after the limit of GPU minutes is reached, no new jobs can be run. No decay, no automatic resource replenishment. After the limit on GPU minutes is reached, each user should ask for more minutes. But despite exceeding the limits users *can* run new jobs. * When I'm adding a user to the cluster I set: sacctmgr --immediate add user name=... ... QOS=2gpu2d GrpTRESMins=gres/gpu=20000 * In the "slurm.conf" ("safe" means limits and associations are automatically set). Storage is MariaDB with SlurmDBD: GresTypes=gpu AccountingStorageTRES=gres/gpu AccountingStorageEnforce=qos,safe # This disables GPU minutes replenishing. PriorityDecayHalfLife=0 PriorityUsageResetPeriod=NONE But when I look at a user's account info and usage, you can see that the limits are not enforced. Account User Partition QOS GrpTRESMins ---------- ---------------- ------------ ------------ -------------------- redacted redacted a6000 2gpu2d gres/gpu=10000 -------------------------------------------------------------------------------- Top 1 Users 2024-01-05T00:00:00 - 2024-01-17T19:59:59 (1108800 secs) Usage reported in TRES Minutes -------------------------------------------------------------------------------- Login Used TRES Name ------------ -------- ---------------- redacted 184311 gres/gpu redacted 1558558 cpu Could someone explain, where could the problem be? Am I missing something? Apparently yes :) Kind regards -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

1 0

Suspend/Resume request limit
by 김종록 17 Jan '24

17 Jan '24

2 1

preemptable queue
by Davide DelVento 12 Jan '24

12 Jan '24

I would like to add a preemptable queue to our cluster. Actually I already have. We simply want jobs submitted to that queue be preempted if there are no resources available for jobs in other (high priority) queues. Conceptually very simple, no conditionals, no choices, just what I wrote. However it does not work as desired. This is the relevant part: grep -i Preemp /opt/slurm/slurm.conf #PreemptType = preempt/partition_prio PartitionName=regular DefMemPerCPU=4580 Default=True Nodes=node[01-12] State=UP PreemptMode=off PriorityTier=200 PartitionName=All DefMemPerCPU=4580 Nodes=node[01-36] State=UP PreemptMode=off PriorityTier=500 PartitionName=lowpriority DefMemPerCPU=4580 Nodes=node[01-36] State=UP PreemptMode=cancel PriorityTier=100 That PreemptType setting (now commented) fully breaks slurm, everything refuses to run with errors like $ squeue squeue: error: PreemptType and PreemptMode values incompatible squeue: fatal: Unable to process configuration file If I understand correctly the documentation at https://slurm.schedmd.com/preempt.html that is because preemption cannot cancel jobs based on partition priority, which (if true) is really unfortunate. I understand that allowing cross-partition time-slicing could be tricky and so I understand why that isn't allowed, but cancelling? Anyway, I have to questions: 1) is that correct and so should I avoid using either partition priority or cancelling? 2) is there an easy way to trick slurm into requeing and then have those jobs cancelled instead? 3) I guess the cleanest option would be to implement QoS, but I've never done it and we don't really need it for anything else other than this. The documentation looks complicated, but is it? The great Ole's website is unavailable at the moment... Thanks!!

2 4

Re: [slurm-users] sacct --name --status filtering
by Drucker, Daniel 11 Jan '24

11 Jan '24

Yes, that makes sense. Thank you! The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

sacct --name --status filtering
by Drucker, Daniel 11 Jan '24

11 Jan '24

What am I misunderstanding about how sacct filtering works here? I would have expected the second command to show the exact same results as the first. [root@mickey ddrucker]# sacct --starttime $(date -d "7 days ago" +"%Y-%m-%d") -X --format JobID,JobName,State,Elapsed --name zsh JobID JobName State Elapsed ------------ ---------- ---------- ---------- 257713 zsh COMPLETED 00:01:02 257714 zsh COMPLETED 00:04:01 257715 zsh COMPLETED 00:03:01 257716 zsh COMPLETED 00:03:01 [root@mickey ddrucker]# sacct --starttime $(date -d "7 days ago" +"%Y-%m-%d") -X --format JobID,JobName,State,Elapsed --name zsh --state COMPLETED JobID JobName State Elapsed ------------ ---------- ---------- ---------- [root@mickey ddrucker]# sinfo --version slurm 21.08.8-2 -- Daniel M. Drucker, Ph.D. Director of IT, MGB Imaging at Belmont McLean Hospital, a Harvard Medical School Affiliate The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

3 2

Re: [slurm-users] sacct --name --status filtering
by Drucker, Daniel 11 Jan '24

11 Jan '24

> All I can say is that this has to do with --starttime and that you have to read the manual really carefully about how they interact, including when you have --endtime set. It’s a bit fiddly and annoying, IMO, and I can never quite remember how it works. Oh, I think I understand. --starttime actually behaves differently when --state is present: If states are given with the '-s' option then only jobs in this state at this time will be returned. So is there a way to do what I want? I want to see jobs which - started later than 7 days ago - whose state is COMPLETED Surely that's possible without resorting to grep? Daniel The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

Cleanup of old clusters in database
by Jeffrey R. Lang 11 Jan '24

11 Jan '24

We have shuttered two clusters and need to remove them from the database. To do this, do we remove the table spaces associated with the cluster names from the Slurm database? Thanks, Jeff

1 0

Re: [slurm-users] DBD_SEND_MULT_MSG - invalid uid error (Timony, Mick)
by Craig Stark 10 Jan '24

10 Jan '24

All good ideas Mick – - I've restarted slurmd on all nodes – no effect - Ran this on all nodes: #!/bin/bash uname -n id slurm id 59999 scontrol show config | grep SlurmUser All show slurm being that 59999 user. - The firewalld already has the internal network interface being used set to the trusted zone I get a bit more info out of setting the slurmctld to debug level, but I'm not sure what to make of it TBH. I'm not sure what "_handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474)" is trying to tell me. Jan 10 10:48:26 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:31 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:35 kirby slurmctld[461138]: slurmctld: debug: accounting_storage/slurmdbd: _handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474): DBD_SEND_MULT_MSG message from invalid uid Jan 10 10:48:36 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:41 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:46 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:46 kirby slurmctld[461138]: slurmctld: debug: sched/backfill: _attempt_backfill: beginning Jan 10 10:48:46 kirby slurmctld[461138]: slurmctld: debug: sched/backfill: _attempt_backfill: no jobs to backfill Jan 10 10:48:51 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:53 kirby slurmctld[461138]: slurmctld: debug: accounting_storage/slurmdbd: _handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474): DBD_SEND_MULT_MSG message from invalid uid Jan 10 10:48:56 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:01 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:06 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:11 kirby slurmctld[461138]: slurmctld: debug: accounting_storage/slurmdbd: _handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474): DBD_SEND_MULT_MSG message from invalid uid Jan 10 10:49:11 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:16 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:17 kirby slurmctld[461138]: slurmctld: debug: sched: Running job scheduler for full queue. Jan 10 10:49:21 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid A bit more info / another possible clue. While "sacctmgr list Account" or "sacctmgr list user" shows expected account groups and users, "sreport user top start=12/1/23" and "sreport cluster utilization start=12/1/23" both report empty tables. Craig Stark, Ph.D. Professor, Department of Neurobiology and Behavior Director, Facility for Imaging and Brain Research (FIBRE) Director, Campus Center for Neuroimaging (CCNI) School of Biological Sciences, University of California, Irvine cestark(a)uci.edu<mailto:cestark@uci.edu>

1 0

Beginner admin question: Prioritization within a partition based on time limit
by Kenneth Chiu 09 Jan '24

09 Jan '24

I'm just learning about slurm. I understand that different different partitions can be prioritized separately, and can have different max time limits. I was wondering whether or not there was a way to have a finer-grained prioritization based on the time limit specified by a job, within a single partition. Or perhaps this is already happening by default? Would the backfill scheduler be best for this?

2 1

Re: [slurm-users] DBD_SEND_MULT_MSG - invalid uid error
by Craig Stark 08 Jan '24

08 Jan '24

This ticket with SchedMD implies it's a munged issue: https://urldefense.com/v3/__https://bugs.schedmd.com/show_bug.cgi?id=1293__… Is the munge daemon running on all systems? If it is, are all servers running a network time daemon such chronyd or ntpd and the time is in sync on all hosts? Thanks Mick, munge is seemingly running on all systems (systemctl status munge). I do get a warning about the munge file changing on disk, but I'm pretty sure that's from warewulf sync'ing files every minute. A sha256sum on the munge.key file on the compute nodes and host node says they're the same, so I think I can put that aside. The management node runs chrony and the compute nodes sync to the management node. [root@kirby uber]# chronyc tracking Reference ID : 4A06A849 (t2.time.gq1.yahoo.com) Stratum : 3 Ref time (UTC) : Mon Jan 08 22:26:44 2024 System time : 0.000032525 seconds slow of NTP time Last offset : -0.000021390 seconds RMS offset : 0.000055729 seconds Frequency : 38.797 ppm slow Residual freq : +0.001 ppm Skew : 0.018 ppm Root delay : 0.033342984 seconds Root dispersion : 0.000524800 seconds Update interval : 256.8 seconds Leap status : Normal vs [root@sonic01 ~]# chronyc tracking Reference ID : C0A80102 (warewulf) Stratum : 4 Ref time (UTC) : Mon Jan 08 22:31:02 2024 System time : 0.000000120 seconds slow of NTP time Last offset : -0.000000092 seconds RMS offset : 0.000014737 seconds Frequency : 47.495 ppm slow Residual freq : +0.000 ppm Skew : 0.066 ppm Root delay : 0.033458963 seconds Root dispersion : 0.000283949 seconds Update interval : 64.2 seconds Leap status : Normal So, the compute node is talking to the host and the host is talking to generic NTP sources. "date" shows the same time on the compute nodes

1 0

Re: [slurm-users] DBD_SEND_MULT_MSG - invalid uid error
by Craig Stark 08 Jan '24

08 Jan '24

3rd time trying to get this to come through to the list - hopefully this time works. I've been running SLURM for several years now, but in setting it up on a new cluster, I'm hitting a recurring issue. I'm using a MariaDB and configured it just as I had in my several-year-ago setup and in the docs. There's a "slurm" user (59999) on the OS (Rocky 9), that's on all the nodes, and I've added the slurm@localhost as instructed (grant all on slurm_acct_db.* TO 'slurm'@'localhost' identified by 'PASSWORD'). But, I keep getting things like this: ``` Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: DBD_SEND_MULT_MSG message from invalid uid 59999 Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: Processing last message from connection 7(192.168.1.2) uid(59999) Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: CONN:7 DBD_REGISTER_CTLD message from invalid uid 59999 Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: CONN:7 Security violation, DBD_REGISTER_CTLD Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: Processing last message from connection 7(192.168.1.2) uid(59999) ``` I'm a total SQL noob, but can at least verify that the user is in there: MariaDB [(none)]> SELECT User, Host, Password FROM mysql.user; +-------------+-----------+-------------------------------------------+ | User | Host | Password | +-------------+-----------+-------------------------------------------+ | mariadb.sys | localhost | | | root | localhost | invalid | | mysql | localhost | invalid | | slurm | localhost | *D6665ECF4F3CB12BCA836117F7727B6D0B78D644 | +-------------+-----------+-------------------------------------------+ 4 rows in set (0.002 sec) Any thoughts as to where I might look to fix this? Craig

2 1

Multifactor fair-share with single account
by Kamil Wilczek 08 Jan '24

08 Jan '24

Dear All, I have a question regarding the fair-share factor of the multifactor priority algorithm. My current understanding is that the fair-share makes sure that different *accounts* have a fair share of the computational power. But what if my organisation structure is flat and I have only one account where all my user reside. Is fair-share algorithm working in this situation -- does it take into account users (associations) from this single account, and tries to assing a fair-factor to each user? Or each user from this account have the same fair-factor at each iteration? And what if I have, say 3 accounts, but I do not wan't to calculate fair-share between accounts, but between all associations from all 3 accounts? In other words, is there a fair-share factor for users/associations instead of accounts? Kind regards -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

4 6

Tool for profiling resource usage by slurm jobs
by Nicolas Granger 08 Jan '24

08 Jan '24

Hi all, Happy new year everyone! I've been looking for a simple tool that reports how much resources are actually consumed by a job to help my colleagues and I adjust job requirements. I could not find such a tool, or the ones mentioned on this ML were not easy to install and use, so I have written a new one: https://github.com/CEA-LIST/sprofile It's a simple python script which parses cgroup and nvml data from the nvidia driver. It reports duration, cpu load, peak RAM, GPU load and peak GPU memory like so: |-- sprofile report (node03) -- Time: 0:00:03 / 1:00:00 CPU load: 2.0 / 4.0 RAM peak mem: 7G / 8G GPU load: 0.2 / 2.0 GPU peak mem: 7G / 40G| The requirements are to use the slurm cgroup plugin and to enable accounting on the GPU (nvidia-smi --accounting-mode=1). I hope you find this useful and let me know I you find bugs or want to contribute. Regards, Nicolas Granger

1 0

A fairshare policy that spans multiple clusters
by David Baker 05 Jan '24

05 Jan '24

Hello, We are soon to install new Slurm cluster at our site. That means that we will have a total of three clusters running Slurm. Only two, that is the new clusters, will share a common file system. The original cluster has its own file system is independent of the new arrivals. If possible, we would like to try to prevent users from making significant user of all the clusters and get a 'triple whammy'. In other words, is there any way to share the fairshare information between the clusters so that a user's use of one of the clusters impacts their usage on the other clusters – if that makes sense. Does anyone have any thoughts on this question, please? Am I correct in thinking that federating clusters is related to my question? Do I gather correctly, however, that federation only works if there is a common database on a shared file system? Best regards, David

2 1

slurmdb endpoints for slurmrestd
by Jackson, Gary L. 05 Jan '24

05 Jan '24

I get an HTTP 404 when I try to GET /slurmdb/v0.0.39/clusters or any other /slurmdb endpoint. I get this against multiple versions of Slurm, including 23.11.1. Using GET against /slurm/v0.0.39/ping works just fine. Is there something I need to do to turn slurmdb endpoints on? -- Gary

1 0

Slurp for sw builds
by Duane Ellis 04 Jan '24

04 Jan '24

In my case I would like to use a slurm cluster for a sw ci/cd like solution for building sw images My current scripted full system build takes 3-5 hours and is done serially we could easily find places where we can choose to build things in parallel hence the idea is to spawn parallel builds on the Linux slurm cluster example: we have a list of images we iterate over in a for loop to build each thing the steps are: cd somedir then type make or run a shell script in that directory The last step after the for loop would be wait for all of the child builds to complete Once all child jobs are done we have a single job that combines or packages all the intermediate images Really want to use slurm because our FPGA team will have a giant slurm linux cluster for Xilinix FPGA builds and those nodes can do what we need for sw purposes (reusing the existing cluster is a huge win for us) My question is this: Can somebody point me to some sw build examples for or using slurm? All I can seem to find is how to install I see the srun and sbatch command man pages but no good examples Bonus would be something that integrates into a gitlab runner example or Jenkins in some way All I can seem to find is how to install and administer slurm not a how to use slurm Sent from my iPhone

3 2

Fwd: Fairshare: users not added
by Alex Ninaber 04 Jan '24

04 Jan '24

Hi all, A problem on slurm-23.02.4-1, 10.6.16-MariaDB; Maria and Slurmctld in active/active, SlurmDB in active/off, shared IP. Shared spool via Gluster. DB is an upgraded version of Slurm from somewhere 2017 (upgraded various times). The question is whether we should give up and start from scratch or if there's an easy fix. Problem: whenever we add a new user and add it to sacctmgr, the user shows up properly in sacct/mgr – but never shows up with the sshare commands after running some jobs. After restarting slurm a couple of times it shows up. Problem seems to be there also in the previous version. Only error we can see in slurmdb log: [2023-12-21T09:43:30.586] error: slurm_persist_conn_open: Something happened with the receiving/processing of the persistent connection init message to 10.141.255.253:6817 : (null) [2023-12-21T09:43:30.586] error: slurmdb_send_accounting_update_persist: Unable to open connection to registered cluster cluster. [2023-12-21T09:43:30.586] error: slurm_receive_msg: No response to persist_init [2023-12-21T09:43:30.586] error: update cluster: No error to cluster at 10.141.255.253(6817) [2023-12-21T09:43:30.586] debug2: DBD_FINI: CLOSE:1 COMMIT:0 [2023-12-21T09:43:30.586] debug4: accounting_storage/as_mysql: acct_storage_p_commit: got 0 commits AccountingStorageType=accounting_storage/slurmdbd # jobaccounting JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/linux SlurmctldTimeout=60 SlurmdTimeout=60 TCPTimeout=60 MessageTimeout=60 Best regards, Alex

1 0

All nodes within one partition reboot unexpectedly
by Jinglei Hu 02 Jan '24

02 Jan '24

Hi all, I had a slurm partition gpu_gmx with the following configuration (Slurm version: 20.11.9): > NodeName=node[09-11] Gres=gpu:rtx4080:1 Sockets=1 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=62000 State=UNKNOWN > NodeName=node[12-14] Gres=gpu:rtx4070ti:1 Sockets=1 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=62000 State=UNKNOWN > PartitionName=gpu_gmx Nodes=node[09-14] Default=NO MaxTime=UNLIMITED State=UP A job running on node11 had a problem, which then triggered the reboot of all nodes (node[09-14]) within the same partition (cat /var/log/slurmctld.log): > [2023-12-26T23:04:23.200] Batch JobId=25061 missing from batch node node11 (not found BatchStartTime after startup), Requeuing job > [2023-12-26T23:04:23.200] _job_complete: JobId=25061 WTERMSIG 126 > [2023-12-26T23:04:23.200] _job_complete: JobId=25061 cancelled by node failure > [2023-12-26T23:04:23.200] _job_complete: requeue JobId=25061 due to node failure > [2023-12-26T23:04:23.200] _job_complete: JobId=25061 done > [2023-12-26T23:04:23.200] validate_node_specs: Node node11 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:23.222] validate_node_specs: Node node09 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:23.579] Batch JobId=25060 missing from batch node node10 (not found BatchStartTime after startup), Requeuing job > [2023-12-26T23:04:23.579] _job_complete: JobId=25060 WTERMSIG 126 > [2023-12-26T23:04:23.579] _job_complete: JobId=25060 cancelled by node failure > [2023-12-26T23:04:23.579] _job_complete: requeue JobId=25060 due to node failure > [2023-12-26T23:04:23.579] _job_complete: JobId=25060 done > [2023-12-26T23:04:23.579] validate_node_specs: Node node10 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:23.581] validate_node_specs: Node node14 unexpectedly rebooted boot_time=1703603051 last response=1703602983 > [2023-12-26T23:04:23.654] validate_node_specs: Node node13 unexpectedly rebooted boot_time=1703603052 last response=1703602983 > [2023-12-26T23:04:24.681] validate_node_specs: Node node12 unexpectedly rebooted boot_time=1703603053 last response=1703602983 > [2023-12-27T04:46:42.461] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=25060 uid 0 > [2023-12-27T04:46:43.822] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=25061 uid 0 The operating systems are CentOS 7.9.2009 on master node, and CentOS 8.5.2111 on node[09-14]. Does anyone have a similar experience and have a clue how to resolve this? Thanks in advance. Best, Jinglei

1 0

2025

2024

slurm-users January 2024

2025

2024

slurm-users January 2024 ----- 2025 ----- April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users January 2024