January 2024 - slurm-users - lists.schedmd.com

job_container/tmpfs and srun.
by Phill Harvey-Smith 17 Jan '25

17 Jan '25

Hi all, On our setup we are using job_container/tmpfs to give each job it's own temp space. Since our compute nodes have reasonably sized disks for tasks that do a lot of disk I/O on user's data we have asked users to copy their data to the local disk at the beginning of the task and (if needed) copy it back at the end. This saves lots of NFS thrashing slowing down both the task and the NFS servers. However some of our users are having problems with this, their initial sbatch script will create a temp directory in their private /tmp copy their data to it and then try to srun a program. The srun will fall over as it doesn't seem to have have access to the copied data. I suspect this is because the srun task is getting it's own private /tmp. So my question is, is there a way to have the srun task inherit the /tmp of the initial sbatch? I'll include a sample of the script our user is using below. If any further information is required please feel free to ask. Cheers. Phill. #!/usr/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:00:10 #SBATCH --mem-per-cpu=3999 #SBATCH --output=script_out.log #SBATCH --error=script_error.log # The above options puts the STDOUT and STDERR of sbatch in # log files prefixed with 'script_'. # Create a randomly-named directory under /tmp jobtmpdir=$(mktemp -d) # Register a function to try and cleanup in case of job failure cleanup_handler() { echo "Cleaning up ${jobtmpdir}" rm -rf ${jobtmpdir} } trap 'cleanup_handler' SIGTERM EXIT # Change working directory to this directory cd ${jobtmpdir} # Copy the executable and input files from # where the job was submitted to the temporary directory. cp ${SLURM_SUBMIT_DIR}/a.out . cp ${SLURM_SUBMIT_DIR}/input.txt . # Run the executable, handling the collection of stdout # and stderr ourselves by redirecting to file srun ./a.out 2> task_error.log > task_out.log # Copy output data back to the submit directory. cp output.txt ${SLURM_SUBMIT_DIR} cp task_out.log ${SLURM_SUBMIT_DIR} cp task_error.log ${SLURM_SUBMIT_DIR} # Cleanup cd ${SLURM_SUBMIT_DIR} cleanup_handler

2 1

Issues with Slurm 23.11.1
by Fokke Dijkstra 15 Mar '24

15 Mar '24

Dear all, Since the upgrade from Slurm 22.05 to 23.11.1 we are having problems with the communication between the slurmctld and slurmd processes. We are running a cluster with 183 nodes and almost 19000 cores. Unfortunately some nodes are in a different network preventing full internode communication. A network topology and setting TopologyParam RouteTree have been used to make sure no slurmd communication happens between nodes on different networks. In the new Slurm version we see the following issues, which did not appear in 22.05: 1. slurmd processes acquire many network connections in CLOSE-WAIT (or CLOSE_WAIT depending on the tool used) causing the processes to hang, when trying to restart slurmd. When checking for CLOSE-WAIT processes we see the following behaviour: Recv-Q Send-Q Local Address:Port Peer Address:Port Process 1 0 10.5.2.40:6818 10.5.0.43:58572 users:(("slurmd",pid=1930095,fd=72),("slurmd",pid=1930067,fd=72)) 1 0 10.5.2.40:6818 10.5.0.43:58284 users:(("slurmd",pid=1930095,fd=8),("slurmd",pid=1930067,fd=8)) 1 0 10.5.2.40:6818 10.5.0.43:58186 users:(("slurmd",pid=1930095,fd=22),("slurmd",pid=1930067,fd=22)) 1 0 10.5.2.40:6818 10.5.0.43:58592 users:(("slurmd",pid=1930095,fd=76),("slurmd",pid=1930067,fd=76)) 1 0 10.5.2.40:6818 10.5.0.43:58338 users:(("slurmd",pid=1930095,fd=19),("slurmd",pid=1930067,fd=19)) 1 0 10.5.2.40:6818 10.5.0.43:58568 users:(("slurmd",pid=1930095,fd=68),("slurmd",pid=1930067,fd=68)) 1 0 10.5.2.40:6818 10.5.0.43:58472 users:(("slurmd",pid=1930095,fd=69),("slurmd",pid=1930067,fd=69)) 1 0 10.5.2.40:6818 10.5.0.43:58486 users:(("slurmd",pid=1930095,fd=38),("slurmd",pid=1930067,fd=38)) 1 0 10.5.2.40:6818 10.5.0.43:58316 users:(("slurmd",pid=1930095,fd=29),("slurmd",pid=1930067,fd=29)) The first IP address is that of the compute node, the second that of the node running slurmctld. The nodes can communicate using these IP addresses just fine. 2. slurmd cannot be properly restarted [2024-01-18T10:45:26.589] slurmd version 23.11.1 started [2024-01-18T10:45:26.593] error: Error binding slurm stream socket: Address already in use [2024-01-18T10:45:26.593] fatal: Unable to bind listen port (6818): Address already in use This is probably because of the processes being in CLOSE-WAIT, which can only be killed using signal -9. 3. We see jobs stuck in completing CG state, probably due to communication issues between slurmctld and slurmd. The slurmctld sends repeated kill requests but those do not seem to be acknowledged by the client. This happens more often in large job arrays, or generally when many jobs start at the same time. However, this could be just a biased observation (i.e., it is more noticeable on large job arrays because there are more jobs to fail in the first place). 4. Since the new version we also see messages like: [2024-01-17T09:58:48.589] error: Failed to kill program loading user environment [2024-01-17T09:58:48.590] error: Failed to load current user environment variables [2024-01-17T09:58:48.590] error: _get_user_env: Unable to get user's local environment, running only with passed environment The effect of this is that the users run with the wrong environment and can’t load the modules for the software that is needed by their jobs. This leads to many job failures. The issue appears to be somewhat similar to the one described at: https://bugs.schedmd.com/show_bug.cgi?id=18561 In that case the site downgraded the slurmd clients to 22.05 which got rid of the problems. We’ve now downgraded the slurmd on the compute nodes to 23.02.7 which also seems to be a workaround for the issue. Does anyone know of a better solution? Kind regards, Fokke Dijkstra -- Fokke Dijkstra <f.dijkstra(a)rug.nl> <f.dijkstra(a)rug.nl> Team High Performance Computing Center for Information Technology, University of Groningen Postbus 11044, 9700 CA Groningen, The Netherlands

2 3

slurmctld: slurm_bufs_sendto(msg_type=SRUN_STEP_SIGNAL) failed: Connection reset by peer
by Rike-Benjamin Schuppner 16 Feb '24

16 Feb '24

Hi, I am getting the following error in the logs whenever I run a few srun jobs in a batch. Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: debug: _send_timeout: Socket POLLERR: Connection reset by peer Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: error: slurm_send_node_msg: [socket:[921897]] slurm_bufs_sendto(msg_type=SRUN_STEP_SIGNAL) failed: Connection reset by peer Jan 25 11:24:03 slurmctl.XYZ slurmctld[272961]: slurmctld: debug: laying out the 1 tasks on 1 hosts compute2 dist 1 The slurm version is 23.11.3 and an example sbatch file is: #!/bin/bash #SBATCH --job-name=slurm_test #SBATCH --mem=1gb #SBATCH --time=00:05:00 #SBATCH --output=slurm_test_%j.log pwd; hostname; date srun --exclusive -c1 -N1 -n1 bash -c "hostname ; sleep 10" & srun --exclusive -c1 -N1 -n1 bash -c "hostname ; sleep 10" & wait The more sruns I have in the script (&-backgrounded or not), the more the error shows up. Is there anything I could do to fix this error? Best /rike

2 1

Compilation question
by Sylvain MARET 10 Feb '24

10 Feb '24

Hello everyone ! I'm trying to compile slurm 22.05.11 on Rocky linux 8.7 with freeipmi support I've seen the documentation so I've done the configure step : ./configure --with-pmix=$PMIXHOME --with-ucx=$UCXHOME --with-nvml=$NVMLHOME --prefix=$SLURMHOME --with-freeipmi=/usr but when I run make I end up with the following error : /bin/sh ../../../../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -L/usr/lib64 -lhdf5_hl -lhdf5 -lsz -lz -ldl -lm -o sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -L../../../../../src/api/.libs -lslurmfull -ldl ../libhdf5_api.la -lpthread -lm -lresolv libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o .libs/sh5util sh5util.o -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -Wl,--export-dynamic -L/usr/lib64 -L../../../../../src/api/.libs /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so ../.libs/libhdf5_api.a -lhdf5_hl -lhdf5 -lsz -lz -ldl -lpthread -lm -lresolv -pthread -Wl,-rpath -Wl,/softs/batch/slurm/22.05.11/lib/slurm sh5util.o:(.init_array+0x0): undefined reference to `__nv_init_env' sh5util.o:(.init_array+0x8): undefined reference to `__flushz' sh5util.o:(.init_array+0x10): undefined reference to `__daz' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_transfer_unique' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_sort_key_pairs' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrchr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_unsetenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_sort' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_for_each' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin__pgi_isnanld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_extra_conf_path' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_ctzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_running_in_slurmctld' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mcopy1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_clzll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_count' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__builtin_va_gparg1' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_destroy_config_key_pair' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xfree_ptr' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_getenvp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_free_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_get_log_level' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__c_mset8' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup_printf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_delete_first' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_append' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_error' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_init_buf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_log_var' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_parse_file' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrdup' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_fatal' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `__blt_pgi_popcountll' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_xstrcmp' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_destroy' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_s_p_hashtbl_create' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_setenvpf' /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: undefined reference to `slurm_list_destroy' collect2: error: ld returned 1 exit status What can I do to resolve these undefined reference error ? Regards, Sylvain Maret

3 2

after upgrade to 23.11.1 nodes stuck in completion state
by Paul Raines 01 Feb '24

01 Feb '24

I finally had downtime on our cluster running 20.11.3 and decided to upgrade SLURM. All daemons were stopped on nodes and master. Rocky 8 Linux OS was updated but not changed configuration-wise in anyway. On the master, when I first installed 23.11.1 and tried to run slurmdbd -D -vvv at the command line, it balked as the more than 2 major version jump. I then installed 22.05.6 and run slurmdbd -D -vvv which took a long time but completed fine. I reinstalled 23.11.1 and did slurmdbd -D --vvv which went quick. I stopped it then ran slurmdbd normally with systemctl. When I ran slurmctld it complained about incompatable state files. I didn't care about loosing state so ran with the -i option to ignore. Seemed happy. I then started up slurmd on my various GPU and non-GPU nodes. I ran a test interactive job just to see if I would get a shell with expected SLURM environment on the nodes and it worked. Except on the GPU nodes I got errors about not being able to get/set GPU frequences. I removed GpuFreqDef=medium from slurm.conf and that went away. I had my people start running jobs last night. This morning I come in and see a lot of nodes are stuck in completing state. All these are GPU nodes. For example on node 'sevilla' squeue reports JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 3679433 lcna100 sjob_123 sg1526 CG 8:29 1 sevilla 3679432 lcna100 sjob_123 sg1526 CG 8:35 1 sevilla 3679431 lcna100 sjob_123 sg1526 CG 8:42 1 sevilla In the slurmd.log on sevilla I see every minute the lines: [2024-01-28T09:37:52.003] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug: _rpc_terminate_job: uid = 1150 JobId=3679432 [2024-01-28T09:37:52.004] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug: _rpc_terminate_job: uid = 1150 JobId=3679431 [2024-01-28T09:37:52.004] debug2: Finish processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug: _rpc_terminate_job: uid = 1150 JobId=3679433 [2024-01-28T09:37:52.004] debug2: Finish processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T09:37:52.004] debug2: Finish processing RPC: REQUEST_TERMINATE_JOB Back at the time 3679433 was cancelled by the user I see [2024-01-28T00:27:48.658] debug2: Start processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T00:27:48.658] debug2: Processing RPC: REQUEST_TERMINATE_JOB [2024-01-28T00:27:48.658] debug: _rpc_terminate_job: uid = 1150 JobId=3679433 [2024-01-28T00:27:48.658] debug: credential for job 3679433 revoked [2024-01-28T00:27:48.658] debug: _rpc_terminate_job: sent SUCCESS for 3679433, waiting for prolog to finish [2024-01-28T00:27:48.658] debug: Waiting for job 3679433's prolog to complete On sevilla itself there are no processes currently running for user sg1526 so no sign of these jobs. There ARE processes running for user mu40 though [root@sevilla ~]# pgrep -f slurm_script | xargs -n1 pstree -p | grep ^slurm slurm_script(30691)---starter-suid(30764)-+-python3(30784)-+-{python3}(30817) slurm_script(30861)---starter-suid(30873)-+-python3(30888)-+-{python3}(30915) slurm_script(30963)---starter-suid(30975)-+-python3(30992)-+-{python3}(31020) [root@sevilla ~]# strings /proc/30691/cmdline /bin/sh /var/slurm/spool/d/job3679220/slurm_script /usr/bin/sacct -p --jobs=3679220 --format=JobID,User,ReqTRES,NodeList,Start,End,Elapsed,CPUTime,State,ExitCode JobID|User|ReqTRES|NodeList|Start|End|Elapsed|CPUTime|State|ExitCode| 3679220|mu40|billing=6,cpu=3,gres/gpu=1,mem=32G,node=1|sevilla|2024-01-27T22:33:27|2024-01-27T23:26:39|00:53:12|02:39:36|NODE_FAIL|1:0| 3679220.batch|||sevilla|2024-01-27T22:33:27|2024-01-27T23:26:39|00:53:12|02:39:36|CANCELLED|| 3679220.extern|||sevilla|2024-01-27T22:33:27|2024-01-27T23:26:39|00:53:12|02:39:36|CANCELLED|| The slurmd.log has at the time it started lines like: [2024-01-27T22:33:28.679] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30691, probably hasn't started yet or has already finished [2024-01-27T22:33:58.092] [3679220.extern] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30650, probably hasn't started yet or has already finished [2024-01-27T22:33:58.166] [3679220.batch] debug2: profile signaling type Task [2024-01-27T22:33:58.168] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30691, probably hasn't started yet or has already finished [2024-01-27T22:33:58.170] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30764, probably hasn't started yet or has already finished [2024-01-27T22:33:58.172] [3679220.batch] debug2: gpu/nvml: _get_gpuutil: Couldn't find pid 30784, probably hasn't started yet or has already finished ... [2024-01-27T22:46:39.018] debug: _step_connect: connect() failed for /var/slurm/spool/d/sevilla_3679220.4294967292: Connection refused [2024-01-27T22:46:39.018] debug: Cleaned up stray socket /var/slurm/spool/d/sevilla_3679220.4294967292 [2024-01-27T22:46:39.018] debug: _step_connect: connect() failed for /var/slurm/spool/d/sevilla_3679220.4294967291: Connection refused [2024-01-27T22:46:39.018] debug: Cleaned up stray socket /var/slurm/spool/d/sevilla_3679220.4294967291 [2024-01-27T22:46:39.018] _handle_stray_script: Purging vestigial job script /var/slurm/spool/d/job3679220/slurm_script then later at the time of termination [2024-01-27T23:26:39.042] debug: _rpc_terminate_job: uid = 1150 JobId=3679220 [2024-01-27T23:26:39.042] debug: credential for job 3679220 revoked [2024-01-27T23:26:39.042] debug2: No steps in jobid 3679220 to send signal 998 [2024-01-27T23:26:39.043] debug2: No steps in jobid 3679220 to send signal 18 [2024-01-27T23:26:39.043] debug2: No steps in jobid 3679220 to send signal 15 [2024-01-27T23:26:39.044] debug2: set revoke expiration for jobid 3679220 to 1706416119 UTS [2024-01-27T23:26:39.044] debug: Waiting for job 3679220's prolog to complete [2024-01-27T23:26:39.044] debug: Finished wait for job 3679220's prolog to complete [2024-01-27T23:26:39.068] debug: completed epilog for jobid 3679220 [2024-01-27T23:26:39.071] debug: JobId=3679220: sent epilog complete msg: rc = 0 So I have a node stuck in 'comp' state that claims to have jobs completing that are definite NOT running on the box BUT there are jobs running on the box that SLURM thinks are done --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

4 7

Why is Slurm 20 the latest RPM in RHEL 8/Fedora repo?
by Robert Kudyba 31 Jan '24

31 Jan '24

According to these links: https://rpmfind.net/linux/rpm2html/search.php?query=slurm https://src.fedoraproject.org/rpms/slurm Why doesn't RHEL 8 get a newer version? Can someone update the repo maintainer Philip Kovacs <https://src.fedoraproject.org/user/pkfed> < pkfed(a)fedoraproject.org>? There was a ticket at https://bugzilla.redhat.com/show_bug.cgi?id=1912491 but no movement on RHEL 8.

5 4

Mailing list upgrade - slurm-users list paused
by Tim Wickberg 30 Jan '24

30 Jan '24

Hey folks - The mailing list will be offline for about an hour as we upgrade the host, upgrade the mailing list software, and change the mail configuration around. As part of these changes, the "From: " field will no longer be the original sender, but instead use the mailing list ID itself. This is to comply with DMARC sending options, and allow us to start DKIM signing messages to ensure deliverability once Google and Yahoo impose new policy changes in February. This is the last post on the current (mailman2) list. I'll send a welcome message on the upgraded (mailman3) list once finished, and when the list is open to new traffic again. - Tim -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 1

Testing to SchedMD folks
by Tim Wickberg 30 Jan '24

30 Jan '24

Hey - this is hopefully a final test before the list is re-enabled for everyone. - Tim

1 0

Socket timed out - tuning
by Reed Dier 29 Jan '24

29 Jan '24

Hoping someone can help point me towards some tweaks to help prevent denial-of-service issues. > sbatch: error: Batch job submission failed: Socket timed out on send/recv operation Root cause is understood, issues with shared storage for the slurmctld’s was impacted, leading to an increase in write latency to the StateSaveLocation. Then with a large enough avalanche of job submissions, things the RPC’s would stack up and stop responding. I’ve been running well with some tweaks sourced from the “high-throughput” guide <https://slurm.schedmd.com/high_throughput.html>. > SchedulerParameters=max_rpc_cnt=400,\ > sched_min_interval=50000,\ > sched_max_job_start=300,\ > batch_sched_delay=6 > KillWait=30 > MessageTimeout=30 I’m assuming that I was running into batch_sched_delay because looking at sdiag after the fact, it was averaging .2s, and total time is 5.5h out of 16h8m18s at the time of the sdiag sample. > ******************************************************* > sdiag output at Thu Jan 25 11:08:18 2024 (1706198898) > Data since Wed Jan 24 19:00:00 2024 (1706140800) > ******************************************************* > REQUEST_SUBMIT_BATCH_JOB ( 4003) count:98400 ave_time:201442 total_time:19821991013 Currently on 22.05.8, but hoping to get to 23.02.7 soon™, and I think this could possible resolve the issue well enough if I’m reading it correctly from the release notes <https://slurm.schedmd.com/archive/slurm-23.02-latest/news.html>? > HIGHLIGHTS > ========== > -- slurmctld - Add new RPC rate limiting feature. This is enabled through > SlurmctldParameters=rl_enable, otherwise disabled by default. > rl_enable <https://slurm.schedmd.com/slurm.conf.html#OPT_rl_enable>Enable per-user RPC rate-limiting support. Client-commands will be told to back off and sleep for a second once the limit has been reached. This is implemented as a "token bucket", which permits a certain degree of "bursty" RPC load from an individual user before holding them to a steady-state RPC load established by the refill period and rate. But given that the hardware seems to be well over provisioned, CPU never drops below 5% idle, it feels like there is more room to squeeze some optimization out of here that I’m missing in the interim, and hoping to get a better overall understanding in the process. I scrape the DBD Agent queue size from sdiag every 30s and the largest value I saw was 115, which is much higher than normal, but should be well below MaxDBDMsgs, where the minimum value is 10000. I would really hope that I didn’t potentially hit a 30s MessageTimeout value, but I guess thats on the table all well, not knowing if that would potentially trigger the sbatch submission failure like that. Just moving the max_rpc_cnt value up seems like an easy button, but also seems like it could have some adverse effects for backfill scheduling, and may be diminishing returns for actually keeping RPCs flowing? > Setting max_rpc_cnt to more than 256 will be only useful to let backfill continue scheduling work after locks have been yielded (i.e. each 2 seconds) if there are a maximum of MAX(max_rpc_cnt/10, 20) RPCs in the queue. i.e. max_rpc_cnt=1000, the scheduler will be allowed to continue after yielding locks only when there are less than or equal to 100 pending RPCs. Obviously, fix the storage is the real solution, but hoping that there may be more goodness to unlock, even if it is as simple as “upgrade to 23.02”. Appreciate any insight, Reed

1 0

Two jobs each with a different partition running on same node?
by Loris Bennett 29 Jan '24

29 Jan '24

Hi, I seem to remember that in the past, if a node was configured to be in two partitions, the actual partition of the node was determined by the partition associated with the jobs running on it. Moreover, at any instance where the node was running one or more jobs, the node could only actually be in a single partition. Was this indeed the case and is it still the case with version Slurm 23.02.7? Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT (ex-ZEDAT), Freie Universität Berlin

2 1