- slurm-users - lists.schedmd.com

sbatch problem
by Mihai Ciubancan 29 May '24

29 May '24

Hello, My name is Mihai and a have an issue with a small GPU cluster manage with slurm 22.05.11. I got 2 different output when I'm trying to find out the name of the nodes(one correct and one wrong). The script is: #!/bin/bash #SBATCH --job-name=test #SBATCH --output=/data/mihai/res.txt #SBATCH --partition=eli #SBATCH --nodes=2 srun echo Running on host: $(hostname) srun hostname srun sleep 15 And the output look like this: cat res.txt Running on host: mihai-x8640 Running on host: mihai-x8640 mihaigpu2 mihai-x8640 As you can see the output of the command 'srun echo Running on host: $(hostname)' is the same, as the jobs was running twice on the same node, while command 'srun hostname' it's giving me the correct output. Do you have any idea why the outputs of the 2 commands are different? Thank you, Mihai

2 6

slurm_bufs_sendto failed
by shaobo liu 28 May '24

28 May '24

slurm 23.11.0 , the slurmd error is as follows，what is the reason for this? [2024-05-29T09:21:32.450] task/affinity: task_p_slurmd_batch_request: task_p_slurmd_batch_request: 32259 [2024-05-29T09:21:32.451] task/affinity: batch_bind: job 32259 CPU input mask for node: 0xFFFFFFFFFFFFFFFF [2024-05-29T09:21:32.451] task/affinity: batch_bind: job 32259 CPU final HW mask for node: 0xFFFFFFFFFFFFFFFF [2024-05-29T09:21:32.451] Launching batch job 32259 for UID 66003 [2024-05-29T09:21:32.868] launch task StepId=32259.0 request from UID:66003 GID:66003 HOST:192.168.87.45 PORT:31104 [2024-05-29T09:21:32.868] task/affinity: lllp_distribution: JobId=32259 implicit auto binding: cores,one_thread, dist 2 [2024-05-29T09:21:32.868] task/affinity: _task_layout_lllp_cyclic: _task_layout_lllp_cyclic [2024-05-29T09:21:32.868] task/affinity: _lllp_generate_cpu_bind: _lllp_generate_cpu_bind jobid [32259]: mask_cpu,one_thread, 0x0000000000000001,0x0000000100000000,0x0000000000000002,0x0000000200000000,0x0000000000000004,0x0000000400000000,0x0000000000000008,0x0000000800000000,0x0000000000000010,0x0000001000000000,0x0000000000000020,0x0000002000000000,0x0000000000000040,0x0000004000000000,0x0000000000000080,0x0000008000000000,0x0000000000000100,0x0000010000000000,0x0000000000000200,0x0000020000000000,0x0000000000000400,0x0000040000000000,0x0000000000000800,0x0000080000000000,0x0000000000001000,0x0000100000000000,0x0000000000002000,0x0000200000000000,0x0000000000004000,0x0000400000000000,0x0000000000008000,0x0000800000000000,0x0000000000010000,0x0001000000000000,0x0000000000020000,0x0002000000000000,0x0000000000040000,0x0004000000000000,0x0000000000080000,0x0008000000000000,0x0000000000100000,0x0010000000000000,0x0000000000200000,0x0020000000000000,0x0000000000400000,0x0040000000000000,0x0000000000800000,0x0080000000000000,0x0000000001000000,0x0100000000000000,0x0000000002000000,0x0200000000000000,0x0000000004000000,0x0400000000000000,0x0000000008000000,0x0800000000000000,0x0000000010000000,0x1000000000000000,0x0000000020000000,0x2000000000000000,0x0000000040000000,0x4000000000000000,0x0000000080000000,0x8000000000000000 [2024-05-29T09:32:04.541] [32259.batch] done with step [2024-05-29T09:33:31.000] error: slurm_send_node_msg: [socket:[7270627]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:34:02.000] error: slurm_send_node_msg: [socket:[7270629]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:34:03.016] [32259.extern] done with step [2024-05-29T09:34:41.000] error: slurm_send_node_msg: [socket:[7270635]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:35:13.000] error: slurm_send_node_msg: [socket:[7270637]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:35:52.000] error: slurm_send_node_msg: [socket:[7270639]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:36:01.000] error: slurm_send_node_msg: [socket:[7270630]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:36:24.000] error: slurm_send_node_msg: [socket:[7270641]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:37:03.000] error: slurm_send_node_msg: [socket:[7270644]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:37:31.000] [32259.0] error: *** STEP 32259.0 STEPD TERMINATED ON node045 AT 2024-05-29T09:37:30 DUE TO JOB NOT RUNNING *** [2024-05-29T09:37:33.000] [32259.0] stepd_cleanup: done with step (rc[0x0]:No error, cleanup_rc[0xfbb]:Job step not running) [2024-05-29T09:37:35.000] error: slurm_send_node_msg: [socket:[7270646]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:38:14.000] error: slurm_send_node_msg: [socket:[7270649]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:38:46.000] error: slurm_send_node_msg: [socket:[7270651]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-05-29T09:39:25.000] error: slurm_send_node_msg: [socket:[7270657]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error

1 0

Final Call for SLUG Presenters & Early Bird Registration!
by Victoria Hobson 28 May '24

28 May '24

Slurm User Group (SLUG) 2024 is set for September 12-13 at the University of Oslo in Oslo, Norway. Registration information and a high-level schedule can be found here:https://slug24.splashthat.com/ The last day to register at the early bird pricing is this Friday, May 31st. Friday is also the deadline to submit a presentation abstract. We do not intend to extend this deadline. If you are interested in presenting your own usage, developments, site report, tutorial, etc about Slurm, please fill out the following form:https://forms.gle/N7bFo5EzwuTuKkBN7 Notifications of final presentations accepted will go out by Friday, June 14th. -- Victoria Hobson SchedMD LLC Vice President of Marketing

1 0

slurmdbd archive format
by O'Neal, Doug (NIH/NCI) [C] 28 May '24

28 May '24

My organization needs to access historic job information records for metric reporting and resource forecasting. slurmdbd is archiving only the job information, which should be sufficient for our numbers, but is using the default archive script. In retrospect, this data should have been migrated to a secondary MariaDB instance, but that train has passed. The format of the archive files is not well documented. Does anyone have a program (python/C/whatever) that will read a job_table_archive file and decode it into a parsable structure? Douglas O'Neal, Ph.D. (contractor) Manager, HPC Systems Administration Group, ITOG Frederick National Laboratory for Cancer Research Leidos Biomedical Research, Inc. Phone: 301-228-4656 Email: Douglas.O'Neal(a)nih.gov<mailto:Doug%20O'Neal%20%3cDouglas.O'Neal(a)nih.gov%3e>

2 2

Fwd: Re: Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs.
by Hongyi Zhao 28 May '24

28 May '24

---------- Forwarded message --------- From: Hermann Schwärzler <hermann.schwaerzler(a)uibk.ac.at> Date: Tue, May 28, 2024 at 4:10 PM Subject: Re: [slurm-users] Re: Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs. To: Hongyi Zhao <hongyi.zhao(a)gmail.com> Hi Zhao, On 5/28/24 03:08, Hongyi Zhao wrote: [...] > > What's the complete content of cli_filter.lua and where should I put this file? [...] Below you find the complete content of our cli_filter.lua. It has to be put into the same directory as "slurm.conf". --------------------------------- 8< --------------------------------- -- see https://github.com/SchedMD/slurm/blob/master/etc/cli_filter.lua.example function slurm_cli_pre_submit(options, pack_offset) return slurm.SUCCESS end function slurm_cli_setup_defaults(options, early_pass) -- Make --hint=nomultithread the default behavior -- if users specify an other --hint=XX option then -- it will override the setting done here options['hint'] = 'nomultithread' return slurm.SUCCESS end function slurm_cli_post_submit(offset, job_id, step_id) return slurm.SUCCESS end --------------------------------- >8 --------------------------------- Hopefully this helps... Regards, Hermann -- Assoc. Prof. Hongsheng Zhao <hongyi.zhao(a)gmail.com> Theory and Simulation of Materials Hebei Vocational University of Technology and Engineering No. 473, Quannan West Street, Xindu District, Xingtai, Hebei province

1 0

Performance Discrepancy between Slurm and Direct mpirun for VASP Jobs.
by Hongyi Zhao 27 May '24

27 May '24

Dear Slurm Users, I am experiencing a significant performance discrepancy when running the same VASP job through the Slurm scheduler compared to running it directly with mpirun. I am hoping for some insights or advice on how to resolve this issue. System Information: Slurm Version: 21.08.5 OS: Ubuntu 22.04.4 LTS (Jammy) Job Submission Script: #!/usr/bin/env bash #SBATCH -N 1 #SBATCH -D . #SBATCH --output=%j.out #SBATCH --error=%j.err ##SBATCH --time=2-00:00:00 #SBATCH --ntasks=36 #SBATCH --mem=64G echo '#######################################################' echo "date = $(date)" echo "hostname = $(hostname -s)" echo "pwd = $(pwd)" echo "sbatch = $(which sbatch | xargs realpath -e)" echo "" echo "WORK_DIR = $WORK_DIR" echo "SLURM_SUBMIT_DIR = $SLURM_SUBMIT_DIR" echo "SLURM_JOB_NUM_NODES = $SLURM_JOB_NUM_NODES" echo "SLURM_NTASKS = $SLURM_NTASKS" echo "SLURM_NTASKS_PER_NODE = $SLURM_NTASKS_PER_NODE" echo "SLURM_CPUS_PER_TASK = $SLURM_CPUS_PER_TASK" echo "SLURM_JOBID = $SLURM_JOBID" echo "SLURM_JOB_NODELIST = $SLURM_JOB_NODELIST" echo "SLURM_NNODES = $SLURM_NNODES" echo "SLURMTMPDIR = $SLURMTMPDIR" echo '#######################################################' echo "" module purge > /dev/null 2>&1 module load vasp ulimit -s unlimited mpirun vasp_std Performance Observation: When running the job through Slurm: werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ grep LOOP OUTCAR LOOP: cpu time 14.4893: real time 14.5049 LOOP: cpu time 14.3538: real time 14.3621 LOOP: cpu time 14.3870: real time 14.3568 LOOP: cpu time 15.9722: real time 15.9018 LOOP: cpu time 16.4527: real time 16.4370 LOOP: cpu time 16.7918: real time 16.7781 LOOP: cpu time 16.9797: real time 16.9961 LOOP: cpu time 15.9762: real time 16.0124 LOOP: cpu time 16.8835: real time 16.9008 LOOP: cpu time 15.2828: real time 15.2921 LOOP+: cpu time 176.0917: real time 176.0755 When running the job directly with mpirun: werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ mpirun -n 36 vasp_std werner@x13dai-t:~/Public/hpc/servers/benchmark/Cr72_3x3x3K_350eV_10DAV$ grep LOOP OUTCAR LOOP: cpu time 9.0072: real time 9.0074 LOOP: cpu time 9.0515: real time 9.0524 LOOP: cpu time 9.1896: real time 9.1907 LOOP: cpu time 10.1467: real time 10.1479 LOOP: cpu time 10.2691: real time 10.2705 LOOP: cpu time 10.4330: real time 10.4340 LOOP: cpu time 10.9049: real time 10.9055 LOOP: cpu time 9.9718: real time 9.9714 LOOP: cpu time 10.4511: real time 10.4470 LOOP: cpu time 9.4621: real time 9.4584 LOOP+: cpu time 110.0790: real time 110.0739 Could you provide any insights or suggestions on what might be causing this performance issue? Are there any specific configurations or settings in Slurm that I should check or adjust to align the performance more closely with the direct mpirun execution? Thank you for your time and assistance. Best regards, Zhao -- Assoc. Prof. Hongsheng Zhao <hongyi.zhao(a)gmail.com> Theory and Simulation of Materials Hebei Vocational University of Technology and Engineering No. 473, Quannan West Street, Xindu District, Xingtai, Hebei province

4 10

Building Slurm debian package vs building from source
by Arnuld 23 May '24

23 May '24

We have several nodes, most of which have different Linux distributions (distro for short). Controller has a different distro as well. The only common thing between controller and all the does is that all of them ar x86_64. I can install Slurm using package manager on all the machines but this will not work because controller will have a different version of Slurm compared to the nodes (21.08 vs 23.11) If I build from source then I see two solutions: - build a deb package - build a custom package (./configure, make, make install) Building a debian package on the controller and then distributing the binaries on nodes won't work either because that binary will start looking for the shared libraries that it was built for and those don't exist on the nodes. So the only solution I have is to build a static binary using a custom package. Am I correct or is there another solution here?

4 7

Slurm Build Error
by Arnuld 23 May '24

23 May '24

Getting this error when I run "make install": echo >>"lib_ref.lo" /bin/bash ../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o lib_ref.la lib_ref.lo -lpthread -lm -lresolv libtool: link: ar cr .libs/lib_ref.a libtool: link: ranlib .libs/lib_ref.a libtool: link: ( cd ".libs" && rm -f "lib_ref.la" && ln -s "../lib_ref.la" " lib_ref.la" ) /bin/bash ../../libtool --tag=CC --mode=link gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -o sacctmgr account_functions.o archive_functions.o association_functions.o config_functions.o cluster_functions.o common.o event_functions.o federation_functions.o file_functions.o instance_functions.o runaway_job_functions.o job_functions.o reservation_functions.o resource_functions.o sacctmgr.o qos_functions.o txn_functions.o user_functions.o wckey_functions.o problem_functions.o tres_function.o -Wl,-rpath=/root/slurm-slurm-23-11-7-1/z/lib/slurm -L../../src/api/.libs -lslurmfull -export-dynamic -lreadline -lhistory lib_ref.la -lpthread -lm -lresolv libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 -fno-strict-aliasing -o sacctmgr account_functions.o archive_functions.o association_functions.o config_functions.o cluster_functions.o common.o event_functions.o federation_functions.o file_functions.o instance_functions.o runaway_job_functions.o job_functions.o reservation_functions.o resource_functions.o sacctmgr.o qos_functions.o txn_functions.o user_functions.o wckey_functions.o problem_functions.o tres_function.o -Wl,-rpath=/root/slurm-slurm-23-11-7-1/z/lib/slurm -Wl,--export-dynamic -L../../src/api/.libs /root/slurm-slurm-23-11-7-1/src/api/.libs/libslurmfull.a -lreadline -lhistory ./.libs/lib_ref.a -lpthread -lm -lresolv -pthread /usr/bin/ld: sacctmgr.o: warning: relocation against `_binary_usage_txt_end' in read-only section `.text' /usr/bin/ld: sacctmgr.o: in function `_usage': /root/slurm-slurm-23-11-7-1/src/sacctmgr/sacctmgr.c:926:(.text+0x1f): undefined reference to `_binary_usage_txt_start' /usr/bin/ld: /root/slurm-slurm-23-11-7-1/src/sacctmgr/sacctmgr.c:926:(.text+0x26): undefined reference to `_binary_usage_txt_end' /usr/bin/ld: warning: creating DT_TEXTREL in a PIE collect2: error: ld returned 1 exit status make[3]: *** [Makefile:672: sacctmgr] Error 1 make[3]: Leaving directory '/root/slurm-slurm-23-11-7-1/src/sacctmgr' make[2]: *** [Makefile:520: all-recursive] Error 1 make[2]: Leaving directory '/root/slurm-slurm-23-11-7-1/src' make[1]: *** [Makefile:621: all-recursive] Error 1 make[1]: Leaving directory '/root/slurm-slurm-23-11-7-1' make: *** [Makefile:520: all] Error 2 ----------------------------------------------------------------------------------------- I used these config options: ./configure --enable-debug --enable-pam --disable-sview --disable-shared --with-munge --with-json --with-yaml --with-http-parser --with-pmix --with-lz4 --with-hwloc --with-jwt --with-libcurl --with-freeipmi --with-rdkafka --with-bpf --prefix=/root/slurm-slurm-23-11-7-1/z

1 0

memory high water mark reporting
by Emyr James 22 May '24

22 May '24

Hi, We are trying out slurm having been running grid engine for a long while. In grid engine, the cgroups peak memory and max_rss are generated at the end of a job and recorded. It logs the information from the cgroup hierarchy as well as doing a getrusage call right at the end on the parent pid of the whole job "container" before cleaning up. With slurm it seems that the only way memory is recorded is by the acct gather polling. I am trying to add something in an epilog script to get the memory.peak but It looks like the cgroup hierarchy has been destroyed by the time the epilog is run. Where in the code is the cgroup hierarchy cleared up ? Is there no way to add something in so that the accounting is updated during the job cleanup process so that peak memory usage can be accurately logged ? I can reduce the polling interval from 30s to 5s but don't know if this causes a lot of overhead and in any case this seems to not be a sensible way to get values that should just be determined right at the end by an event rather than using polling. Many thanks, Emyr

5 10

Slurm version 23.11.7 is now available
by Marshall Garey 21 May '24

21 May '24

We are pleased to announce the availability of Slurm version 23.11.7. The 23.11.7 release fixes a few potential crashes in slurmctld when using less common options on job submission, slurmrestd compatibility with auth/slurm, and some additional minor and moderate severity bugs. Slurm can be downloaded from https://www.schedmd.com/downloads.php . -Marshall > -- slurmrestd - Correct OpenAPI specification for > 'GET /slurm/v0.0.40/jobs/state' having response as null. > -- Allow running jobs on overlapping partitions if jobs don't specify -s. > -- Fix segfault when requesting a shared gres along with an exclusive > allocation. > -- Fix regression in 23.02 where afternotok and afterok dependencies were > rejected for federated jobs not running on the origin cluster of the > submitting job. > -- slurmctld - Disable job table locking while job state cache is active when > replying to `squeue --only-job-state` or `GET /slurm/v0.0.40/jobs/state`. > -- Fix sanity check when setting tres-per-task on the job allocation as well as > the step. > -- slurmrestd - Fix compatiblity with auth/slurm. > -- Fix issue where TRESRunMins gets off correct value if using > QOS UsageFactor != 1. > -- slurmrestd - Require `user` and `association_condition` fields to be > populated for requests to 'POST /slurmdb/v0.0.40/users_association'. > -- Avoid a slurmctld crash with extra_constraints enabled when a job requests > certain invalid --extra values. > -- `scancel --ctld` and `DELETE /slurm/v0.0/40/jobs` - Fix support for job > array expressions (e.g. 1_[3-5]). Also fix signaling a single pending array > task (e.g. 1_10), which previously signaled the whole array job instead. > -- Fix a possible slurmctld segfault when at some point we failed to create an > external launcher step. > -- Allow the slurmctld to open a connection to the slurmdbd if the first > attempt fails due to a protocol error. > -- mpi/cray_shasta - Fix launch for non-het-steps within a hetjob. > -- sacct - Fix "gpuutil" TRES usage output being incorrect when using --units. > -- Fix a rare deadlock on slurmctld shutdown or reconfigure. > -- Fix issue that only left one thread on each core available when "CPUs=" is > configured to total thread count on multi-threaded hardware and no other > topology info ("Sockets=", "CoresPerSocket", etc.) is configured. > -- Fix the external launcher step not being allocated a VNI when requested. > -- jobcomp/kafka - Fix payload length when producing and sending a message. > -- scrun - Avoid a crash if RunTimeDelete is called before the container > finishes. > -- Save the slurmd's cred_state while reconfiguring to prevent the loss job > credentials.

1 0

2024

slurm-users ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users