December 2024 - slurm-users - lists.schedmd.com

slurmctld keeps segfaulting, possibly during or just after backfill
by Marcus Lauer 21 Jun '26

21 Jun '26

We are running into a problem where slurmctld is segfaulting a few times a day. We had this problem with SLURM 23.11.8 and now with 23.11.10 as well, though the problem only appears on one of the several SLURM clusters we have, and all of them use one of those versions of SLURM. I was wondering if anyone has encountered a similar issue and has any thoughts on how to prevent this. Obviously we use "SchedulerType=sched/backfill" but strangely when I switched to sched/builtin for a while there were still slurmctld segfaults. We also set "SchedulerParameters=enable_user_top,bf_max_job_test=2000". I have tried turning those off but it did not help. I have also tried tweaking several other settings to no avail. Most of the cluster runs Rocky Linux 8.10 (including the slurmctld system) though we still have some Scientific Linux 7.9 compute nodes (we compile SLURM separately for those). Here is the crash-time error from journalctl: Oct 02 06:31:20 our.host.name kernel: sched_agent[2048355]: segfault at 8 ip 00007fec755d7ea8 sp 00007fec6bffe7e8 error 4 in libslurmfull.so[7fec7555a000+1f4000] Oct 02 06:31:20 our.host.name kernel: Code: 48 39 c1 7e 19 48 c1 f8 06 ba 01 00 00 00 48 d3 e2 48 f7 da 48 0b 54 c6 10 48 21 54 c7 10 c3 b8 00 00 00 00 eb da 48 8b 4f 08 <48> 39 4e 08 48 0f 4e 4e 08 49 89 c9 48 83 f9 3f 76 4e ba 40 00 00 Oct 02 06:31:20 our.host.name systemd[1]: Started Process Core Dump (PID 2169426/UID 0). Oct 02 06:31:20 our.host.name systemd-coredump[2169427]: Process 2048344 ( slurmctld) of user 991 dumped core. This is followed by a list of each of the dozen or so related threads. The one which is dumping core is first and looks like this: Stack trace of thread 2048355: #0 0x00007fec755d7ea8 bit_and_not (libslurmfull.so) #1 0x000000000044531f _job_alloc (slurmctld) #2 0x000000000044576b _job_alloc_whole_node_internal (slurmctld) #3 0x0000000000446e6d gres_ctld_job_alloc_whole_node (slurmctld) #4 0x00007fec722e29b8 job_res_add_job (select_cons_tres.so) #5 0x00007fec722f7c32 select_p_select_nodeinfo_set (select_cons_tres.so) #6 0x00007fec756e7dc7 select_g_select_nodeinfo_set (libslurmfull.so) #7 0x0000000000496eb3 select_nodes (slurmctld) #8 0x0000000000480826 _schedule (slurmctld) #9 0x00007fec753421ca start_thread (libpthread.so.0) #10 0x00007fec745f78d3 __clone (libc.so.6) I have run slurmctld with "debug5" level logging and it appears that the error occurs right after backfill considers a large number of jobs. Slurmctld could be failing at the end of backfill or when doing something which happens just after backfill runs. Usually this is the last message before the crash: [2024-09-25T18:39:42.076] slurmscriptd: debug: _slurmscriptd_mainloop: finished If anyone has any thoughts or advice on this that would be appreciated. Thank you. -- Marcus Lauer Systems Administrator CETS Group, Research Support

4 4

slurmctld hourly: Unexpected missing socket error
by Jason Ellul 13 Jan '26

13 Jan '26

Hi all, I am hoping someone can help with our problem. Every hour after restarting slurmctld the controller becomes unresponsive to commands for 1 sec, reporting errors such as: [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934767]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934760]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934875]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934906]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[939016]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error It occurs consistently at around the hour mark, but generally not at other times, unless we run a reconfigure or restart the controller. We don’t see any issues in the slurmdbd.log and the errors are also always msg type RESPONSE. We have tried building a new server on different infrastructure, but the problem has persisted. Yesterday we even tried updating slurm to v24.05.1 in the hope that may provide a fix. During our troubleshooting we have: Set: * SchedulerParameters = max_rpc_cnt=400,sched_min_interval=50000,sched_max_job_start=300,batch_sched_delay=20,bf_resolution=600,bf_min_prio_reserve=2000,bf_min_age_reserve=600 * SlurmctldPort = 6808-6817 But although the stats in sdiag have improved we still see the errors. On our monitoring software we also see a drop in network and disk activity during this 1 second, always at approx. 1 hour after restarting the controller. Many Thanks in advance Jason Jason Ellul Head - Research Computing Facility Office of Cancer Research Peter MacCallum Cancer Centre

6 8

job_container/tmpfs and srun.
by Phill Harvey-Smith 17 Jan '25

17 Jan '25

Hi all, On our setup we are using job_container/tmpfs to give each job it's own temp space. Since our compute nodes have reasonably sized disks for tasks that do a lot of disk I/O on user's data we have asked users to copy their data to the local disk at the beginning of the task and (if needed) copy it back at the end. This saves lots of NFS thrashing slowing down both the task and the NFS servers. However some of our users are having problems with this, their initial sbatch script will create a temp directory in their private /tmp copy their data to it and then try to srun a program. The srun will fall over as it doesn't seem to have have access to the copied data. I suspect this is because the srun task is getting it's own private /tmp. So my question is, is there a way to have the srun task inherit the /tmp of the initial sbatch? I'll include a sample of the script our user is using below. If any further information is required please feel free to ask. Cheers. Phill. #!/usr/bin/bash #SBATCH --nodes 1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:00:10 #SBATCH --mem-per-cpu=3999 #SBATCH --output=script_out.log #SBATCH --error=script_error.log # The above options puts the STDOUT and STDERR of sbatch in # log files prefixed with 'script_'. # Create a randomly-named directory under /tmp jobtmpdir=$(mktemp -d) # Register a function to try and cleanup in case of job failure cleanup_handler() { echo "Cleaning up ${jobtmpdir}" rm -rf ${jobtmpdir} } trap 'cleanup_handler' SIGTERM EXIT # Change working directory to this directory cd ${jobtmpdir} # Copy the executable and input files from # where the job was submitted to the temporary directory. cp ${SLURM_SUBMIT_DIR}/a.out . cp ${SLURM_SUBMIT_DIR}/input.txt . # Run the executable, handling the collection of stdout # and stderr ourselves by redirecting to file srun ./a.out 2> task_error.log > task_out.log # Copy output data back to the submit directory. cp output.txt ${SLURM_SUBMIT_DIR} cp task_out.log ${SLURM_SUBMIT_DIR} cp task_error.log ${SLURM_SUBMIT_DIR} # Cleanup cd ${SLURM_SUBMIT_DIR} cleanup_handler

2 1

Permission denied for slurmdbd.conf
by sportlecon＠gmail.com 07 Jan '25

07 Jan '25

ls -ls /usr/local/slurm/etc/slurmdbd.conf 4 -rw------- 1 slurm slurm 497 Dec 28 16:34 /usr/local/slurm/etc/slurmdbd.conf sudo -u slurm /usr/local/slurm/sbin/slurmdbd -Dvvv slurmdbd: error: s_p_parse_file: unable to read "/usr/local/slurm/etc/slurmdbd.conf": Permission denied slurmdbd: fatal: Could not open/read/parse slurmdbd.conf file /usr/local/slurm/etc/slurmdbd.conf

3 2

Resource guarantees
by christopher.furbee＠jhuapl.edu 23 Dec '24

23 Dec '24

Hello, Long time SGE admin, new SLURM admin here. I recently started the transition of all my clusters from SGE to SLURM and everything was great until I hit the "Taco Bell" cluster (fake name). Taco Bell supports 4 projects and under SGE we had a priority system setup using projects to balance the queue. For the life of me I have been unable to replicate this in SLURM. We are looking to configure guaranteed resources based on the project. I had thought we could accomplish this with QOS and accounts but so far we have failed. What we would like to end up with is; When project Gordita is running uncontested 100% of the cluster is available. While Gordita is running, if Crunchwrap submits their jobs we want the scheduler to prioritize those jobs until a 75% Gordita, 25% Crunchwrap balance of jobs is reached. No preempting or priority overriding, just as a Gordita job finishes, if Crunchwrap is less than 25%, start a Crunchwrap job. And then maintain that balance until one of the projects jobs are 100% completed. Any assistance or guidance is greatly appreciated.

1 0

Slurm plugin for custom hardware allocation
by Laura Zharmukhametova 23 Dec '24

23 Dec '24

Hello, Is there an existing Slurm plugin for FPGA allocation? If not, can someone please point me in the right direction for how to approach it. Many thanks

2 1

sending mails wit smail on rocky9
by Marcus Wagner 19 Dec '24

19 Dec '24

Hi all, I have a problem with sending mails on rocky 9 via Slurm. One needs to install s-nail to have "/bin/mail" being available. There are some caveats in smail. In the second part (for the message, when the job began) one need to pipe ( eg echo "") into $MAIL, even in a script with no input, s-nail wants to be interactive. but it suffices to echo an empty text to snail. Nonetheless, I don't get any mail through. it seems the, mailprog for some reason gets killed or errors out for some other reason. While it is perfectionally working if run from the console :/ I all the time get in the slurmctld.log the following: 27212:[2024-12-19T15:54:54.935] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27213:[2024-12-19T15:54:54.945] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27214:[2024-12-19T15:54:54.945] error: MailProg returned error, it's output was '' 27395:[2024-12-19T15:55:55.540] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27396:[2024-12-19T15:55:55.551] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27397:[2024-12-19T15:55:55.551] error: MailProg returned error, it's output was '' 27438:[2024-12-19T15:56:55.981] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27439:[2024-12-19T15:56:55.981] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27440:[2024-12-19T15:56:55.992] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27441:[2024-12-19T15:56:55.992] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 9 27442:[2024-12-19T15:56:55.992] error: MailProg returned error, it's output was '' 27443:[2024-12-19T15:56:55.992] error: MailProg returned error, it's output was '' 27450:[2024-12-19T15:56:58.849] slurmscriptd: error: run_command: killing MailProg operation on shutdown 27451:[2024-12-19T15:56:58.859] slurmscriptd: _run_script: JobId=0 MailProg killed by signal 0 any hints? Best Marcus -- Dipl.-Inf. Marcus Wagner stellv. Gruppenleitung IT Center Gruppe: Server, Storage, HPC Abteilung: Systeme und Betrieb RWTH Aachen University Seffenter Weg 23 52074 Aachen Tel: +49 241 80 24383 wagner(a)itc.rwth-aachen.de www.itc.rwth-aachen.de Social-Media-Kanäle des IT Centers: https://blog.rwth-aachen.de/itc/ https://www.facebook.com/itcenterrwth https://www.linkedin.com/company/itcenterrwth https://twitter.com/ITCenterRWTH https://www.youtube.com/c/ITCenterRWTHAachen

1 0

lua, glib, gtk, kafka not found for slurm 24.11 and Alma 8
by Bernd Melchers 17 Dec '24

17 Dec '24

Dear all, i tried to rpmbuild slurm-24.11.0 for Alma Linux 8. Build failed because some installed Packages are not found by slurms configure script: rdkafka, glib, gtp and lua But all these packages are installed and they are found by slurm-24.05.x: librdkafka-1.6.1-1.el8.x86_64 librdkafka-devel-1.6.1-1.el8.x86_64 lua-5.3.4-12.el8.x86_64 lua-devel-5.3.4-12.el8.x86_64 glib2-2.56.4-165.el8_10.x86_64 glib2-devel-2.56.4-165.el8_10.x86_64 gtk2-2.24.32-5.el8.x86_64 gtk2-devel-2.24.32-5.el8.x86_64 gtk3-3.22.30-12.el8_10.x86_64 gtk3-devel-3.22.30-12.el8_10.x86_64 Mit freundlichen Grüßen Bernd Melchers -- Archiv- und Backup-Service | fab-service(a)zedat.fu-berlin.de Freie Universität Berlin | Tel. +49-30-838-55905

2 3

AllocNode:Sid in scontrol but not sacct?
by Chris Taylor 16 Dec '24

16 Dec '24

Does the accounting database keep this? Maybe I'm missing something but I don't see a way to query for it in sacct. Chris

1 1

job_container/tmpfs, TmpFS and TmpDisk usage
by Paul Musset 16 Dec '24

16 Dec '24

Hello, I have multiple questions about the usage of job_container/tmpfs, and the TmpFS and TmpDisk variables 1) If my job_container.conf files contains: ``` BasePath=/mnt/slurm_tmp Shared=true ``` is it important what I set TmpFS to in slurm.conf ? Should I set it to '/mnt/slurm_tmp' or '/tmp' ? 2) What size should I put in TmpDisk ? the size advertised by df ? 3) Finally, is there any recommended file system for the partition used as the job_container/tmpfs BasePath ? Best regards, Paul Musset Max Planck Institute for Brain Research

1 0