October 2025 - slurm-users - lists.schedmd.com

slurmctld keeps segfaulting, possibly during or just after backfill
by Marcus Lauer 21 Jun '26

21 Jun '26

We are running into a problem where slurmctld is segfaulting a few times a day. We had this problem with SLURM 23.11.8 and now with 23.11.10 as well, though the problem only appears on one of the several SLURM clusters we have, and all of them use one of those versions of SLURM. I was wondering if anyone has encountered a similar issue and has any thoughts on how to prevent this. Obviously we use "SchedulerType=sched/backfill" but strangely when I switched to sched/builtin for a while there were still slurmctld segfaults. We also set "SchedulerParameters=enable_user_top,bf_max_job_test=2000". I have tried turning those off but it did not help. I have also tried tweaking several other settings to no avail. Most of the cluster runs Rocky Linux 8.10 (including the slurmctld system) though we still have some Scientific Linux 7.9 compute nodes (we compile SLURM separately for those). Here is the crash-time error from journalctl: Oct 02 06:31:20 our.host.name kernel: sched_agent[2048355]: segfault at 8 ip 00007fec755d7ea8 sp 00007fec6bffe7e8 error 4 in libslurmfull.so[7fec7555a000+1f4000] Oct 02 06:31:20 our.host.name kernel: Code: 48 39 c1 7e 19 48 c1 f8 06 ba 01 00 00 00 48 d3 e2 48 f7 da 48 0b 54 c6 10 48 21 54 c7 10 c3 b8 00 00 00 00 eb da 48 8b 4f 08 <48> 39 4e 08 48 0f 4e 4e 08 49 89 c9 48 83 f9 3f 76 4e ba 40 00 00 Oct 02 06:31:20 our.host.name systemd[1]: Started Process Core Dump (PID 2169426/UID 0). Oct 02 06:31:20 our.host.name systemd-coredump[2169427]: Process 2048344 ( slurmctld) of user 991 dumped core. This is followed by a list of each of the dozen or so related threads. The one which is dumping core is first and looks like this: Stack trace of thread 2048355: #0 0x00007fec755d7ea8 bit_and_not (libslurmfull.so) #1 0x000000000044531f _job_alloc (slurmctld) #2 0x000000000044576b _job_alloc_whole_node_internal (slurmctld) #3 0x0000000000446e6d gres_ctld_job_alloc_whole_node (slurmctld) #4 0x00007fec722e29b8 job_res_add_job (select_cons_tres.so) #5 0x00007fec722f7c32 select_p_select_nodeinfo_set (select_cons_tres.so) #6 0x00007fec756e7dc7 select_g_select_nodeinfo_set (libslurmfull.so) #7 0x0000000000496eb3 select_nodes (slurmctld) #8 0x0000000000480826 _schedule (slurmctld) #9 0x00007fec753421ca start_thread (libpthread.so.0) #10 0x00007fec745f78d3 __clone (libc.so.6) I have run slurmctld with "debug5" level logging and it appears that the error occurs right after backfill considers a large number of jobs. Slurmctld could be failing at the end of backfill or when doing something which happens just after backfill runs. Usually this is the last message before the crash: [2024-09-25T18:39:42.076] slurmscriptd: debug: _slurmscriptd_mainloop: finished If anyone has any thoughts or advice on this that would be appreciated. Thank you. -- Marcus Lauer Systems Administrator CETS Group, Research Support

4 4

Optimizing CPU allocation in Slurm with hyperthreading enabled
by Guillaume COCHARD 02 Mar '26

02 Mar '26

Hello, The Slurm documentation states that "The default allocation method within a node is cyclic allocation (allocate available CPUs in a round-robin fashion across the sockets within a node)." In our case, we are using hyperthreading, which means that if two jobs requesting one CPU each arrive at the same time on an empty node, they will be assigned to the two threads of the same physical core. To improve performance, we would like Slurm to allocate CPUs (in our case, threads) in a way that prioritizes using one thread per physical core first, and only once all cores are occupied, assign the second thread of each core. This would be similar to the Least Loaded Node (LLN) approach, but instead of distributing jobs across nodes, it would distribute them across physical cores (Least Loaded Core). We do not want to disable hyperthreading; we simply want to optimize core usage when a machine is not fully loaded. We also want this to be a cluster-wide setting, and not a user and/or job setting. Our current slurm.conf contains: ConstrainCores = no SelectType = select/cons_tres SelectTypeParameters = CR_CPU_MEMORY And our cgroup.conf contains: ConstrainDevices=yes ConstrainCores=yes Does anyone know how to achieve this? Thanks, Guillaume

3 2

slurmctld hourly: Unexpected missing socket error
by Jason Ellul 13 Jan '26

13 Jan '26

Hi all, I am hoping someone can help with our problem. Every hour after restarting slurmctld the controller becomes unresponsive to commands for 1 sec, reporting errors such as: [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934767]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934760]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934875]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934906]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[939016]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error It occurs consistently at around the hour mark, but generally not at other times, unless we run a reconfigure or restart the controller. We don’t see any issues in the slurmdbd.log and the errors are also always msg type RESPONSE. We have tried building a new server on different infrastructure, but the problem has persisted. Yesterday we even tried updating slurm to v24.05.1 in the hope that may provide a fix. During our troubleshooting we have: Set: * SchedulerParameters = max_rpc_cnt=400,sched_min_interval=50000,sched_max_job_start=300,batch_sched_delay=20,bf_resolution=600,bf_min_prio_reserve=2000,bf_min_age_reserve=600 * SlurmctldPort = 6808-6817 But although the stats in sdiag have improved we still see the errors. On our monitoring software we also see a drop in network and disk activity during this 1 second, always at approx. 1 hour after restarting the controller. Many Thanks in advance Jason Jason Ellul Head - Research Computing Facility Office of Cancer Research Peter MacCallum Cancer Centre

6 8

Slurm release candidate version 25.11.0rc1 is available for testing
by Tim Wickberg 28 Oct '25

28 Oct '25

We are pleased to announce the availability of Slurm release candidate 25.11.0rc1. To highlight some new features coming in 25.11: * Added new "Expedited Requeue" mode for batch jobs. Batch jobs with --requeue=expedite will automatically requeue on node failure, or if the batch script returns a non-zero exit code and one or more Epilog scripts fail. Expedited requeue jobs are eligible to restart immediately, are treated as the highest priority job in the system, and their previously allocated set of nodes will be prevented from launching other work. * Added a new "Mode 3" of operation to Hierarchical Resources. This mode complements the existing Mode 1 and Mode 2 by summing usage from lower levels automatically. This can be used, e.g., to implement a power-capping mode modeling power distribution between the datacenter, local distribution, and individual racks. * Added direct support for exporting OpenMetrics (Prometheus) telemetry from slurmctld. This is accessible on SlurmctldPort on SlurmctldHost by default, or can be disabled if desired. * Added an experimental asynchronous-reply mode to slurmctld. If enabled with "SlurmctldParameters=enable_async_reply", RPC responses are offloaded to the kernel for further processing, freeing individual worker threads for new traffic. This is the first release candidate of the upcoming 25.11 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 25.11.x version and we will address them before the first production 25.11.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/download-slurm/. The changelog for 25.11 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-25.11.md -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

AuthInfo broken in 25.05.1 ?
by michael＠mayer.cx 27 Oct '25

27 Oct '25

I am using the AuthInfo in both slurm.conf and slurmdbd.conf to point to a different location for the munge socket. This works as expected in slurmdbd, but it does not work for slurmctld. In slurmctld it seems to ignore this setting and try to locate the munge socket at the default location and then fail. When using the same approach in 23.11.11, for example, AuthInfo is working as expected for both slurmctld and slurmdbd. Sample config from slurm.conf and slurmdbd.conf AuthType=auth/munge AuthInfo=socket=/my/non-default/location/of/munge.socket I'd be grateful for any pointer whether this is a genuine bug/regression or not. Many thanks, Michael.

3 3

Possible to require jobs to use NVLink-ed pairs of GPUs?
by Marcus Lauer 27 Oct '25

27 Oct '25

One of our researchers asked whether it was possible to require a job to use NVLink-ed pairs of GPUs. I see that there is a support ticket on the SchedMD site which covers this (https://support.schedmd.com/show_bug.cgi?id=15995) That ticket is a few years old though. Does anyone happen to know whether support for this has been added in newer releases of SLURM? The cluster in question does use "AutoDetect=nvml" in its gres.conf and the output of "slurmd -G" shows that SLURM is aware of the NVLink pairs. I assume the scheduler is trying to use that information. What I want to know is whether there is some way for an end-user to add a constraint (for example) to a job such that it only runs on an NVLink-ed pair of GPUs. I do know that there are other ways to implement this such as requiring jobs to run with even numbers of GPUs, perhaps just on some nodes to allow single GPU jobs to run on the remaining nodes. I'm specifically asking about a flag or setting a user could apply to their jobs. If there is such a thing maybe someone here knows about it. If so I'd love to hear about it. Thanks! -- *Mr. Marcus Lauer* Systems Administrator Penn Engineering University of Pennsylvania https://www.seas.upenn.edu/

1 0

Limit number of allocated GPUs
by Gestió Servidors 23 Oct '25

23 Oct '25

Hello, I have three nodes, serving each one 2 GPUs. I would like to limit (qos??) that a user could user only one GPU from earch server, but user could user simultaneously three GPUs if each GPU belongs to different servers. With this QoS "sacctmgr add qos test-limit-GPUs MaxJobsPerUser=3 MaxTRESPerUser=gres/gpu=1" I can limit to one GPU, but then user can't run other job in a GPU from other server. How must I configure QoS (or other method) to allow more than one job requesting GPUs but never in the same server? Thanks.

3 2

Job remains "PENDING" with reason "QOSMaxGRESPerUser"
by Gestió Servidors 23 Oct '25

23 Oct '25

Hello, I have added a new "qos" with these parameters: sacctmgr add qos test-GPUs MaxJobsPerUser=6 MaxTRESPerUser=gres/gpu=1 MaxSubmitJobsPerUser=25. With it, I only allow 6 running jobs per user, a total of 25 pending+running job per user and only 1 GPU. I have applied this qos directly to a partition in slurm.conf. When a user submits to that partition requesting 2 or more GPUs, job remains "PD" (pending) and notifies "QOSMaxGRESPerUser" in NODELIST column, but I would like to know if it would be possible to direcly reject job and avoid that job remains at queue? For example, if I submit 50 jobs, after number 25 I get message "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) sbatch: error: QOSMaxSubmitJobPerUserLimit" 25 times) Thanks.

2 1

dynamic node slurm node list order
by Assmann, Greta Marie 23 Oct '25

23 Oct '25

Hello, we would like to understand how the internal SLURM node list order works. More detailed info: Our setup: We have a Slurm cluster with N dynamic nodes (heterogeneous node types) in a partition and we observe that if all resources are free, jobs always get submitted to one specific node first. This node is also the node that turns up at the top of the list when doing an $ scontrol show nodes . As we do not have a defined node list in slurm.conf as it is done for non-dynamic nodes, we were wondering how the order is set up. Interestingly, when deleting (and unregistering) this node from the cluster and re-registering it again, the node is still at the same position when doing scontrol show nodes. Is there some internal node list order caching or similar? How is the node list order defined ? Thanks , Greta Dr. Greta Assmann Data Analysis and Research Infrastructure OBBA 230 Forschungsstrasse 111 5232 Villigen-PSI email: greta.assmann(a)psi.ch

4 5

Slurm and Slinky events this fall, update events page
by Tim Wickberg 14 Oct '25

14 Oct '25

We'll have a bit more details as conference season quickly approaches this November, but SchedMD staff are presenting at KubeCon NA on Slinky [1]. We'll be manning the Slurm Booth at SC25 [2], as well as hosting the annual Slurm Community Birds-of-a-Feather session [3]. I'll also send a link out to the survey questions for the BoF to the slurm-users list ahead of the conference, and we'll be going into a bit more depth on the answers during the BoF this year. The events page on the SchedMD website has more detail on future events as well: https://www.schedmd.com/events/ A few folks had asked, and apparently we never mentioned this more publicly, but: SchedMD does not plan to hold an in-person SLUG in 2025 or 2026. We are working to bring some of the same content to our YouTube channel [4] as a way to more broadly disseminate some of that same content, starting with the Slurm 25.11 release overview in December. - Tim [1] https://kccncna2025.sched.com/event/27FW5/ [2] The Slurm Booth is #1641. [3] https://sc25.conference-program.com/presentation/?id=bof101&sess=sess471 [4] https://www.youtube.com/SchedMDSlurm -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0