November 2025 - slurm-users - lists.schedmd.com

slurmctld keeps segfaulting, possibly during or just after backfill
by Marcus Lauer 21 Jun '26

21 Jun '26

We are running into a problem where slurmctld is segfaulting a few times a day. We had this problem with SLURM 23.11.8 and now with 23.11.10 as well, though the problem only appears on one of the several SLURM clusters we have, and all of them use one of those versions of SLURM. I was wondering if anyone has encountered a similar issue and has any thoughts on how to prevent this. Obviously we use "SchedulerType=sched/backfill" but strangely when I switched to sched/builtin for a while there were still slurmctld segfaults. We also set "SchedulerParameters=enable_user_top,bf_max_job_test=2000". I have tried turning those off but it did not help. I have also tried tweaking several other settings to no avail. Most of the cluster runs Rocky Linux 8.10 (including the slurmctld system) though we still have some Scientific Linux 7.9 compute nodes (we compile SLURM separately for those). Here is the crash-time error from journalctl: Oct 02 06:31:20 our.host.name kernel: sched_agent[2048355]: segfault at 8 ip 00007fec755d7ea8 sp 00007fec6bffe7e8 error 4 in libslurmfull.so[7fec7555a000+1f4000] Oct 02 06:31:20 our.host.name kernel: Code: 48 39 c1 7e 19 48 c1 f8 06 ba 01 00 00 00 48 d3 e2 48 f7 da 48 0b 54 c6 10 48 21 54 c7 10 c3 b8 00 00 00 00 eb da 48 8b 4f 08 <48> 39 4e 08 48 0f 4e 4e 08 49 89 c9 48 83 f9 3f 76 4e ba 40 00 00 Oct 02 06:31:20 our.host.name systemd[1]: Started Process Core Dump (PID 2169426/UID 0). Oct 02 06:31:20 our.host.name systemd-coredump[2169427]: Process 2048344 ( slurmctld) of user 991 dumped core. This is followed by a list of each of the dozen or so related threads. The one which is dumping core is first and looks like this: Stack trace of thread 2048355: #0 0x00007fec755d7ea8 bit_and_not (libslurmfull.so) #1 0x000000000044531f _job_alloc (slurmctld) #2 0x000000000044576b _job_alloc_whole_node_internal (slurmctld) #3 0x0000000000446e6d gres_ctld_job_alloc_whole_node (slurmctld) #4 0x00007fec722e29b8 job_res_add_job (select_cons_tres.so) #5 0x00007fec722f7c32 select_p_select_nodeinfo_set (select_cons_tres.so) #6 0x00007fec756e7dc7 select_g_select_nodeinfo_set (libslurmfull.so) #7 0x0000000000496eb3 select_nodes (slurmctld) #8 0x0000000000480826 _schedule (slurmctld) #9 0x00007fec753421ca start_thread (libpthread.so.0) #10 0x00007fec745f78d3 __clone (libc.so.6) I have run slurmctld with "debug5" level logging and it appears that the error occurs right after backfill considers a large number of jobs. Slurmctld could be failing at the end of backfill or when doing something which happens just after backfill runs. Usually this is the last message before the crash: [2024-09-25T18:39:42.076] slurmscriptd: debug: _slurmscriptd_mainloop: finished If anyone has any thoughts or advice on this that would be appreciated. Thank you. -- Marcus Lauer Systems Administrator CETS Group, Research Support

4 4

Optimizing CPU allocation in Slurm with hyperthreading enabled
by Guillaume COCHARD 02 Mar '26

02 Mar '26

Hello, The Slurm documentation states that "The default allocation method within a node is cyclic allocation (allocate available CPUs in a round-robin fashion across the sockets within a node)." In our case, we are using hyperthreading, which means that if two jobs requesting one CPU each arrive at the same time on an empty node, they will be assigned to the two threads of the same physical core. To improve performance, we would like Slurm to allocate CPUs (in our case, threads) in a way that prioritizes using one thread per physical core first, and only once all cores are occupied, assign the second thread of each core. This would be similar to the Least Loaded Node (LLN) approach, but instead of distributing jobs across nodes, it would distribute them across physical cores (Least Loaded Core). We do not want to disable hyperthreading; we simply want to optimize core usage when a machine is not fully loaded. We also want this to be a cluster-wide setting, and not a user and/or job setting. Our current slurm.conf contains: ConstrainCores = no SelectType = select/cons_tres SelectTypeParameters = CR_CPU_MEMORY And our cgroup.conf contains: ConstrainDevices=yes ConstrainCores=yes Does anyone know how to achieve this? Thanks, Guillaume

3 2

slurmctld hourly: Unexpected missing socket error
by Jason Ellul 13 Jan '26

13 Jan '26

Hi all, I am hoping someone can help with our problem. Every hour after restarting slurmctld the controller becomes unresponsive to commands for 1 sec, reporting errors such as: [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934767]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934760]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934875]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934906]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[939016]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error It occurs consistently at around the hour mark, but generally not at other times, unless we run a reconfigure or restart the controller. We don’t see any issues in the slurmdbd.log and the errors are also always msg type RESPONSE. We have tried building a new server on different infrastructure, but the problem has persisted. Yesterday we even tried updating slurm to v24.05.1 in the hope that may provide a fix. During our troubleshooting we have: Set: * SchedulerParameters = max_rpc_cnt=400,sched_min_interval=50000,sched_max_job_start=300,batch_sched_delay=20,bf_resolution=600,bf_min_prio_reserve=2000,bf_min_age_reserve=600 * SlurmctldPort = 6808-6817 But although the stats in sdiag have improved we still see the errors. On our monitoring software we also see a drop in network and disk activity during this 1 second, always at approx. 1 hour after restarting the controller. Many Thanks in advance Jason Jason Ellul Head - Research Computing Facility Office of Cancer Research Peter MacCallum Cancer Centre

6 8

Invalid generic resource (gres) specification after RMA
by Lee 04 Dec '25

04 Dec '25

Good afternoon, I have a cluster that is managed by Base Command Manager (v10) and it has several Nvidia DGXs. dgx09 is a problem child. The entire node was RMA'd and now it no longer behaves the same as my other DGXs. I think the below symptoms are caused by a single underlying issue. *Symptoms : * 1. When I look at our 8 non-MIG DGXs, via `scontrol show node=dgxXY | grep Gres`, 7/8 DGXs report "Gres=gpu:*H100*:8(S:0-1)" while dgx09 reports " Gres=gpu:*h100*:8(S:0-1)" 2. When I submit a job to this node, I get : $ srun --reservation=g09_test --gres=gpu:2 --pty bash srun: error: Unable to create step for job 105035: Invalid generic resource (gres) specification ### No job is running on the node, yet AllocTRES shows consumed resources... $ scontrol show node=dgx09 | grep -i AllocTRES *AllocTRES=gres/gpu=2* ### dgx09 : /var/log/slurmd contains no information ### slurmctld shows : root@h01:# grep 105035 /var/log/slurmctld [2025-11-13T07:44:56.380] sched: _slurm_rpc_allocate_resources JobId=105035 NodeList=dgx09 usec=3420 [2025-11-13T07:44:56.482] _job_complete: JobId=105035 WTERMSIG 1 [2025-11-13T07:44:56.483] _job_complete: JobId=105035 done *Configuration : * 1. gres.conf : # This section of this file was automatically generated by cmd. Do not edit manually! # BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE AutoDetect=NVML NodeName=dgx[01,02] Name=gpu Type=1g.20gb Count=32 AutoDetect=NVML NodeName=dgx[03-10] Name=gpu Type=h100 Count=8 AutoDetect=NVML # END AUTOGENERATED SECTION -- DO NOT REMOVE 2. grep NodeName slurm.conf root@h01:# grep NodeName slurm.conf NodeName=dgx[01,02] RealMemory=2063937 Boards=1 SocketsPerBoard=2 CoresPerSocket=56 ThreadsPerCore=2 MemSpecLimit=30017 Gres=gpu:1g.20gb:32 Feature=location=local NodeName=dgx03-10] RealMemory=2063937 Boards=1 SocketsPerBoard=2 CoresPerSocket=56 ThreadsPerCore=2 MemSpecLimit=30017 Gres=gpu:H100:8 Feature=location=local 3. What slurmd detects on dgx09 root@dgx09:~# slurmd -C NodeName=dgx09 CPUs=224 Boards=1 SocketsPerBoard=2 CoresPerSocket=56 ThreadsPerCore=2 RealMemory=2063937 UpTime=8-00:39:10 root@dgx09:~# slurmd -G slurmd: gpu/nvml: _get_system_gpu_list_nvml: 8 GPU system device(s) detected slurmd: Gres Name=gpu Type=h100 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-55 CoreCnt=224 Links=-1,0,0,0,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-55 CoreCnt=224 Links=0,-1,0,0,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0-55 CoreCnt=224 Links=0,0,-1,0,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=0-55 CoreCnt=224 Links=0,0,0,-1,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=56-111 CoreCnt=224 Links=0,0,0,0,-1,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=5 ID=7696487 File=/dev/nvidia5 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,-1,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=6 ID=7696487 File=/dev/nvidia6 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,0,-1,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=7 ID=7696487 File=/dev/nvidia7 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,0,0,-1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML *Questions : * 1. As far as I can tell, dgx09 is identical to all my non-MIG DGX nodes in terms of configuration and hardware. Why does scontrol report it having 'h100' with a lower case 'h' unlike the other dgxs which report with an upper case 'H'? 2. Why is dgx09 not accepting GPU jobs and afterwards it artificially thinks that there are GPUs allocated even though no jobs are on the node? 3. Are there additional tests / configurations that I can do to probe the differences between dgx09 and all my other nodes? Best regards, Lee

6 12

MIG and eBPF issues (Slurm 24.11.6)
by wk5ng＠uwaterloo.ca 28 Nov '25

28 Nov '25

Hi all, I'm having some trouble getting Slurm 24.11.6 to work with MIG, and the slurmd logs seem to point to an issue with eBPF. For some context, this is an LXD unprivileged container where I'm trying to get MIG to work with Slurm. Other compute nodes without MIG work fine and isolate the GPUs accordingly. What I'm seeing in slurmd logs: [2025-11-24T23:32:50.197] [331.interactive] cgroup/v2: cgroup_p_constrain_apply: CGROUP: EBPF Closing and loading bpf program into /sys/fs/cgroup/system.slice/slurmstepd.scope/job_331 [2025-11-24T23:32:50.197] [331.interactive] error: load_ebpf_prog: BPF load error (Operation not permitted). Please check your system limits (MEMLOCK). I've tried increasing the system limits for MEMLOCK by setting DefaultLimitMEMLOCK=infinity in /etc/systemd/system.conf, and I've copied my slurmd.service file below where I've set Delegate=yes and LimitMEMLOCK=infinity. Previously only Delegate=yes wasn't set (I had rifled through the cgroupv2 documentaton for Slurm and found that setting), but in both cases I see the same BPF load error. Just wondering if this was something that other people had come across before and maybe I'm doing something silly here. I've checked that my slurm.conf has the corresponding parameters set according to Slurm's own documentation for cgroup.conf and my cgroup.conf is also copied below. Some portion of the gres.conf is also copied below, and even though I tried AutoDetect=nvml for this node, it's still doesn't work, which was why I changed to manually setting it based off the output of slurmd -G. Maybe I should try switching back to cgroupv1 and see if that helps fix things, but I'm not sure at this point if MIG and Slurm are compatible using cgroupv1. I can send other parts of logs, configuration files etc. Any help would be greatly appreciated! ###### slurmd.service file [Unit] Description=Slurm node daemon After=network.target munge.service ConditionPathExists=/etc/slurm/slurm.conf [Service] Type=forking EnvironmentFile=-/etc/sysconfig/slurmd ExecStart=/usr/sbin/slurmd -d /usr/sbin/slurmstepd $SLURMD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID PIDFile=/var/run/slurmd.pid KillMode=process LimitNOFILE=51200 Delegate=yes LimitMEMLOCK=infinity LimitSTACK=infinity [Install] WantedBy=multi-user.target ###### cgroup.conf CgroupPlugin=autodetect ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ##### gres.conf NodeName=gpu-3 AutoDetect=nvml Name=gpu NodeName=gpu-4 Name=gpu MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap30,/dev/nvidia-caps/nvidia-cap31

3 4

Slinky version 1.0.0 is now available
by Marrlow Warnicke 20 Nov '25

20 Nov '25

We are pleased to announce the availability of Slinky version 1.0.0! Slinky is SchedMD's set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.0.0: https://github.com/SlinkyProject/slurm-operator/tree/release-1.0 New features include hybrid support, integration points for external tooling, and workload protection and isolation. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge handles the cases where you want Slurm scheduling on your cluster and to be able to run either Kubernetes or Slurm jobs. Release v1.0.0: https://github.com/SlinkyProject/slurm-bridge/tree/release-1.0 New features include support for DRA Extended Resources, support for TaintToleration and VolumeBinding plugins, and integration of new Slurm 25.11 support for granular node resource allocation assignments with GRES. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… The SlinkyProject registry now has containers that support both amd64 (x86_64) and arm64 (aarch64) architectures. You may find these here: https://github.com/orgs/SlinkyProject/packages Apologies for the redundant email-we needed to clarify the release version. -- Marlow Warnicke Principal Cloud Engineer Commercial Slurm - and Slinky - Development and Support

1 0

Slinky version 1.0.0 is now available
by Marrlow Warnicke 20 Nov '25

20 Nov '25

We are pleased to announce the availability of Slinky version 1.0.0-rc1. Slinky is SchedMD’s set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-operator/tree/release-1.0 New features include hybrid support, integration points for external tooling, and workload protection and isolation. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge handles the cases where you want Slurm scheduling on your cluster and to be able to run either Kubernetes or Slurm jobs. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-bridge/tree/release-1.0 New features include support for DRA Extended Resources, support for TaintToleration and VolumeBinding plugins, and integration of new Slurm 25.11 support for granular node resource allocation assignments with GRES. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… The SlinkyProject registry now has containers that support both amd64 (x86_64) and arm64 (aarch64) architectures. You may find these here: https://github.com/orgs/SlinkyProject/packages -- Marlow Warnicke Principal Cloud Engineer, SchedMD LLC Commercial Slurm - and Slinky - Development and Support

1 0

Recommended Stable Slurm Version for >100P Scale Clusters
by KK 18 Nov '25

18 Nov '25

We are currently planning to deploy a new HPC system with a total compute capacity exceeding 100 PF. As part of our preparation, we would like to understand which Slurm versions are considered stable and widely used at this scale. Could you please share your recommendations or experience regarding: 1. Which Slurm version is currently running reliably on very large-scale clusters (>100 PF or >10k nodes)? 2. Whether there are any versions we should avoid due to known issues at large scale. 3. Any best practices or configuration considerations for Slurm deployments of this size.

3 2

Slurm versions 25.05.5 and 24.11.7 are now available
by Marshall Garey 11 Nov '25

11 Nov '25

We are pleased to announce the availability of Slurm versions 25.05.5 and 24.11.7. These releases fix some mild to moderate severity issues and potential crashes. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md https://github.com/SchedMD/slurm/blob/slurm-24.11/CHANGELOG/slurm-24.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Split node into 2 partitions
by Ratnasamy, Fritz 11 Nov '25

11 Nov '25

Hi, Is there a straightforward way to split a gpu node with 8 GPUs into 2 partitions with 4 GPUs on each? How do we edit the slurm.conf and gres.conf in this case? Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

3 2