- slurm-users - lists.schedmd.com

Cluster ID changed in Slurm DB
by Burian, John 10 Jul '26

10 Jul '26

I was working on adding some partitions to our test cluster to test preemption. This is Slurm 24.11.1, we have a single slurmctld and slurmdbd in this cluster. I’m using partition based preemption, not QOS based preemption. I added a QOS to use with the preempting partition to enforce per-job resource limits, and then added the new partitions to slurm.conf. It took me many tries to get the slurm.conf correct, so many attempts to start slurmctld that failed due to conf file parsing errors. However, once I got the slurm.conf correct, slurmctld would not start, complaining that the cluster ID in the saved state information was wrong: [2026-07-02T14:55:38.573] fatal: CLUSTER ID MISMATCH. slurmctld has been started with "ClusterID=3013" from the state files in StateSaveLocation, but the DBD thinks it should be "3391". Running multiple clusters from a shared StateSaveLocation WILL CAUSE CORRUPTION. Remove /export/slurm/state/clustername to override this safety check if this is intentional. The clustername file in the save state location shows the cluster ID 3013. In the Slurm DB, in clusters_table, I see the row for my test cluster shows the (wrong) ID that appears in the logs, 3391, and the mod_time field has a timestamp in the same time frame as when I was doing the work on adding the QOS and partition configuration. Any thoughts on what could have caused the Slurm DB to be updated with a different cluster ID than what is saved in state? I’d like to avoid something similar happening when I go to deploy the new partitions in production. John

1 0

Expected behavior for preempt/partition_prio
by Massimo Sgaravatto 10 Jul '26

10 Jul '26

Dear all I have these settings in slurm.conf: . ... PreemptType=preempt/partition_prio PreemptMode=REQUEUE PartitionName=gpus-opp Nodes=btc-ter-gpu-[03-05],btc-dfa-gpu-06,btc-dfa-gpu-02 State=UP PriorityTier=15 PartitionName=mpicms Nodes=btc-ter-03,btc-ter-gpu-05 AllowAccounts=cms,operators State=UP PriorityTier=20 .... My understanding was that the expected behavior is that a job submitted to the gpus-opp partition is always preempted (requeued) if the resources allocated by this job are requested by a job submitted to the mpicms partition (since 20 > 15) Did I get it right? I am asking because I see that sometimes this doesn't happen. I am still trying to debug the issue, but it looks like also the QoS is taken into account (and this doesn't make sense to me unless I am missing something) Thanks, Massimo

3 3

Slinky 1.2.0 released
by Marlow Warnicke 02 Jul '26

02 Jul '26

We are pleased to announce the availability of Slinky version 1.2.0! Slinky is SchedMD (now part of NVIDIA)'s set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.2.0: https://github.com/SlinkyProject/slurm-operator/releases/tag/v1.2.0 New features include a ScheduledUpdate strategy for NodeSets to control when updates are applied, support for Google GKE A3 Mega instance types, and configurable pod security contexts for operator and webhook deployments. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge brings Kubernetes and Slurm jobs together under one scheduler. Both types of workloads share the same cluster resources. Release v1.2.0: https://github.com/SlinkyProject/slurm-bridge/releases/tag/v1.2.0 New features include support for LeaderWorkerSet workloads introduced in Kubernetes 1.37, priority annotations for workloads to control Slurm job scheduling order, and dynamic topology reconciliation for external and hybrid Slurm nodes. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… #kubernetes #slurm #dra #ai #training #inference #hpc #nvidia #gpu #cloudnative #slinky

1 0

Avoid killing GPU jobs during preemption
by Ratnasamy, Fritz 29 Jun '26

29 Jun '26

Hello, We have a recurring demand where some users would like to use idle GPU resources on periods where the usage is low. Their normal GPU/qos limits prevent this, which is expected as part of the fair share scheduler principles. A pattern we're considering is the preempt option on slurm to have these jobs being pre-empted when another user is submitting his jobs within his quota. The jobs that were submitted earlier by a faculty member taking advantage of available resources on top of his regular qos, will be preempted however, there is no suspend option for gpu jobs. There is only requeue, which kills the jobs to make the gpu available. I believe you can use suspend for cpu/mem without killing the job. I know that would require the users to set up checkpoints and re-start their jobs. The goal is to let heavy users fill otherwise idle nodes over low usage time without impacting smaller users when activity picks up. Has anyone implemented something similar? I am in general against preemption because that might lead to conflict between users. Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

2 1

ISC 2026 is here! Come see us in Hamburg.
by Victoria Hobson 22 Jun '26

22 Jun '26

The wait is over! ISC 2026 is officially underway in Hamburg, Germany, and we'd love to see you! As part of the newly expanded NVIDIA family, the Slurm team is on the floor and ready to connect. Swing by the Slurm booth (Z10) for sessions on GPU workload management, Slinky, and the latest releases and don't miss the NVIDIA booth (E30) for even more great content all week long. Here's what's on tap at booth Z10: Tuesday, June 23 12:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 17:00 – What's New: Latest Slurm and Slinky Release 19:00 – Slinky: Slurm for Kubernetes 101 Wednesday, June 24 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 14:00 – Slinky: Slurm for Kubernetes 101 15:00 – What's New: Latest Slurm and Slinky Release 17:00 – Redefining Resiliency for HPC Research and AI Factories: Slurm and NVIDIA Mission Control Thursday, June 25 10:00 – Slinky: Slurm for Kubernetes 101 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement Come say hello — we'll see you on the floor! Victoria Hobson | NVIDIA Campaign Manager, CUDA and Slurm

1 0

slurmctld keeps segfaulting, possibly during or just after backfill
by Marcus Lauer 21 Jun '26

21 Jun '26

We are running into a problem where slurmctld is segfaulting a few times a day. We had this problem with SLURM 23.11.8 and now with 23.11.10 as well, though the problem only appears on one of the several SLURM clusters we have, and all of them use one of those versions of SLURM. I was wondering if anyone has encountered a similar issue and has any thoughts on how to prevent this. Obviously we use "SchedulerType=sched/backfill" but strangely when I switched to sched/builtin for a while there were still slurmctld segfaults. We also set "SchedulerParameters=enable_user_top,bf_max_job_test=2000". I have tried turning those off but it did not help. I have also tried tweaking several other settings to no avail. Most of the cluster runs Rocky Linux 8.10 (including the slurmctld system) though we still have some Scientific Linux 7.9 compute nodes (we compile SLURM separately for those). Here is the crash-time error from journalctl: Oct 02 06:31:20 our.host.name kernel: sched_agent[2048355]: segfault at 8 ip 00007fec755d7ea8 sp 00007fec6bffe7e8 error 4 in libslurmfull.so[7fec7555a000+1f4000] Oct 02 06:31:20 our.host.name kernel: Code: 48 39 c1 7e 19 48 c1 f8 06 ba 01 00 00 00 48 d3 e2 48 f7 da 48 0b 54 c6 10 48 21 54 c7 10 c3 b8 00 00 00 00 eb da 48 8b 4f 08 <48> 39 4e 08 48 0f 4e 4e 08 49 89 c9 48 83 f9 3f 76 4e ba 40 00 00 Oct 02 06:31:20 our.host.name systemd[1]: Started Process Core Dump (PID 2169426/UID 0). Oct 02 06:31:20 our.host.name systemd-coredump[2169427]: Process 2048344 ( slurmctld) of user 991 dumped core. This is followed by a list of each of the dozen or so related threads. The one which is dumping core is first and looks like this: Stack trace of thread 2048355: #0 0x00007fec755d7ea8 bit_and_not (libslurmfull.so) #1 0x000000000044531f _job_alloc (slurmctld) #2 0x000000000044576b _job_alloc_whole_node_internal (slurmctld) #3 0x0000000000446e6d gres_ctld_job_alloc_whole_node (slurmctld) #4 0x00007fec722e29b8 job_res_add_job (select_cons_tres.so) #5 0x00007fec722f7c32 select_p_select_nodeinfo_set (select_cons_tres.so) #6 0x00007fec756e7dc7 select_g_select_nodeinfo_set (libslurmfull.so) #7 0x0000000000496eb3 select_nodes (slurmctld) #8 0x0000000000480826 _schedule (slurmctld) #9 0x00007fec753421ca start_thread (libpthread.so.0) #10 0x00007fec745f78d3 __clone (libc.so.6) I have run slurmctld with "debug5" level logging and it appears that the error occurs right after backfill considers a large number of jobs. Slurmctld could be failing at the end of backfill or when doing something which happens just after backfill runs. Usually this is the last message before the crash: [2024-09-25T18:39:42.076] slurmscriptd: debug: _slurmscriptd_mainloop: finished If anyone has any thoughts or advice on this that would be appreciated. Thank you. -- Marcus Lauer Systems Administrator CETS Group, Research Support

4 4

controller node in Ubuntu 22.04.5 LTS does not work with compute node in Debian GNU/Linux 12
by allspace＠qq.com 17 Jun '26

17 Jun '26

download slurm-26.05.1.tar.bz2 and build it on Ubuntu 22.04.5 LTS. Use it as controller node. download slurm-26.05.1.tar.bz2 and build it on Debian GNU/Linux 12. Use it as compute node(node-3). Run command "srun -w node-3 hostname" on controller node, got the following error message: srun: error: Task launch for StepId=10.0 failed on node node-3: Header lengths are longer than data received srun: error: Application launch failed: Header lengths are longer than data received srun: Job step aborted Here is the logs on node-3: [2026-06-16T12:45:20.645] error: _verify_signature: failed decode [2026-06-16T12:45:20.645] error: Malformed RPC of type REQUEST_LAUNCH_TASKS(6001) received [2026-06-16T12:45:20.653] error: slurm_unpack_msg_and_forward: [192.168.245.1:59308] failed: Header lengths are longer than data received [2026-06-16T12:45:20.663] error: wrap_on_data: [192.168.245.133:6818(fd:13)] on_data returned rc: Header lengths are longer than data received How can I fix this problem? Thanks

5 7

How to weight nodes properly based on RAM to prevent small instances clogging big instances with backfilling
by Xaver Stiensmeier 15 Jun '26

15 Jun '26

Dearl Slurm User List, I am currently reviewing a slurm.conf where the developer set Weight manually to attribute a greater weight to machines that have more RAM to force smaller jobs on smaller instances. However, I feel like there is something already in place or better than manually setting the weights, but I couldn't find it. If I understand correctly Slurm does not schedule jobs to the smallest possible node on default. So small jobs can be scheduled to large instances and a big job might have to wait indefinitely when using backfilling. I thought that Slurm does have mechanisms to prevent this but was unable to find it again in the documentation. Is there really no automatism at place or am I overlooking something? Best, Xaver

3 2

Memory requirements: seff vs jobstats
by Loris Bennett 15 Jun '26

15 Jun '26

Hi, For quite a few jobs which run on our system, 'seff' reports higher values for 'Memory Utilized' than 'jobstats' does via 'CPU memory usage' (see below for example of a job running on a single node). I was under the impression that 'seff' reports MaxRSS (from 'sacct'), whereas 'jobstats' reports 'Used' (from cgroups), which, as I understand it, is RSS plus page cache and possibly some kernel memory. Therefore, I would expect 'jobstats' to generally produce *larger* estimates of memory usage. However, the discrepancies I see seem always to be such that 'jobstats' report *lower* memory usage. Any thoughts on this? Cheers, Loris $ seff 25800851 Job ID: 25800851 Cluster: curta User/Group: foo/agbar State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 24 CPU Utilized: 10-09:05:31 CPU Efficiency: 39.05% of 26-13:51:36 core-walltime Job Wall-clock time: 1-02:34:39 Memory Utilized: 18.12 GB Memory Efficiency: 28.64% of 63.28 GB (2.64 GB/core) $ jobstats 25800851 ================================================================================ Slurm Job Statistics ================================================================================ Job ID: 25800851 User/Account: foo/agbar Job Name: foobar State: COMPLETED Nodes: 1 CPU Cores: 24 CPU Memory: 64800MB (2.7GB per CPU-core) QOS/Partition: standard/main Cluster: curta Start Time: Mon Jun 8, 2026 at 3:53 PM Run Time: 1-02:34:39 Time Limit: 5-00:00:00 Overall Utilization ================================================================================ CPU utilization [||||||||||||||||||| 39%] CPU memory usage [||||| 11%] Detailed Utilization ================================================================================ CPU utilization per node (CPU time used/run time) c036.curta.zedat.fu-berlin.de: 10-09:03:12/26-13:51:36 (efficiency=39.0%) CPU memory usage per node - used/allocated c036.curta.zedat.fu-berlin.de: 7.1GB/63.3GB (304.3MB/2.6GB per core of 24) ... -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

1 0

Configurator tool on Slurm site is not found
by Valerio Bellizzomi 11 Jun '26

11 Jun '26

Please check. VB

1 0