- slurm-users - lists.schedmd.com

Avoid killing GPU jobs during preemption
by Ratnasamy, Fritz 26 Jun '26

26 Jun '26

Hello, We have a recurring demand where some users would like to use idle GPU resources on periods where the usage is low. Their normal GPU/qos limits prevent this, which is expected as part of the fair share scheduler principles. A pattern we're considering is the preempt option on slurm to have these jobs being pre-empted when another user is submitting his jobs within his quota. The jobs that were submitted earlier by a faculty member taking advantage of available resources on top of his regular qos, will be preempted however, there is no suspend option for gpu jobs. There is only requeue, which kills the jobs to make the gpu available. I believe you can use suspend for cpu/mem without killing the job. I know that would require the users to set up checkpoints and re-start their jobs. The goal is to let heavy users fill otherwise idle nodes over low usage time without impacting smaller users when activity picks up. Has anyone implemented something similar? I am in general against preemption because that might lead to conflict between users. Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

1 0

ISC 2026 is here! Come see us in Hamburg.
by Victoria Hobson 22 Jun '26

22 Jun '26

The wait is over! ISC 2026 is officially underway in Hamburg, Germany, and we'd love to see you! As part of the newly expanded NVIDIA family, the Slurm team is on the floor and ready to connect. Swing by the Slurm booth (Z10) for sessions on GPU workload management, Slinky, and the latest releases and don't miss the NVIDIA booth (E30) for even more great content all week long. Here's what's on tap at booth Z10: Tuesday, June 23 12:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 17:00 – What's New: Latest Slurm and Slinky Release 19:00 – Slinky: Slurm for Kubernetes 101 Wednesday, June 24 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 14:00 – Slinky: Slurm for Kubernetes 101 15:00 – What's New: Latest Slurm and Slinky Release 17:00 – Redefining Resiliency for HPC Research and AI Factories: Slurm and NVIDIA Mission Control Thursday, June 25 10:00 – Slinky: Slurm for Kubernetes 101 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement Come say hello — we'll see you on the floor! Victoria Hobson | NVIDIA Campaign Manager, CUDA and Slurm

1 0

slurmctld keeps segfaulting, possibly during or just after backfill
by Marcus Lauer 21 Jun '26

21 Jun '26

We are running into a problem where slurmctld is segfaulting a few times a day. We had this problem with SLURM 23.11.8 and now with 23.11.10 as well, though the problem only appears on one of the several SLURM clusters we have, and all of them use one of those versions of SLURM. I was wondering if anyone has encountered a similar issue and has any thoughts on how to prevent this. Obviously we use "SchedulerType=sched/backfill" but strangely when I switched to sched/builtin for a while there were still slurmctld segfaults. We also set "SchedulerParameters=enable_user_top,bf_max_job_test=2000". I have tried turning those off but it did not help. I have also tried tweaking several other settings to no avail. Most of the cluster runs Rocky Linux 8.10 (including the slurmctld system) though we still have some Scientific Linux 7.9 compute nodes (we compile SLURM separately for those). Here is the crash-time error from journalctl: Oct 02 06:31:20 our.host.name kernel: sched_agent[2048355]: segfault at 8 ip 00007fec755d7ea8 sp 00007fec6bffe7e8 error 4 in libslurmfull.so[7fec7555a000+1f4000] Oct 02 06:31:20 our.host.name kernel: Code: 48 39 c1 7e 19 48 c1 f8 06 ba 01 00 00 00 48 d3 e2 48 f7 da 48 0b 54 c6 10 48 21 54 c7 10 c3 b8 00 00 00 00 eb da 48 8b 4f 08 <48> 39 4e 08 48 0f 4e 4e 08 49 89 c9 48 83 f9 3f 76 4e ba 40 00 00 Oct 02 06:31:20 our.host.name systemd[1]: Started Process Core Dump (PID 2169426/UID 0). Oct 02 06:31:20 our.host.name systemd-coredump[2169427]: Process 2048344 ( slurmctld) of user 991 dumped core. This is followed by a list of each of the dozen or so related threads. The one which is dumping core is first and looks like this: Stack trace of thread 2048355: #0 0x00007fec755d7ea8 bit_and_not (libslurmfull.so) #1 0x000000000044531f _job_alloc (slurmctld) #2 0x000000000044576b _job_alloc_whole_node_internal (slurmctld) #3 0x0000000000446e6d gres_ctld_job_alloc_whole_node (slurmctld) #4 0x00007fec722e29b8 job_res_add_job (select_cons_tres.so) #5 0x00007fec722f7c32 select_p_select_nodeinfo_set (select_cons_tres.so) #6 0x00007fec756e7dc7 select_g_select_nodeinfo_set (libslurmfull.so) #7 0x0000000000496eb3 select_nodes (slurmctld) #8 0x0000000000480826 _schedule (slurmctld) #9 0x00007fec753421ca start_thread (libpthread.so.0) #10 0x00007fec745f78d3 __clone (libc.so.6) I have run slurmctld with "debug5" level logging and it appears that the error occurs right after backfill considers a large number of jobs. Slurmctld could be failing at the end of backfill or when doing something which happens just after backfill runs. Usually this is the last message before the crash: [2024-09-25T18:39:42.076] slurmscriptd: debug: _slurmscriptd_mainloop: finished If anyone has any thoughts or advice on this that would be appreciated. Thank you. -- Marcus Lauer Systems Administrator CETS Group, Research Support

4 4

controller node in Ubuntu 22.04.5 LTS does not work with compute node in Debian GNU/Linux 12
by allspace＠qq.com 17 Jun '26

17 Jun '26

download slurm-26.05.1.tar.bz2 and build it on Ubuntu 22.04.5 LTS. Use it as controller node. download slurm-26.05.1.tar.bz2 and build it on Debian GNU/Linux 12. Use it as compute node(node-3). Run command "srun -w node-3 hostname" on controller node, got the following error message: srun: error: Task launch for StepId=10.0 failed on node node-3: Header lengths are longer than data received srun: error: Application launch failed: Header lengths are longer than data received srun: Job step aborted Here is the logs on node-3: [2026-06-16T12:45:20.645] error: _verify_signature: failed decode [2026-06-16T12:45:20.645] error: Malformed RPC of type REQUEST_LAUNCH_TASKS(6001) received [2026-06-16T12:45:20.653] error: slurm_unpack_msg_and_forward: [192.168.245.1:59308] failed: Header lengths are longer than data received [2026-06-16T12:45:20.663] error: wrap_on_data: [192.168.245.133:6818(fd:13)] on_data returned rc: Header lengths are longer than data received How can I fix this problem? Thanks

5 7

How to weight nodes properly based on RAM to prevent small instances clogging big instances with backfilling
by Xaver Stiensmeier 15 Jun '26

15 Jun '26

Dearl Slurm User List, I am currently reviewing a slurm.conf where the developer set Weight manually to attribute a greater weight to machines that have more RAM to force smaller jobs on smaller instances. However, I feel like there is something already in place or better than manually setting the weights, but I couldn't find it. If I understand correctly Slurm does not schedule jobs to the smallest possible node on default. So small jobs can be scheduled to large instances and a big job might have to wait indefinitely when using backfilling. I thought that Slurm does have mechanisms to prevent this but was unable to find it again in the documentation. Is there really no automatism at place or am I overlooking something? Best, Xaver

3 2

Memory requirements: seff vs jobstats
by Loris Bennett 15 Jun '26

15 Jun '26

Hi, For quite a few jobs which run on our system, 'seff' reports higher values for 'Memory Utilized' than 'jobstats' does via 'CPU memory usage' (see below for example of a job running on a single node). I was under the impression that 'seff' reports MaxRSS (from 'sacct'), whereas 'jobstats' reports 'Used' (from cgroups), which, as I understand it, is RSS plus page cache and possibly some kernel memory. Therefore, I would expect 'jobstats' to generally produce *larger* estimates of memory usage. However, the discrepancies I see seem always to be such that 'jobstats' report *lower* memory usage. Any thoughts on this? Cheers, Loris $ seff 25800851 Job ID: 25800851 Cluster: curta User/Group: foo/agbar State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 24 CPU Utilized: 10-09:05:31 CPU Efficiency: 39.05% of 26-13:51:36 core-walltime Job Wall-clock time: 1-02:34:39 Memory Utilized: 18.12 GB Memory Efficiency: 28.64% of 63.28 GB (2.64 GB/core) $ jobstats 25800851 ================================================================================ Slurm Job Statistics ================================================================================ Job ID: 25800851 User/Account: foo/agbar Job Name: foobar State: COMPLETED Nodes: 1 CPU Cores: 24 CPU Memory: 64800MB (2.7GB per CPU-core) QOS/Partition: standard/main Cluster: curta Start Time: Mon Jun 8, 2026 at 3:53 PM Run Time: 1-02:34:39 Time Limit: 5-00:00:00 Overall Utilization ================================================================================ CPU utilization [||||||||||||||||||| 39%] CPU memory usage [||||| 11%] Detailed Utilization ================================================================================ CPU utilization per node (CPU time used/run time) c036.curta.zedat.fu-berlin.de: 10-09:03:12/26-13:51:36 (efficiency=39.0%) CPU memory usage per node - used/allocated c036.curta.zedat.fu-berlin.de: 7.1GB/63.3GB (304.3MB/2.6GB per core of 24) ... -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

1 0

Configurator tool on Slurm site is not found
by Valerio Bellizzomi 11 Jun '26

11 Jun '26

Please check. VB

1 0

Visit the Slurm & NVIDIA Booths at ISC 2026
by Victoria Hobson 10 Jun '26

10 Jun '26

If you're heading to ISC in Hamburg, Germany (June 22–26), come find us on the floor! Stop by the Slurm booth (Z10) for sessions on HPC workload management, Kubernetes, and more: Tuesday, June 23 12:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 17:00 – What's New: Latest Slurm and Slinky Release 19:00 – Slinky: Slurm for Kubernetes 101 Wednesday, June 24 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 14:00 – Slinky: Slurm for Kubernetes 101 15:00 – What's New: Latest Slurm and Slinky Release 17:00 – Redefining Resiliency for HPC Research and AI Factories: Slurm and NVIDIA Mission Control Thursday, June 25 10:00 – Slinky: Slurm for Kubernetes 101 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement Then head over to the NVIDIA booth (E30) for even more great content and conversations. We hope to see you there! Victoria Hobson | NVIDIA Campaign Manager, CUDA and Slurm

1 0

Use of scontrol update PartitionName
by Richard Lefebvre 09 Jun '26

09 Jun '26

Hi, To adapt to changing users need, we change which nodes are in what partitions. We use the command: scontrol update PartitionName=partition01 Nodes=nd[001-200] and modify slurm.conf at the same time to match. But when I try to just modify slurm.conf and restart the process "slurmctld". The command "scontrol show PartitionName=partition01" still show the old list of nodes. Even after I reboot the slurmctld server completely. I don't know where the cached version of the list is or how it keeps it. Is there a some slurm.conf configuration that dictated that behavior? Any hints? Richard

2 1

We are pleased to announce the availability of Slurm 26.05.1
by Marshall Garey 09 Jun '26

09 Jun '26

We are pleased to announce the availability of Slurm 26.05.1. Some of the changes in 26.05.1 are highlighted below: * Fix a significant performance regression in slurmctld. * Fix a database performance regression handling step completion records. * Fix an incompatibility between 26.05 slurmd and 25.05+ slurmstepd when using a 24.11 sattach that would cause the sattach to hang. The full list of changes are available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Marshall Garey Senior System Software Engineer NVIDIA

1 0