- slurm-users - lists.schedmd.com

Slinky 1.2.0 released
by Marlow Warnicke 02 Jul '26

02 Jul '26

We are pleased to announce the availability of Slinky version 1.2.0! Slinky is SchedMD (now part of NVIDIA)'s set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.2.0: https://github.com/SlinkyProject/slurm-operator/releases/tag/v1.2.0 New features include a ScheduledUpdate strategy for NodeSets to control when updates are applied, support for Google GKE A3 Mega instance types, and configurable pod security contexts for operator and webhook deployments. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge brings Kubernetes and Slurm jobs together under one scheduler. Both types of workloads share the same cluster resources. Release v1.2.0: https://github.com/SlinkyProject/slurm-bridge/releases/tag/v1.2.0 New features include support for LeaderWorkerSet workloads introduced in Kubernetes 1.37, priority annotations for workloads to control Slurm job scheduling order, and dynamic topology reconciliation for external and hybrid Slurm nodes. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… #kubernetes #slurm #dra #ai #training #inference #hpc #nvidia #gpu #cloudnative #slinky

1 0

Avoid killing GPU jobs during preemption
by Ratnasamy, Fritz 29 Jun '26

29 Jun '26

Hello, We have a recurring demand where some users would like to use idle GPU resources on periods where the usage is low. Their normal GPU/qos limits prevent this, which is expected as part of the fair share scheduler principles. A pattern we're considering is the preempt option on slurm to have these jobs being pre-empted when another user is submitting his jobs within his quota. The jobs that were submitted earlier by a faculty member taking advantage of available resources on top of his regular qos, will be preempted however, there is no suspend option for gpu jobs. There is only requeue, which kills the jobs to make the gpu available. I believe you can use suspend for cpu/mem without killing the job. I know that would require the users to set up checkpoints and re-start their jobs. The goal is to let heavy users fill otherwise idle nodes over low usage time without impacting smaller users when activity picks up. Has anyone implemented something similar? I am in general against preemption because that might lead to conflict between users. Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

2 1

ISC 2026 is here! Come see us in Hamburg.
by Victoria Hobson 22 Jun '26

22 Jun '26

The wait is over! ISC 2026 is officially underway in Hamburg, Germany, and we'd love to see you! As part of the newly expanded NVIDIA family, the Slurm team is on the floor and ready to connect. Swing by the Slurm booth (Z10) for sessions on GPU workload management, Slinky, and the latest releases and don't miss the NVIDIA booth (E30) for even more great content all week long. Here's what's on tap at booth Z10: Tuesday, June 23 12:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 17:00 – What's New: Latest Slurm and Slinky Release 19:00 – Slinky: Slurm for Kubernetes 101 Wednesday, June 24 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 14:00 – Slinky: Slurm for Kubernetes 101 15:00 – What's New: Latest Slurm and Slinky Release 17:00 – Redefining Resiliency for HPC Research and AI Factories: Slurm and NVIDIA Mission Control Thursday, June 25 10:00 – Slinky: Slurm for Kubernetes 101 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement Come say hello — we'll see you on the floor! Victoria Hobson | NVIDIA Campaign Manager, CUDA and Slurm

1 0

slurmctld keeps segfaulting, possibly during or just after backfill
by Marcus Lauer 21 Jun '26

21 Jun '26

We are running into a problem where slurmctld is segfaulting a few times a day. We had this problem with SLURM 23.11.8 and now with 23.11.10 as well, though the problem only appears on one of the several SLURM clusters we have, and all of them use one of those versions of SLURM. I was wondering if anyone has encountered a similar issue and has any thoughts on how to prevent this. Obviously we use "SchedulerType=sched/backfill" but strangely when I switched to sched/builtin for a while there were still slurmctld segfaults. We also set "SchedulerParameters=enable_user_top,bf_max_job_test=2000". I have tried turning those off but it did not help. I have also tried tweaking several other settings to no avail. Most of the cluster runs Rocky Linux 8.10 (including the slurmctld system) though we still have some Scientific Linux 7.9 compute nodes (we compile SLURM separately for those). Here is the crash-time error from journalctl: Oct 02 06:31:20 our.host.name kernel: sched_agent[2048355]: segfault at 8 ip 00007fec755d7ea8 sp 00007fec6bffe7e8 error 4 in libslurmfull.so[7fec7555a000+1f4000] Oct 02 06:31:20 our.host.name kernel: Code: 48 39 c1 7e 19 48 c1 f8 06 ba 01 00 00 00 48 d3 e2 48 f7 da 48 0b 54 c6 10 48 21 54 c7 10 c3 b8 00 00 00 00 eb da 48 8b 4f 08 <48> 39 4e 08 48 0f 4e 4e 08 49 89 c9 48 83 f9 3f 76 4e ba 40 00 00 Oct 02 06:31:20 our.host.name systemd[1]: Started Process Core Dump (PID 2169426/UID 0). Oct 02 06:31:20 our.host.name systemd-coredump[2169427]: Process 2048344 ( slurmctld) of user 991 dumped core. This is followed by a list of each of the dozen or so related threads. The one which is dumping core is first and looks like this: Stack trace of thread 2048355: #0 0x00007fec755d7ea8 bit_and_not (libslurmfull.so) #1 0x000000000044531f _job_alloc (slurmctld) #2 0x000000000044576b _job_alloc_whole_node_internal (slurmctld) #3 0x0000000000446e6d gres_ctld_job_alloc_whole_node (slurmctld) #4 0x00007fec722e29b8 job_res_add_job (select_cons_tres.so) #5 0x00007fec722f7c32 select_p_select_nodeinfo_set (select_cons_tres.so) #6 0x00007fec756e7dc7 select_g_select_nodeinfo_set (libslurmfull.so) #7 0x0000000000496eb3 select_nodes (slurmctld) #8 0x0000000000480826 _schedule (slurmctld) #9 0x00007fec753421ca start_thread (libpthread.so.0) #10 0x00007fec745f78d3 __clone (libc.so.6) I have run slurmctld with "debug5" level logging and it appears that the error occurs right after backfill considers a large number of jobs. Slurmctld could be failing at the end of backfill or when doing something which happens just after backfill runs. Usually this is the last message before the crash: [2024-09-25T18:39:42.076] slurmscriptd: debug: _slurmscriptd_mainloop: finished If anyone has any thoughts or advice on this that would be appreciated. Thank you. -- Marcus Lauer Systems Administrator CETS Group, Research Support

4 4

controller node in Ubuntu 22.04.5 LTS does not work with compute node in Debian GNU/Linux 12
by allspace＠qq.com 17 Jun '26

17 Jun '26

download slurm-26.05.1.tar.bz2 and build it on Ubuntu 22.04.5 LTS. Use it as controller node. download slurm-26.05.1.tar.bz2 and build it on Debian GNU/Linux 12. Use it as compute node(node-3). Run command "srun -w node-3 hostname" on controller node, got the following error message: srun: error: Task launch for StepId=10.0 failed on node node-3: Header lengths are longer than data received srun: error: Application launch failed: Header lengths are longer than data received srun: Job step aborted Here is the logs on node-3: [2026-06-16T12:45:20.645] error: _verify_signature: failed decode [2026-06-16T12:45:20.645] error: Malformed RPC of type REQUEST_LAUNCH_TASKS(6001) received [2026-06-16T12:45:20.653] error: slurm_unpack_msg_and_forward: [192.168.245.1:59308] failed: Header lengths are longer than data received [2026-06-16T12:45:20.663] error: wrap_on_data: [192.168.245.133:6818(fd:13)] on_data returned rc: Header lengths are longer than data received How can I fix this problem? Thanks

5 7

How to weight nodes properly based on RAM to prevent small instances clogging big instances with backfilling
by Xaver Stiensmeier 15 Jun '26

15 Jun '26

Dearl Slurm User List, I am currently reviewing a slurm.conf where the developer set Weight manually to attribute a greater weight to machines that have more RAM to force smaller jobs on smaller instances. However, I feel like there is something already in place or better than manually setting the weights, but I couldn't find it. If I understand correctly Slurm does not schedule jobs to the smallest possible node on default. So small jobs can be scheduled to large instances and a big job might have to wait indefinitely when using backfilling. I thought that Slurm does have mechanisms to prevent this but was unable to find it again in the documentation. Is there really no automatism at place or am I overlooking something? Best, Xaver

3 2

Memory requirements: seff vs jobstats
by Loris Bennett 15 Jun '26

15 Jun '26

Hi, For quite a few jobs which run on our system, 'seff' reports higher values for 'Memory Utilized' than 'jobstats' does via 'CPU memory usage' (see below for example of a job running on a single node). I was under the impression that 'seff' reports MaxRSS (from 'sacct'), whereas 'jobstats' reports 'Used' (from cgroups), which, as I understand it, is RSS plus page cache and possibly some kernel memory. Therefore, I would expect 'jobstats' to generally produce *larger* estimates of memory usage. However, the discrepancies I see seem always to be such that 'jobstats' report *lower* memory usage. Any thoughts on this? Cheers, Loris $ seff 25800851 Job ID: 25800851 Cluster: curta User/Group: foo/agbar State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 24 CPU Utilized: 10-09:05:31 CPU Efficiency: 39.05% of 26-13:51:36 core-walltime Job Wall-clock time: 1-02:34:39 Memory Utilized: 18.12 GB Memory Efficiency: 28.64% of 63.28 GB (2.64 GB/core) $ jobstats 25800851 ================================================================================ Slurm Job Statistics ================================================================================ Job ID: 25800851 User/Account: foo/agbar Job Name: foobar State: COMPLETED Nodes: 1 CPU Cores: 24 CPU Memory: 64800MB (2.7GB per CPU-core) QOS/Partition: standard/main Cluster: curta Start Time: Mon Jun 8, 2026 at 3:53 PM Run Time: 1-02:34:39 Time Limit: 5-00:00:00 Overall Utilization ================================================================================ CPU utilization [||||||||||||||||||| 39%] CPU memory usage [||||| 11%] Detailed Utilization ================================================================================ CPU utilization per node (CPU time used/run time) c036.curta.zedat.fu-berlin.de: 10-09:03:12/26-13:51:36 (efficiency=39.0%) CPU memory usage per node - used/allocated c036.curta.zedat.fu-berlin.de: 7.1GB/63.3GB (304.3MB/2.6GB per core of 24) ... -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

1 0

Configurator tool on Slurm site is not found
by Valerio Bellizzomi 11 Jun '26

11 Jun '26

Please check. VB

1 0

Visit the Slurm & NVIDIA Booths at ISC 2026
by Victoria Hobson 10 Jun '26

10 Jun '26

If you're heading to ISC in Hamburg, Germany (June 22–26), come find us on the floor! Stop by the Slurm booth (Z10) for sessions on HPC workload management, Kubernetes, and more: Tuesday, June 23 12:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 17:00 – What's New: Latest Slurm and Slinky Release 19:00 – Slinky: Slurm for Kubernetes 101 Wednesday, June 24 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 14:00 – Slinky: Slurm for Kubernetes 101 15:00 – What's New: Latest Slurm and Slinky Release 17:00 – Redefining Resiliency for HPC Research and AI Factories: Slurm and NVIDIA Mission Control Thursday, June 25 10:00 – Slinky: Slurm for Kubernetes 101 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement Then head over to the NVIDIA booth (E30) for even more great content and conversations. We hope to see you there! Victoria Hobson | NVIDIA Campaign Manager, CUDA and Slurm

1 0

Use of scontrol update PartitionName
by Richard Lefebvre 09 Jun '26

09 Jun '26

Hi, To adapt to changing users need, we change which nodes are in what partitions. We use the command: scontrol update PartitionName=partition01 Nodes=nd[001-200] and modify slurm.conf at the same time to match. But when I try to just modify slurm.conf and restart the process "slurmctld". The command "scontrol show PartitionName=partition01" still show the old list of nodes. Even after I reboot the slurmctld server completely. I don't know where the cached version of the list is or how it keeps it. Is there a some slurm.conf configuration that dictated that behavior? Any hints? Richard

2 1

We are pleased to announce the availability of Slurm 26.05.1
by Marshall Garey 09 Jun '26

09 Jun '26

We are pleased to announce the availability of Slurm 26.05.1. Some of the changes in 26.05.1 are highlighted below: * Fix a significant performance regression in slurmctld. * Fix a database performance regression handling step completion records. * Fix an incompatibility between 26.05 slurmd and 25.05+ slurmstepd when using a 24.11 sattach that would cause the sattach to hang. The full list of changes are available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Preemptible qos
by Steen Lysgaard 04 Jun '26

04 Jun '26

Hello, We have a recurring situation where some users would like to use idle cluster resources on weekends. Their normal CPU limits prevent this, which is generally desirable during the work week to ensure fair sharing. A pattern we're considering is a special low-priority QOS that allows users to exceed their normal association limits, and that is preemptible by jobs submitted with the normal QOS. The goal is to let heavy users fill otherwise idle nodes over weekends without impacting smaller users when activity picks up again on Monday. The walltime limit (50 hours) is not something we would like to change. Has anyone implemented something similar? If so, I'd be interested in hearing about experiences and any pitfalls to this approach. Thanks, Steen Lysgaard [http://www.dtu.dk/-/media/DTU_Generelt/Andet/mail-signature-logo.png] Steen Lysgaard Scientific Software Developer DTU Energy Mob. +45 51 94 48 86 stly(a)dtu.dk<mailto:stly@dtu.dk> Agnes Nielsens Vej Building 301 2800 Kgs. Lyngby www.dtu.dk/english<https://www.dtu.dk/english>

2 1

Authentication
by Sterner, Douglas E. 02 Jun '26

02 Jun '26

New kubernetes deployment and I can't authenticate via ssh using AD credentials to the login controller. Both id and getent resolve user correctly. I have tried every sssd parameter known to mankind. Can someone provide a working example of an sssd.conf for active directory non ssl that I can compare against. Also do I need to do this in the values.yaml? slurm: config: # Explicitly enable PAM for node environments and pam_slurm_adopt UsePAM: 1 Thanks, Douglas Sterner Johns Hopkins Applied Physics Laboratory NSAD / JAF 11100 Johns Hopkins Road. Laurel, MD 20723-6091 [cid:image001.png@01DCF28D.A50E98A0]

2 1

Packing or spreading single-CPU jobs
by Gerhard Strangar 02 Jun '26

02 Jun '26

Hello, I'd like to somehow achieve this: Group A of nodes runs jobs that are supposed to be packed, i.e. if each node has 16 cpus, pack 16 jobs on the first node before using node 2. But for group B, assign every job their own node until all of them have one job, then put a second one on each node until all of them have 2 ... Ir number of jobs doesn't work, load or available memory would do as well, but it seems there's no way of telling slurm to sort the possible nodes by some criteria. Gerhard

2 1

Purging of job_script and job_env from the database in 26.05
by Ole Holm Nielsen 01 Jun '26

01 Jun '26

If you are using AccountingStoreFlags=job_script and/or job_env in slurm.conf to store job scripts in the Slurm database, you may be interested in a highly useful feature introduced in Slurm 26.05: Previously, the stored job_scripts and job_env entries would accumulate without bounds in the database, potentially causing database performance or size issues, especially while upgrading Slurm. With 26.05 there are some new slurmdbd.conf options which can be used to purge job_script or job_env entries in the database, see ticket_23818 [1]. This new feature isn't mentioned at all in the Release notes [4], and is only inconspicuously mentioned way down in the Changes section [3] as: > Adding new archive/purge options to allow for explicit archiving of job_scripts and job_env without jobs. The safe way to enable the new purge parameters is to introduce them very gradually, for example: PurgeJobScriptAfter=2000days PurgeJobEnvAfter=2000days and lower the values little by little over time (restarting slurmdbd each time). The MaxPurgeLimit [5] parameter may alternatively be used to limit the amount of database work. Best regards, Ole [1] https://support.schedmd.com/show_bug.cgi?id=23818 [2] https://slurm.schedmd.com/slurm.conf.html#OPT_AccountingStoreFlags [3] https://github.com/SchedMD/slurm/releases [4] https://slurm.schedmd.com/release_notes.html [5] https://slurm.schedmd.com/slurmdbd.conf.html#OPT_MaxPurgeLimit -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

2 3

Large overhead for --task-epilog
by Alan O'Cais 29 May '26

29 May '26

Hi all, I've been experimenting with task prolog/epilog and noticed that task epilogs have significant overhead. The overhead seems to be independent of the epilog script content: $ cat test.sh #!/bin/bash $ time srun --overlap --task-prolog=$PWD/test.sh --ntasks=64 sleep 1 real 0m1.730s user 0m0.211s sys 0m0.009s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=2 sleep 1 real 0m3.530s user 0m0.213s sys 0m0.007s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=64 sleep 1 real 1m5.936s user 0m0.216s sys 0m0.005s This test was on one node, with 64 tasks in the allocation. The overhead seems linked only to number of tasks per node, as adding additional nodes did not increase the overhead. I saw this on $ srun --version slurm 23.11.10-BullSequana.1.2.1 and someone else reproduced similar results with |slurm 25.05.7|. I can live with the overhead, but I am curious as to why it happens and to understand the design of the implementation. Does the task epilog script only run when all tasks finish, or does it run immediately at exit for each task? Are the calls serialised? My use case is a fusermount that I want to mount in userspace at task start and clean up at task finish, one task per node is responsible for the mount/unmount and I want to understand if I have a race condition for the unmount. Any advice very much appreciated, Alan

1 0

Technical Update: Native Slurm Simulation Environment Successfully Configured
by sami sami 28 May '26

28 May '26

Hi All, I am currently working on a single Linux machine running *Ubuntu 25.10* and am looking to conduct performance testing on the *Slurm backfill algorithm*. My objective is to simulate a cluster environment consisting of *1,024 nodes*. While researching existing solutions, I found that many modern Slurm-simulator implementations (such as various plugin-based designs) rely heavily on *Docker and Docker Compose*. However, I would like to avoid using containerization and instead run the simulation natively on my host OS. I understand that older versions of the Slurm source code included a native sim directory that allowed for simulations without external dependencies like Docker. Given this, I have the following technical questions: - *Native Compatibility:* Is it still feasible to compile and run a Slurm simulator natively on Ubuntu 25.10? - *Version Recommendations:* Would you recommend using a specific legacy version of Slurm that still contains the original simulation files, or is there a way to port that functionality to a more recent release? - *Configuration:* Are there specific build flags or configuration steps required to enable the simulated node environment (1,024 nodes) on a single machine without triggering the overhead of containerized networking? I would appreciate any guidance or documentation you could provide on achieving a high-node-count simulation in a "bare-metal" Linux environment. Best regards,

3 2

GPU fallen of the bus
by Ratnasamy, Fritz 27 May '26

27 May '26

Hello, We are noticing that some of the gpus on a specific node have "fallen of the bus". We would like to remove this specific gpu from the slurm scheduler. For example, let's say GPU0 has fallen off the bus, we would need the rest of the GPU1-8 to be available and make GPU0 not able to be allocated. How can we achieve that? I have read about blacklist on the slurm forum but it seems there is no satisfying solution. Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Slurm version 26.05 is now available
by Marshall Garey 26 May '26

26 May '26

We are pleased to announce the availability of Slurm 26.05. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md The Slurm documentation has also been updated to the 26.05 release: https://slurm.schedmd.com Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Core Specialization Feature
by Dan Healy 21 May '26

21 May '26

I'm curious how to report to our users that the number of available CPU cores on a given node are *(CoresPerSocket * SocketsPerBoard * ThreadsPerCore) - CpuSpecCount*. In a basic example, if the node has 32 cores and CpuSpecCount=1, then there's actually 31 cores available. If a user runs `srun -n32 -N1 bash` he/she will be denied for resources. Then, If a user runs `sinfo -O cpus,nodes` shows there's 32 cores available and the user is confused. How can the user clearly identify that in the above case, the exact number of available CPU cores available are 31 and *not* 32? -- Thanks, Daniel Healy

2 2

cgroup error in SLURM version 25.11.2
by bradford.kerr＠srs.gov 19 May '26

19 May '26

I'm trying to install and run SLURM on a stand-alone (no network access) cluster of four PCs running Red Hat Linux 9. On the first compute node of the cluster, I can not get past the error message "cannot create cgroup context for cgroup/v2". It appears that file cgroup_v2.so does not exist. How do I get past this error?

1 0

We are pleased to announce the availability of Slurm versions 25.11.6 and 25.05.8.
by Marshall Garey 14 May '26

14 May '26

We are pleased to announce the availability of Slurm versions 25.11.6 and 25.05.8. Fixes in these releases primarily cover various stability issues when reconfiguring or shutting down and improved support for newer libraries and compilers. The 25.11.6 release also includes a performance fix for scheduling with topology/block and a regression from 25.11.4 that could cause a "No records archived...but some found" message when archiving job data. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

inconsistent 'Requested node configuration is not available' error
by Paul Raines 14 May '26

14 May '26

In my cluster right now we have: $ sinfo ... pubgpu-req up 7-00:00:00 7 mix cobra,l40s-[01,03],luisa,rtx-03,sevilla,shelob pubgpu-req up 7-00:00:00 5 idle fiber,glaurung,gothmog,l40s-02,leo The following works fine: $ srun -p pubgpu-req -A sysadm --nodelist=cobra,l40s-02 --gres=gpu:1 -N 1 \ --ntasks-per-node=1 --mem=1G --time=1:00:00 --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 cobra[0]:~$ exit exit However just change the order of the nodelist and you get $ srun -p pubgpu-req -A sysadm --nodelist=l40s-02,cobra --gres=gpu:1 -N 1 \ --ntasks-per-node=1 --mem=1G --time=1:00:00 --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 srun: error: Unable to create step for job 8255390: Requested node configuration is not available More experimentation with this and it appears nodelist HAS to be in alphabetical order $ srun -p pubgpu-req -A sysadm --nodelist=l40s-02,leo,fiber \ --gres=gpu:1 -N 1 --ntasks-per-node=1 --mem=1G --time=1:00:00 \ --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 srun: error: Unable to create step for job 8255401: Requested node configuration is not available $ srun -p pubgpu-req -A sysadm --nodelist=fiber,l40s-02,leo \ --gres=gpu:1 -N 1 --ntasks-per-node=1 --mem=1G --time=1:00:00 \ --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 fiber[0]:~$ Surely there is no good reason for this? --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

2 1

SLURM config option to not tie up a host completely.
by Ron Gould 08 May '26

08 May '26

I have a user asking about a SLURM config setting that would allow a job to not tie up a node's unused resources, thereby allowing another job to run concurrently. User: On our cluster, our Fluent GPU jobs each use only 1 of the 2 GPUs on GPUServer[1,2]. Currently the node is fully allocated to whichever job lands on it first, leaving the second GPU, 38 CPU cores, and 270+ GB of RAM idle. Has anyone dealt with this? What options would facilitate this? Are there any gotchas or pitfalls?

4 3

Moving a Slurm database from one slurm database host to another
by James A Allsopp 08 May '26

08 May '26

Hi, We'd like to move our Slurm database from its existing machine running mariadb-server-5.5.68 to another running mariadb-server-10.5.29-3, both of which are running slurm 24.11.7. We dumped the database successfully from the old server, reimported it into the new server using mysqldump and *mysql < db_dump.sql*. The new slurmdbd service was then started. The clusters have different names so there were a few inconsistencies, but when we started the new slurm cluster, we could see the jobs according to sacct (using the -M option) matched those of the original cluster, but none of the associated job steps were shown. I manually logged into the new database and could see the jobs in the job_table but it looked like the foreign key link had been broken in the step table as there were entries in that table. Am I missing some information about how to export and import properly? I was wondering if the import had triggered auto-increment on keys and that was breaking things? Ideally, we'd like to change the name of the cluster as well. Is that just a matter of altering the prefix on the relevant tables, which is something I can do by hand on the database dump before reimporting? For instance, changing the table <old_name>_event_table to <new_cluster_name>_event_table. Thanks James -- Dr James Allsopp | Research Platforms team IT Services | University of Sheffield I do not work Tuesday and Wednesday 3-5pm. I also send emails at odd times, but respond at your convenience.

2 1

Slurm release candidate version 26.05.0rc1 is available for testing
by Tim Wickberg 07 May '26

07 May '26

We are pleased to announce the availability of Slurm release candidate 26.05.0rc1. To highlight some new features coming in 26.05: - New "srun --async" step mode that submits step processes to stepmgr to queue and eventually launch. This avoids issues with keeping a large number of srun processes backgrounded to queue up future step-based workflows. (This was previously described as "mini-batch" in the Slurm roadmap presentations.) - New topology/ring and topology/torus3d topology plugins. These implement a single-dimensional and three-dimensional topology respectively. - Slurm REST API - Add support for creating/updating/removing partitions, and for viewing the active slurmctld and slurmdbd configurations. - Dynamic Memory Resizing. A job can now release memory (and have the cgroup limits updated) using 'scontrol update' while running. A new "sbatch --mem-update=<margin>@<delay>" option can also automatically reduce the memory limit to the current usage plus a given margin percentage after a specified time. - Add topology-based sorting for node ranks when using dynamic nodes with the topology plugins. This is also available generically for topology/flat (no topology) with a new alpha_step_rank option. - Add an optimized single-node path through the scheduling logic for increased performance. - Expanded the openmetrics (Prometheus) nodes, partitions, and jobs endpoints with gpu allocation statistics. This is the first release candidate of the upcoming 26.05 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://support.schedmd.com against the 26.05.x version and we will address them before the first production 26.05.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://github.com/SchedMD/slurm/releases. The changelog for 26.05 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-26.05.md -- Tim Wickberg Director, System Software, Slurm + Slinky, NVIDIA

1 0

cgroup/v2 regression in 25.11.2 — memory.peak read disabled after first task
by Emyr James 06 May '26

06 May '26

Dear all, Heads-up for anyone running 25.11.2 with cgroup/v2 + jobacct_gather/cgroup: there's a regression in the recent series that effectively disables memory.peak accounting for every task after the first one per slurmstepd. The commit is f2d05c08e6 ("cgroup/v2 - Remove static qualifier from variable", ticket 23646, cherry-picked from e1835a1602). It dropped static from memory_peak_interface in cgroup_p_task_get_acct_data while leaving interfaces_checked static. The function now looks like: bool memory_peak_interface = false; static bool interfaces_checked = false; ... if (!interfaces_checked) { memory_peak_interface = cgroup_p_has_feature(CG_MEMCG_PEAK); interfaces_checked = true; } ... if (memory_peak_interface) { common_cgroup_get_param(..., "memory.peak", ...); } interfaces_checked persists across calls; memory_peak_interface no longer does. So the feature-probe block runs exactly once. From the second call onward memory_peak_interface is a fresh stack-local set to false, and the memory.peak read in the second if is skipped. stats->memory_peak ends up INFINITE64 for every task except the first one ever sampled. User-visible effect: MaxRSS in sacct stops reflecting the kernel-tracked peak and falls back to polling memory.current — which, at typical jobacct_gather intervals, routinely misses short-lived peaks. Which is of course exactly the problem memory.peak was added to solve (see the 2024 thread that prompted the original implementation: https://groups.google.com/g/slurm-users/c/BX1_4bztrJA) One-line fix is to put static back on memory_peak_interface. Or, arguably more robust, drop the interfaces_checked cache entirely and run the feature probe on every call — the cost is one access()-equivalent per accounting poll, basically free. he original v2 implementation in ba34862a40 was great, thanks for that — this looks like a small static-qualifier slip on the way through a cleanup. Cheers, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

1 1

slurmdbd and slurmctld prevent alma9 login
by Pols, Maarten 06 May '26

06 May '26

Dear Community, Our Slurm cluster has been running without any issues for several months on version 25.05.3. Last Friday, we experienced a power outage which required us to restart the server. After the restart, we were unable to log in to the master node. Eventually, we managed to access the system via a safe mode workaround. Through a process of elimination, we identified the slurmdbd and slurmctld services as the root cause of the issue. Would you happen to have any idea what might have caused this behavior? We have since upgraded to version 25.11.5, which appears to be running smoothly. However, we would still like to understand the underlying cause of the problem. Thank you in advance for your help. Kind regards, Maarten

3 3

Jobs canceling when nodes become unreachable – need guidance
by Pharthiphan Asokan 05 May '26

05 May '26

Hi, We’re seeing an issue where jobs submitted via salloc are automatically cancelled when a compute node becomes temporarily unreachable. Our goal is to keep jobs pending or requeued instead of being cancelled outright when a node drops offline briefly Slurm sometimes cancels the job rather than requeuing it when the node is marked DOWN/ DRAIN/ DRAINING*. Is there a recommended configuration or additional parameter that ensures jobs remain pending/requeued until the node returns, rather than being cancelled? Any insights or examples from similar setups would be greatly appreciated. Regards, Pharthiphan

3 2

How to track solver usage in Slurm
by fernando.seguro＠rtx.com 23 Apr '26

23 Apr '26

I don't see an easy way of tracking this in slurm. I tried using the --comment="slover" directive, but this gets mapped to the "derived_es" column in the database, which is reserved for job info on errors, from what I read. This would make a good enhancement request. Where would I submit that? Something like: SBATCH --solver=fluent or SBATCH --solver=LSDyna This would be very helpful for tracking solver usage without having to use the job-name. Great for reporting. Please let me know if someone else has requested this previously. If anyone has figured out a way to do it, please share. thanks

4 3

Can't clear the REASON
by Berg, Stephen P CIV USN NRL DET SSC MS (USA) 17 Apr '26

17 Apr '26

Just upgraded my grid to 25.11 a couple days ago. Got two nodes that were down for a bit and when I set them to resume and set the reason="" they are still showing up with the reason I had set while they were down. Nothing I do seems to unset that reason flag for these two nodes. I've set the nodes to down with a reason and then back to idle with the blank reason but it never goes back to "none" like the other nodes. Whatever the last reason set was persists. The behavior seems different since I updated to 25.11. Was running the 20.11 available from the epel-8 repos previously.

2 4

Jobs aborting after slurmctld reload on Intel nodes - AMD unaffected
by Pharthiphan Asokan 16 Apr '26

16 Apr '26

Hi team, We’re observing job aborts on Intel-based nodes immediately after a slurmctld reload. AMD nodes remain stable and jobs continue unaffected. No system or Slurm configuration changes were made before the issue started. Error observed: error: Aborting JobID=1288 due to change in socket/core configuration of allocated nodes Relevant node configuration (Intel node example): NodeName=smc-h4-u19 CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=1 RealMemory=510976 State=UNKNOWN Feature=model_SSG-1228-B From logs: error: valid_job_resources: smc-h4-u19 sockets:2:2, cores 64,32 error: Node configuration differs from hardware: CPUs=128:128(hw) Boards=1:1(hw) SocketsPerBoard=2:2(hw) What we’ve verified: * No changes in BIOS, firmware, or hardware topology * No edits to slurm.conf or slurm_nodes.conf * Reload (scontrol reconfigure) triggers job aborts only on Intel nodes * AMD nodes remain intact through reloads

3 2

We are pleased to announce the availability of Slurm version 25.11.5
by Marshall Garey 14 Apr '26

14 Apr '26

We are pleased to announce the availability of Slurm version 25.11.5. Some of the changes in 25.11.0 are highlighted below: * Fix a regression in 25.11.0 that could cause daemons to hang after reaching conmgr_max_connections. * Fix a regression added in 25.05.0 concerning how the slurmctld inherits /run/slurmctld/sack.socket when using AuthType=auth/slurm to prevent clients that connected during a reconfigure from hanging indefinitely. * Fix a race in tree forwarding that could cause a crash during shutdown * Fix several bugs in the namespace/linux plugin that caused instability The full list of changes are available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Why my job can't start (backfill reservation issue)
by Massimo Sgaravatto 14 Apr '26

14 Apr '26

Dear all I (try to) manage a slurm cluster composed by some CPU-only nodes and some worker nodes which have also GPUs: NodeName=cld-ter-[01-06] Sockets=2 CoresPerSocket=96 ThreadsPerCore=2 RealMemory=1536000 State=UNKNOWN NodeName=cld-ter-gpu-[01-05] Sockets=2 CoresPerSocket=96 ThreadsPerCore=2 Gres=gpu:nvidia-h100:4 RealMemory=1536000 State=UNKNOWN The GPU nodes are exposed through multiple partitions: PartitionName=gpus Nodes=cld-ter-gpu-[01-02] State=UP PriorityTier=20 PartitionName=sparch Nodes=cld-ter-gpu-03 AllowAccounts=sparch,operators QoS=sparch State=UP PriorityTier=20 PartitionName=geant4 Nodes=cld-ter-gpu-03 AllowAccounts=geant4,operators QoS=geant4 State=UP PriorityTier=20 PartitionName=enipred Nodes=cld-ter-gpu-04 AllowAccounts=enipred,operators QoS=enipred State=UP PriorityTier=20 PartitionName=enipiml Nodes=cld-ter-gpu-05 AllowAccounts=enipiml,operators QoS=enipiml State=UP PriorityTier=20 We also set a partition to allow cpu-only jobs on the GPU nodes, but these jobs should be preempted (killed and requeued) if jobs submitted to partitions with higher priorities require those resources: PreemptType=preempt/partition_prio PreemptMode=REQUEUE PartitionName=onlycpus-opp Nodes=cld-ter-gpu-[01-05],cld-dfa-gpu-06,btc-dfa-gpu-02 State=UP PriorityTier=10 Now, I don't understand why this job [*] submitted on the onlycpus-opp partition can't start running e.g. on the cld-ter-gpu-01, since it has a lot of free resources: [sgaravat@cld-ter-ui-01 ~]$ scontrol show node cld-ter-gpu-01 NodeName=cld-ter-gpu-01 Arch=x86_64 CoresPerSocket=96 CPUAlloc=8 CPUEfctv=384 CPUTot=384 CPULoad=5.93 AvailableFeatures=(null) ActiveFeatures=(null) Gres=gpu:nvidia-h100:4 NodeAddr=cld-ter-gpu-01 NodeHostName=cld-ter-gpu-01 Version=25.11.3 OS=Linux 5.14.0-611.45.1.el9_7.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 1 05:56:53 EDT 2026 RealMemory=1536000 AllocMem=560000 FreeMem=1192357 Sockets=2 Boards=1 State=MIXED+PLANNED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpus,onlycpus-opp BootTime=2026-04-09T10:39:35 SlurmdStartTime=2026-04-09T10:40:01 LastBusyTime=2026-04-09T11:54:46 ResumeAfterTime=None CfgTRES=cpu=384,mem=1500G,billing=839,gres/gpu=4,gres/gpu:nvidia-h100=4 AllocTRES=cpu=8,mem=560000M,gres/gpu=4,gres/gpu:nvidia-h100=4 CurrentWatts=0 AveWatts=0 I guess the "MIXED+PLANNED" is the answer, but as far as I can see only a job (283469) is planned for this worker node: sgaravat@cld-ter-ui-01 ~]$ squeue --start | grep ter-gpu-01 JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON) 283469 gpus vllm-pod ciangott PD 2026-04-13T14:31:40 1 cld-ter-gpu-01 (Resources) But job 283469 doesn't require too many resources [**], so the 2 jobs could run together. Why job 283534 can't start ? Any hints ? Thanks, Massimo [*] [sgaravat@cld-ter-ui-01 ~]$ scontrol show job=283534 JobId=283534 JobName=myscript.sh UserId=sgaravat(5008) GroupId=tbadmin(5001) MCS_label=N/A Priority=542954 Nice=0 Account=operators QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:41 TimeLimit=1-00:00:00 TimeMin=N/A SubmitTime=2026-04-13T11:10:13 EligibleTime=2026-04-13T11:10:13 AccrueTime=2026-04-13T11:10:13 StartTime=2026-04-13T11:58:39 EndTime=2026-04-14T11:58:39 Deadline=N/A PreemptEligibleTime=2026-04-13T11:58:39 PreemptTime=None SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-13T11:58:39 Scheduler=Backfill Partition=onlycpus-opp AllocNode:Sid=cld-ter-ui-01:3035857 ReqNodeList=(null) ExcNodeList=(null) NodeList=btc-dfa-gpu-02 BatchHost=btc-dfa-gpu-02 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=100G,node=1,billing=26 AllocTRES=cpu=1,mem=100G,node=1,billing=26 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null) Command=/shared/home/sgaravat/myscript.sh SubmitLine=sbatch myscript.sh WorkDir=/shared/home/sgaravat StdErr=/shared/home/sgaravat/JOB-myscript.sh.283534.4294967294.err StdIn=/dev/null StdOut=/shared/home/sgaravat/JOB-myscript.sh.283534.4294967294.out MailUser=massimo.sgaravatto(a)pd.infn.it MailType=INVALID_DEPEND,BEGIN,END,FAIL,REQUEUE,STAGE_OUT [**] sgaravat@cld-ter-ui-01 ~]$ scontrol show job=283469 JobId=283469 JobName=vllm-pod UserId=ciangott(6054) GroupId=tbuser(6000) MCS_label=N/A Priority=499703 Nice=0 Account=cms QOS=normal JobState=PENDING Reason=Resources Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A SubmitTime=2026-04-13T06:48:37 EligibleTime=2026-04-13T06:48:37 AccrueTime=2026-04-13T06:48:37 StartTime=2026-04-13T14:31:40 EndTime=2026-04-14T14:31:40 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-13T11:59:48 Scheduler=Main Partition=gpus AllocNode:Sid=cld-ter-ui-01:3015801 ReqNodeList=(null) ExcNodeList=(null) NodeList= SchedNodeList=cld-ter-gpu-01 NumNodes=1-1 NumCPUs=32 NumTasks=1 CPUs/Task=32 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=32,mem=190734M,node=1,billing=118,gres/gpu=2,gres/gpu:nvidia-h100=2 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=32 MinMemoryNode=190734M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null) Command=.interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.slurm SubmitLine=sbatch .interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.slurm WorkDir=/shared/home/ciangott StdErr= StdIn=/dev/null StdOut=/shared/home/ciangott/.interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.out TresPerNode=gres/gpu:nvidia-h100:2 TresPerTask=cpu=32

5 10

pam_slurm_adopt
by Ratnasamy, Fritz 09 Apr '26

09 Apr '26

Hi, We have blocked ssh to nodes where users do not have a running job. However, when there is a running job, a user can ssh to that node and use more than the slurm resources allocated on that node. Am I missing something? I usually request an interactive session of 2GB. I then open another terminal and ssh to the node where the interactive session runs. I allocated a 4GB R variable on the node where I just ssh but this does not error out. Additionally, if I end my interactive session, the ssh connection still lives which I believe should not happen. Best, *Fritz Ratnasamy*Data Scientist Information Technology

7 6

How to delete my defaultwckey ?
by Faraz Hussain 07 Apr '26

07 Apr '26

I want every submitted job to have some value for the wckey, i.e: #SBATCH --wckey=myproject I made the appropriate changes to slurm.conf and slurmdb.conf and it works great. I can track how many hours people are using with those wckeys. But now I want to make it mandatory to use a wckey. To do that I need to delete the default wckey associated with the user's account. I tried doing it as follows, but it still lets me submit jobs without a wckey. It probably thinks I have an "empty" default wckey. sacctmgr mod user fhussa set defaultwckey= [root@mas01 ~]# sacctmgr list user fhussa format=user,defaultwckey User Def WCKey ---------- ---------- fhussa

1 0

Difference between sreport and sacct
by hany.ibrahim＠kit.edu 07 Apr '26

07 Apr '26

Hello, I am trying to understand how sreport work. So I run sreport and obtain th estatistics about speciic project. On teh other side, I collected the jobs related to this account using sacct. When I try to sum the numbers, sometimes it don't sum up. I can't see any reservations. This is the sreport I used: sreport --tres cpu cluster AccountUtilizationByUser Start=2021-01-01 account=<account> -t hours -np This is jobs I collected by using sacct: sudo -u slurm sacct -a -S "2021-01-01" -E NOW --format=JobID%100,User,Account%100,State,AllocCPUs,CPUTime%100,CPUTIMERAW%100,ElapsedRaw%100,AllocTres%100,AllocNodes,NodeList,NNodes --noheader --parsable2 | grep <account> I tried to exclude the steps from the main Jobs, so I summed only the main jobs whatever its state (canceled, completed, timeout, fail..). However the sreport occasionally is larger. What I am missing exactly? Warm regards, Hany.

1 0

Shared queue: how to request full node with all resources?
by Dustin Lang 31 Mar '26

31 Mar '26

Hi, With a partition with "OverSubscribe=FORCE" set, is there a way to request all the node resources? I see "--mem=0" does that for memory. But I do not see an option to request all the CPUs and GRES/TRES such as GPUs. I tried "--nodes=1 --ntasks=1 --cpus-per-task=0", but "--cpus-per-task=0" does not do the same thing as "--mem=0". In other words, is it possible to have a queue where nodes can be shared between jobs, but have a simple way for an sbatch script to request the full node with all its memory, cpus, and other resources? We have a heterogeneous cluster, so telling users (or jupyterhub scripts) to list exactly the resources they want doesn't really work. thanks, dustin

3 4

How do you handle GPU node failures during long jobs?
by Antonio Jose Alonso-Stepanov 26 Mar '26

26 Mar '26

Hi all, I'm a Stanford CS student looking into how sites handle GPU node failures during long-running jobs. A couple questions: When a GPU node goes down mid-job, do most sites use Slurm's requeue or --no-kill to handle it, or is it mostly manual drain and resubmit? Is anyone using HealthCheckProgram to catch GPU issues (like ECC errors via DCGM), or do you handle GPU health monitoring outside of Slurm? Curious what's worked and what hasn't. Thanks. Antonio

3 2

_node_config_validate: gres/gpu: Count changed on node (0 != 2)
by Xaver Stiensmeier 23 Mar '26

23 Mar '26

Hey Slurm-users list, while our regular gpu nodes are working fine, our on demand gpu nodes have a weird issue. They power up, I can ssh onto them and execute nvidia-smi on them without issue, but they are marked invalid and slurmctld logs _node_config_validate: gres/gpu: Count changed on node (0 != 2) however, node show shows that the gpus are recognized and the gres.conf are stored on the worker nodes as expected and the node entries in the slurm.conf are fine, too: # slurm.conf NodeName=my_worker_node SocketsPerBoard=16 CoresPerSocket=1 RealMemory=64075 MemSpecLimit=4000 State=CLOUD Gres=gpu:L4:2 # openstack # gres.conf on my_worker_node ubuntu@my_node:~$ cat /etc/slurm/gres.conf # GRES CONFIG Name=gpu Type=L4 File=/dev/nvidia0 Name=gpu Type=L4 File=/dev/nvidia1 Thankful for any ideas and debugging ideas. Best, Xaver PS: By executing: sudo scontrol update NodeName=$(bibiname 0) Gres= sudo scontrol reconfigure sudo scontrol update NodeName=$(bibiname 0) state=RESUME reason=None the node can be resumed. However, this is not a real solution.

2 3

We are pleased to announce the availability of Slurm versions 25.11.4 and 25.05.7.
by Marshall Garey 12 Mar '26

12 Mar '26

We are pleased to announce the availability of Slurm versions 25.11.4 and 25.05.7. Changes in both versions include preventing SIGHUP from terminating slurmrestd. Changes in 25.11.4 include fixing several issues with the switch/nvidia_imex plugin, several memory leaks, a race that could cause jobs to be stuck in the completing state or be requeued and held if the --get-user-env option was used, flex reservations taking nodes from other reservations that do not request full nodes, and a variety of other issues. Changes in 25.05.7 include fixing the gpu/nrt plugin, and fixing some other regressions and stability issues. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Server with two GPUs sharing each GPU to different partitions
by Gestió Servidors 09 Mar '26

09 Mar '26

Hello, First of all, sorry if my question is about something easy, but in my environment, this is the first time I need to do this. I have one server that have 2 identical GPUs. I have two different partitions: one is accessible from some allocation nodes and the other from other allocation nodes. My question is if it's possible to share one GPU with one partition and the other GPU with the other partition. For example, something like this: * gres.conf: NodeName=gpu-node AutoDetect=off Name=gpu Type=RTX5070 File=/dev/nvidia0 Cores=0-127 NodeName=gpu-node AutoDetect=off Name=gpu Type=RTX5070 File=/dev/nvidia1 Cores=128-255 * slurm.conf: NodeName=gpu-node CPUs=256 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=260000 TmpDisk=47000 Gres= RTX5070:2 PartitionName=gpu-int.q Nodes=gpu-node OverSubscribe=No State=UP AllocNodes=submit-node-1 PriorityTier=3 GraceTime=900 PartitionName=gpu-ext.q Nodes=gpu-node OverSubscribe=No State=UP AllocNodes=submit-node-1 PriorityTier=3 GraceTime=900 With this configuration, if simultaneously two different users tries to execute in one of the RTX5070 (one in gpu-int.q and the other in gpu-ext.q), could both of them execute with no problems? If user from gpu-int.q uses device #0 and user from gpu-ext.q uses device #1, could SLURM work without problems if a second simultaneously exection assigns device #1 to user from gpu-int.q and device #0 to user from gpu-ext.q? Thanks.

1 0

Optimizing CPU allocation in Slurm with hyperthreading enabled
by Guillaume COCHARD 02 Mar '26

02 Mar '26

Hello, The Slurm documentation states that "The default allocation method within a node is cyclic allocation (allocate available CPUs in a round-robin fashion across the sockets within a node)." In our case, we are using hyperthreading, which means that if two jobs requesting one CPU each arrive at the same time on an empty node, they will be assigned to the two threads of the same physical core. To improve performance, we would like Slurm to allocate CPUs (in our case, threads) in a way that prioritizes using one thread per physical core first, and only once all cores are occupied, assign the second thread of each core. This would be similar to the Least Loaded Node (LLN) approach, but instead of distributing jobs across nodes, it would distribute them across physical cores (Least Loaded Core). We do not want to disable hyperthreading; we simply want to optimize core usage when a machine is not fully loaded. We also want this to be a cluster-wide setting, and not a user and/or job setting. Our current slurm.conf contains: ConstrainCores = no SelectType = select/cons_tres SelectTypeParameters = CR_CPU_MEMORY And our cgroup.conf contains: ConstrainDevices=yes ConstrainCores=yes Does anyone know how to achieve this? Thanks, Guillaume

3 2

Missing mounts.h in Ubuntu Bionic build
by Michael Gutteridge 25 Feb '26

25 Feb '26

Hi We're running into a problem with building 25.11.3 for Ubuntu Bionic. When the cgroup/v2 plugin is building, ebpf throws an error: ``` ../../../../../src/plugins/cgroup/v2/ebpf.c:38:10: fatal error: linux/mount.h: No such file or directory #include <linux/mount.h> ^~~~~~~~~~~~~~~ compilation terminated. Makefile:638: recipe for target 'ebpf.lo' failed make[6]: *** [ebpf.lo] Error 1 make[6]: Leaving directory '/build/slurm-25.11.3/obj-x86_64-linux-gnu/src/plugins/cgroup/v2' ``` I think this is because on Ubuntu, the `mounts.h` header is in the path `sys`- the manpage for mounts(2) describes including this file via `#include <sys/mounts.h>`. `ebpf.c` seems to be the only place where the `linux/mounts.h` path is used, everywhere else using `sys`. Curiously it seems to build OK on Noble (Ubuntu 24). I have no explanation for the difference. Has anyone run into this problem building on Ubuntu? This include looks to have been a recent add so might be a bug, but I feel like someone else would have run into this already if it were. Thanks - Michael

1 0

Issue with coordinator increasing limits
by Groner, Rob 24 Feb '26

24 Feb '26

I'm seeing a weird issue when I try to increase limits on a child account that a coordinator has created. Here is the setup: 1. Create an account named slurm_parent, set coordinator1 as a coordinator of that account, and give that account a grptresmins=cpu=100 limit. 2. As coordinator1, create a child account named slurm_child whose parent is slurm_parent. 3. As coordinator1, set the grptresmins=cpu=10 limit on the child account. That all works fine. The problem is when I try to INCREASE the cpu limit on the child account by any amount. $coordinator1> sacctmgr modify account test_crch_slurmaccessch set grptresmins=cpu=11 sacctmgr: error: Coordinators can not increase job limits beyond the parent ones Modified account associations... Error with request: Coordinators can not increase job limits beyond the parent ones The first issue is that the limit of cpu=11 is NOT beyond the parent limit of cpu=100. The second issue is that (as a workaround), I can remove the cpu=10 limit on the child account as coordinator (by setting it to -1), and then I can set grptresmins=cpu=1000. That command works. So, I can very clearly set a grptresmins=cpu limit that is beyond the parent limit (I understand that the child account in reality won't be able to access more than the parent account, no matter what the limit is). Note that as coordinator, I can REDUCE the limit on the child account all I want. So I can set it to cpu=9 just fine. But if I try to set it back to 10, I'll again get the error saying I can't go above parent limits. So here are my issues: 1. Inability of coordinator to increase the child account grptresmins limits once set. 2. Incorrect error message about exceeding the parent limit. I originally saw this issue on 24.11.6, and built and tested on 25.11.3, and still see the issue. Thanks for any help figuring out what I'm doing wrong. Rob

1 1

Reliable, Atomic, Idempotent, or Transactional Job Submission?
by Adam Novak 24 Feb '26

24 Feb '26

I'm working on the Slurm integration in our Toil workflow runner project. I'm having a problem where an `sbatch` command to submit a job to Slurm can fail (with exit code 1 and message "sbatch: error: Batch job submission failed: Socket timed out on send/recv operation", in my case, but possibly in other ways), but the job can still actually have been submitted, and can still execute. This causes a problem for Toil because right now, when it sees a submission attempt fail, it backs off and submits the job again a little later. But Toil can't handle multiple copies of the same job running at once, and if a submission appears to the client to have failed but actually succeeded, it's possible to get into that situation if you just submit again. When an sbatch command fails, is it possible to detect cases where the cluster will still execute the job? (For example, is it guaranteed that the job ID will be available on standard output on the client if the job is going to execute on the cluster, no matter when in the client process it might get a socket disconnection, allowing the job to be inquired about later?) Do I maybe need to tag my jobs with unique identifiers myself so I can poll for them in the queue after a supposedly-failed submission? Is it possible to write an idempotent sbatch command, where it can be run any number of times but will only actually submit one copy of the job? Is the Slurm REST API somehow more transactional, or able to promise somehow that a job will not actually go into the queue without the client having acknowledged receipt of the job's assigned ID? Thanks, -Adam -- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code." Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5

4 7

Get the list of added dynamic nodes
by Edmundo Carmona Antoranz 23 Feb '26

23 Feb '26

Hi! Is there a way to get the list of dynamic nodes that have been created but that are pending to get slurmd to run on them? I fail to see them with with either scontrol or sinfo: root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol create NodeName=node004 .... state=FUTURE feature=dyn1 root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol version slurm 25.11.2 Only when I start slurmd from node004 does it become visible (through scontrol only): root@ea-u24-dynamic-2511:~# ssh node004 /cm/local/apps/slurm/current/sbin/slurmd -Z -f /cm/shared/apps/slurm/etc/slurm/slurm.conf root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 NodeName=node004 Arch=x86_64 CoresPerSocket=1 Thanks in advance. Edmundo Carmona Antoranz Software Engineer NVIDIA Corporation

3 4

We are pleased to announce the availability of Slurm version 25.11.3
by Tim Wickberg 19 Feb '26

19 Feb '26

We are pleased to announce the availability of Slurm version 25.11.3. 25.11.3 fixes a number of stability issues, regressions, and a variety of other bugs, including the following: * Fix multiple issues with dynamic topology when power saving is enabled * Fix a backfill performance regression when using bf_licenses * Fix race conditions when shutting down slurmctld or slurmdbd that cause crashes or hangs * Fix slurmctld crashing when using enable_async_reply and TLS * Fix regression from 25.05 which caused usercpu and systemcpu to be missing for job steps * Fix regression that caused thread limits in slurmd to not be respected for incoming RPCs * Prevent the double-removal of accounting usage for jobs being requeued that are in the COMPLETED or COMPLETING state The full list of changes is available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases One note on some updates to our release process - we've started publishing the releases directly on GitHub, in parallel to their usual location on the SchedMD web server. If you 'star' the Slurm repository you can automatically receive notices when new releases are published there. You can also subscribe to the RSS/Atom feed through: https://github.com/SchedMD/slurm/releases.atom - Tim -- Tim Wickberg Director, System Software, Slurm and Slinky, NVIDIA

1 0

preempt/qos is not working as expected
by Marchand Aurélia 19 Feb '26

19 Feb '26

Hello, I want too use QOS preemption. my configuration is : preemptMode = GANG,SUSPEND PreemptParameters = (null) PreemptType = preempt/qos PriorityType = priority/multifactor SelectType = select/cons_tres SelectTypeParameters = CR_CPU I define 3 partitions : PartitionName=DEFAULT Nodes=ALL Default=NO MaxTime=15-0 State=UP OverSubscribe=FORCE:1 PreemptMode=suspend PartitionName=veryhi PriorityTier=30 AllowQos=veryhi_short,gaia_veryhi_short PartitionName=hi PriorityTier=20 AllowQos=hi_short,gaia_hi_short PartitionName=def PriorityTier=10 OverSubscribe=No Default=YES AllowQos=def_short I define 5 QOS : sacctmgr show qos format=name%20,Priority,Preempt%40,PreemptMode Name Priority Preempt PreemptMode ----------------------- ------------ ---------------------------------- --------------- normal 0 cluster veryhi_short 20000 def_short suspend hi_short 10000 def_short suspend def_short 0 normal suspend gaia_veryhi_short 20000 def_short,gaia_hi_short suspend gaia_hi_short 10000 def_short suspend I define account : sacctmgr show ass format=account%20,user,qos%60 Account User QOS -------------------- -------- -------------------------------------------- gaia_hi user1 gaia_hi_short gaia_veryhi user1 gaia_veryhi_short mis_hi user1 hi_short def user1 def_short I submit 2 jobs : JOBID PARTITION QOS ACCOUNT NAME USER STATE PRIORITY 159 hi hi_short mis_hi test user1 SUSPENDE 637 160 veryhi gaia_veryhi_short gaia_veryhi test user1 RUNNING 1150 I don't understand why job 160 suspend job 159, the partition priority is higher, but I want gaia_veryhi_short QOS preempt only def_short and gaia_hi_short QOS. The QOS of job 159 is hi_short. Where is my configuration error ? Thanks for your time, Aurélia

2 4

for srun jobs, if we try to get UserCPU time by sacct. it always shows 0.
by squallabc＠126.com 16 Feb '26

16 Feb '26

Hi we have a slurm v25.05.2 setup. we found any srun jobs are always shows 0 on UserCPU . for sbatch jobs shoiws normal. is this expected or it is a bug ? Thanks. sacct -j 2948 --format=AveCPU,UserCPU AveCPU UserCPU ---------- ---------- 00:00:00 00:02:00 00:00:00

4 4

Sharding GPUs
by alessandr.dauria＠gmail.com 13 Feb '26

13 Feb '26

Dear Community, We are trying to activate sharding. Our Compute Node are configured with 64 cores, 4 phisical GPU MI250x ( 8 logical ) 4 Numa Domain. 1 Phisical Gpu / 2 logical GPU for each Numa Domain. 1 Logical GPU for each l3 cache domain gres.conf AutoDetect=rsmi NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD128 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD129 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD130 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD131 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD132 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD133 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD134 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD135 Count=4 If I ask 2 cores with block:cyclic I have the expected result srun -N1 -n2 -c1 --cpu-bind=cores -m block:cyclic --pty bash cpuset cgroup is 1,17 But if I add 2 shard in the request I don't expect this result srun -N1 -n2 -c1 --cpu-bind=cores --gres=shard:2 -m block:cyclic --pty bash cpuset cgroup is 1-2 ROCR_VISIBILE_DEVICES=0 Is it possibile request 2 sharding in round robin fashion, in order to run a multigpu job on different GPUs? srun -N1 -n2 -c1 --cpu-bind=cores --gres=shard:2 -m block:cyclic --pty bash Practically, I would to have this result cpuset cgroup is 1-17 ROCR_VISIBILE_DEVICES=0,1 Thank you in advance, Alessandro

1 1

Restarting slurmd kills still-running jobs
by Griebel, Christian 13 Feb '26

13 Feb '26

Dear community, Trying to implement the latest fix/patch for munged, we restarted the updated munged locally on the compute nodes with "systemctl restart munged", resulting in the sudden death of a lot of compute nodes' slurmd. Checking the jobs on the affected nodes, we saw a lot of user processes/jobs still running, which was good - yet "systemctl restart slurmd" cancelled all of them, eg. [2026-02-12T17:08:00.325] Cleaning up stray StepId=49695760.extern [2026-02-12T17:08:00.325] [49695760.extern] Sent signal 9 to StepId=49695760.extern [2026-02-12T17:08:00.325] slurmd version 25.05.5 started and all affected user jobs (even though having survived the death of their parent slurmd) were killed and re-queued... We have cgroups v2 (only, no hybrid), "Delegate=yes" in the slurmd unit and "ProctrackType=proctrack/cgroup" configured. Other sites do not see the same behavior (their user jobs survive a slurmd restart without issues), so now we are at a loss figuring out why the h.... this happens within our setup. Anyone experienced similar problems and got them solved...? Thanks in advance - -- ___________________________ Christian Griebel/HPC

6 5

sstat is not able check others job.
by squallabc＠126.com 12 Feb '26

12 Feb '26

Hi slurm version is v25.05.2. when I tried to query others job by sstat, it returns error. is there any configuration we can setup to fix this ? Thanks. sstat -j 3023.batch JobID AllocTRES MaxVMSize MaxVMSizeNode MaxVMSizeTask AveVMSize MaxRSS MaxRSSNode MaxRSSTask AveRSS MaxPages MaxPagesNode MaxPagesTask AvePages MinCPU MinCPUNode MinCPUTask AveCPU NTasks AveCPUFreq ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov ConsumedEnergy MaxDiskRead MaxDiskReadNode MaxDiskReadTask AveDiskRead MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask AveDiskWrite TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask TRESUsageInMin TRESUsageInMinNode TRESUsageInMinTask TRESUsageInTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot ------------ ---------- ---------- -------------- -------------- ---------- ---------- ---------- ---------- ---------- -------- ------------ -------------- ---------- ---------- ---------- ---------- ---------- -------- ---------- ------------- ------------- ------------- -------------- ------------ --------------- --------------- ------------ ------------ ---------------- ---------------- ------------ -------------- -------------- ------------------ ------------------ -------------- ------------------ ------------------ -------------- --------------- --------------- ------------------- ------------------- --------------- ------------------- ------------------- --------------- sstat: error: slurm_job_step_stat: there was an error with the request to atlsde8-reg0001 rc = Invalid user id sstat: error: problem getting step_layout for StepId=3023.batch: Invalid user id

3 2

MUNGE security issue (CVE-2026-25506)
by Tim Wickberg 12 Feb '26

12 Feb '26

MUNGE just announced a serious security issue as CVE-2026-25506: https://github.com/dun/munge/security/advisories/GHSA-r9cr-jf4v-75gh The issue is entirely within MUNGE - specifically in the munged daemon. However, as most Slurm installations explicitly trust MUNGE as the source of authentication, Slurm's security is immediately undermined if an attacker were to exploit that issue and gain access to the MUNGE key. On Slurm clusters you should treat that security issue as if it were local-root exploit. If you have switched to Slurm's built-in authentication ("auth/slurm"), you are not impacted by this issue. (However, we do not recommend switching over as a mitigation - you cannot safely change authentication types in Slurm while jobs are currently running.) I strongly encourage sites to get install the fixed packages from the upstream distros and restart the munged daemon on all nodes. If you install and restart munged promptly then Slurm should be unaffected. You do not need to rebuild Slurm for this issue. The MUNGE ABI/API remain unchanged, so there is no need to recompile Slurm, and there are no patches to apply to Slurm related to this issue. If you are concerned that an attacker could have compromised the MUNGE key itself, you may want to rotate that key. Unfortunately there's no mechanism to safely handle this key rotation with the cluster operational, and I do not recommend key rotation as long as you're installing and restarting MUNGE ASAP. - Tim -- Tim Wickberg Director, System Software, Slurm and Slinky, NVIDIA

11 16

Can I update when jobs are running?
by Gould, Ron (GRC-VBA0)[AEGIS] 11 Feb '26

11 Feb '26

The tl;dr is “This is my first upgrade since inheriting this Cluster, so I’m not sure what can or can’t be running during the upgrades.”. My Cluster is running an old version, 22.05.3. This is my first upgrade since inheriting the Cluster. As such, I’d like to install 22.05.4 because it’s a short jump, and it fixes the bug my users are seeing. The Cluster is composed of mostly Oracle Linux 8. I’m aware that I can upgrade within the two release compatibility window. I’ve read through the Upgrade guide and I’m unclear if downtime is required. Perhaps I’m unifying downtime requirements across different SLURM services where I should be interpreting that certain services have their own downtime requirements. https://slurm.schedmd.com/upgrades.html<https://slurm.schedmd.com/upgrades.html#procedure> In the Upgrade Procedure section<https://slurm.schedmd.com/upgrades.html#procedure>, there’re a couple questionable things. 1. Is downtime required? Does downtime == “all jobs must be halted”? “Downtime”, to me, seems like nothing should be running. This statement indicates that jobs can be running during the upgrade. Before considering the upgrade complete, wait for all jobs that were already running to finish. Any jobs started before the slurmd system was upgraded will be running with the old version of slurmstepd, so starting another upgrade or trying to use new features in the new version may cause problems. within a few paragraphs<https://slurm.schedmd.com/upgrades.html#downtime>, this message indicates I will need downtime: Refer to the expected downtime guidance in the following sections for each relevant Slurm daemon Further in the guide, in SLURMD (COMPUTE NODES)<https://slurm.schedmd.com/upgrades.html#slurmd>, I read Upgrades will not interrupt running jobs as long as SlurmdTimeout is not reached during the process This implies, at least, that existing running jobs can stay running. 2. There’re conflicting suggestions of using “rpm” to install the RPMs I built with “rpmbuild". Should I use “dnf localinstall ./*.rpm”? I’m inferring that dependencies will (not) be handled correctly. NOTE: If RPM/DEB packages are used, all packages present on each system must be upgraded together instead of piecewise. … Avoid using low-level package managers like rpm or dpkg as they may not properly enforce these dependencies However, in SLURMDBD (ACCOUNTING)<https://slurm.schedmd.com/upgrades.html#slurmdbd>, this statement Upgrade the slurmdbd daemon binaries, libraries, and its systemd unit file (if used). If using RPM/DEB packages, the package manager will take care of these indicates I should be using RPM packages. Lastly, to get to a current install, I need to step through multiple versions, with the condition that jobs started with a specific major version must finish within the compatibility window. GitLab has a tool where you plug in your current and intended versions and it tells you explicitly which versions are required along the upgrade path. I’d like a similarly explicit tool for SLURM, but I infer from the Compatibility Window<https://slurm.schedmd.com/upgrades.html#compatibility_window> that I can update like so: 1. Current = 22.05.3 2. 23.11 3. 25.05 4. 26.05 That feels like a big leapfrog between versions. I’d like the practice of upgrading. Is there any detriment to upgrading at a slower pace: 1. Current = 22.05.3 2. 22.05.11 3. 23.02.8 4. 23.11.11 5. 24.05.8 6. 24.11.7 7. 25.05.6 8. 25.11.2

9 26

Adding FPGAs for Slurm Tracking
by Raven Hallsby 08 Feb '26

08 Feb '26

Hey all, We have a very heterogeneous cluster with a mix of Intel & AMD machines, GPUs, and FPGAs. We use Slurm to allocate time on the CPUs right now, but do not manage the GPU or FPGAs with Slurm yet. Some of our work involves hardware/SoC design, and we have been using Firesim (https://fires.im/) for a good amount of our work. Firesim is a bunch of Python that sits on top of invoking programs to provide "seamless" functionality and orchestration. However, Firesim was designed for AWS where a single user controls all of the resources (FPGAs in our case) used for the simulation. We have gone through the effort to pull out the important bits from Firesim to write a shell script that we can execute locally to achieve the same thing as Firesim' orchestration, without the need for an orchestrator. We have two FPGAs in two separate but identical machines and multiple people that want to use them for very long periods of time (days or longer) and want to let people freely move between the two FPGAs. All of our other machines in the cluster use Slurm, so we are looking to have Slurm multiplex these FPGA jobs. We already have the NFS stuff set up, so shared disk is fine. The important thing in our desired Slurm setup is that no user is allocated the same FPGA at the same time for the same job (you cannot have multiple FPGA jobs running the same design in parallel, partly because of bitstreams, partly because of disk images). Slurm does not track FPGAs by default. I have looked into how to teach Slurm about the FPGAs and know that I need to tell Slurm about the FPGAs as a GRES through gres.conf. I just want to confirm that I have the correct steps below: 1. Mark the 2 FPGAs as GRES with gres.conf. What should they be marked as? Do I need anything special? 2. Then enable tracking on the FPGA GRES to make a TRES in slurm.conf. What should this be labeled as? I see other groups using GresTypes=gpu in their configuration. Would that apply here? Since FPGA usage is exclusive already, I could see the GPU GRES type applying. 3. Then the TRES can be used by a QoS to enforce limits? The other important question, will I need to write a Slurm plugin (GRES or otherwise) to manage and detect the FPGAs? I saw the files stuff for GRES-es, but want to be sure. Writing a plugin is not a big deal for us, but it is future maintenance that I would want to document.

1 0

Re: Dynamic MIG Question
by Davide DelVento 06 Feb '26

06 Feb '26

Aaron (or anyone else), Did you manage to get Dynamic MIG working in Slurm? I'm actually surprised that after these many years SchedMD has not implemented this feature yet, especially now that newer GPUs allow MIG repartitioning without being root. The only mention of this in their ticketing system is at https://support.schedmd.com/show_bug.cgi?id=11091#c8 (and subsequent c10) which say that it's not on their roadmap, but that was 5 years ago. I have heard that some users manage dynamic changes by draining nodes, running scripts to reconfigure MIG via nvidia-smi, bringing the node back and then submitting the job. Anybody here has tried that and with what success? I speculate that now that NVIDIA owns SchedMD perhaps this feature will be at a higher priority, but maybe not? Anybody knows anything about it and is not bound by an NDA to keep mum? Thanks On Wed, Nov 22, 2023 at 1:22 PM Davide DelVento <davide.quantum(a)gmail.com> wrote: > I assume you mean the sentence about dynamic MIG at > https://slurm.schedmd.com/gres.html#MIG_Management > Could it be supported? I think so, but only if one of their paying > customers (that could be you) asks for it. > > On Wed, Nov 22, 2023 at 11:24 AM Aaron Kollmann < > aaron.kollmann(a)student.hpi.de> wrote: > >> Hello All, >> >> I am currently working in a research project and we are trying to find >> out whether we can use NVIDIAs multi-instance GPU (MIG) dynamically in >> SLURM. >> >> For instance: >> >> - a user requests a job and wants a GPU but none is available >> >> - now SLURM will reconfigure a MIG GPU to create a partition (e.g. >> 1g.5gb) which becomes available and allocated immediately >> >> I can already reconfigure MIG + SLURM within a few seconds to start jobs >> on newly partitioned resources, but Jobs get killed when I restart slurmd >> on nodes with a changed MIG config. (see script example below) >> >> *Do you think it is possible to develop a plugin or change SLURM to the >> extent that dynamic MIG will be supported one day? * >> >> (The website says it is not supported) >> >> >> >> Best >> >> - Aaron >> >> >> >> >> #!/usr/bin/bash >> >> # Generate Start Config >> killall slurmd >> killall slurmctld >> nvidia-smi mig -dci >> nvidia-smi mig -dgi >> nvidia-smi mig -cgi 19,14,5 -i 0 -C >> nvidia-smi mig -cgi 0 -i 1 -C >> cp -f ./slurm-19145-0.conf /etc/slurm/slurm.conf >> slurmd -c >> slurmctld -c >> sleep 5 >> >> # Start a running and a pending job (the first job gets killed by slurm) >> srun -w gx06 -c 2 --mem 1G --gres=gpu:a100_1g.5gb:1 sleep 300 & >> srun -w gx06 -c 2 --mem 1G --gres=gpu:a100_1g.5gb:1 sleep 300 & >> sleep 5 >> >> # Simulate MIG Config Change >> nvidia-smi mig -i 1 -dci >> nvidia-smi mig -i 1 -dgi >> nvidia-smi mig -cgi 19,14,5 -i 1 -C >> cp -f ./slurm-2x19145.conf /etc/slurm/slurm.conf >> killall slurmd >> killall slurmctld >> slurmd >> slurmctld >> >

2 3

Can we avoid jobs near wall time from being preempted?
by iazaceta＠vicomtech.org 06 Feb '26

06 Feb '26

We have an HPC where the average job length is measured in days, not hours. Users are careful to add checkpoints to their jobs but even in that case, preempting a job that is close to its walltime (max: 14 days) can be very disruptive. I checked what options preemption offers but none seem to protect jobs near their finishing line. PreemptExempTime ensures a minimum job runtime and GraceTime allows for a grace time period after the job has been selected for preemption. Is there anything I am missing to achieve what I want? Thank you!

2 1

systemd unit ordering for dbd host
by Michael Gutteridge 05 Feb '26

05 Feb '26

We have a host dedicated to slurmdbd and mysql (Ubuntu 24.04, Slurm 24.11.4, mysql 8.0.45). A couple days ago unattended-upgrades upgraded the MySQL package which triggered a restart of mysql.service which looks to have caused some errors in slurmdbd. I don't think those errors were permanent, everything looks to be running as per usual currently. Enough backstory- the question is should we configure the slurmdbd systemd unit to stop slurmdbd before mysql is stopped? Currently there are minimal dependencies built into the unit: After=network-online.target remote-fs.target munge.service mysql.service mysqld.service mariadb.service sssd.service Wants=network-online.target But nothing to tell systemd to take down slurmdbd when the mysql unit is stopped. Is there *any* reason we'd want slurmdbd running if mysql wasn't? Are there any problems I'd cause by creating such a dependency? I do plan to exclude mysql from future unattended upgrading, but I'd also like to keep our systemd units sensible. Thanks - Michael

3 3

How to get strigger event information?
by Mauricio Pinto 03 Feb '26

03 Feb '26

Greetings, I am managing a cluster using SLURM, and want to use strigger to automate notifications about certain events like jobs or nodes failing or changing states, slurmctld failing, etc. I want the notifications I send to include information about the event, like which node, which state, which job ID, and so on. However, so far I haven't found documentation on how to get these values. I did find a related thread <https://groups.google.com/g/slurm-users/c/NhrlL32q0Io/m/HKVd9l1yDgAJ> in the google groups archive. One of the replies includes a script which uses environment variables such as $SLURM_JOB_ID and $SLURM_JOB_NAME. How are these environment variables available? Is there documentation on which environment variables are available to strigger scripts? I'd appreciate any guidance you can provide. On a separate note, I just joined this mailing list and haven't seen any community guidelines, preferred email format, etc. Please feel free to point them out to me or make suggestions if you have any. Thanks, Mauricio

2 2

Configless Slurm Error: failed to fetch remote configs
by Xaver Stiensmeier 30 Jan '26

30 Jan '26

Hey everyone, in the past we set up clusters with configs on each node. Now we want to explore configless. Without changing anything else, we therefore followed: https://slurm.schedmd.com/configless_slurm.html and added 'enable_configless' in the config on the master: SlurmctldParameters=cloud_dns,idle_on_node_suspend,enable_configless,reconfig_on_restart and start each worker's slurmd with the conf-server parameter: # Override systemd service to set conditional path [Service] ExecStart= ExecStart=/usr/sbin/slurmd --conf-server=master However, this leads to: slurmd: error: _fetch_child: failed to fetch remote configs: Protocol authentication error slurmd: error: _establish_configuration: failed to load configs. Retrying in 10 seconds. on the workers and on the master (/var/log/slurm/slurmctld) to: [2026-01-16T10:00:06.681] error: Munge decode failed: Invalid credential [2026-01-16T10:00:06.681] auth/munge: _print_cred: ENCODED: Thu Jan 01 00:00:00 1970 [2026-01-16T10:00:06.681] auth/munge: _print_cred: DECODED: Thu Jan 01 00:00:00 1970 [2026-01-16T10:00:06.681] error: slurm_unpack_received_msg: [[worker]:24295] auth_g_verify: REQUEST_CONFIG has authentication error: Unspecified error [2026-01-16T10:00:06.681] error: slurm_unpack_received_msg: [[worker]:24295] Protocol authentication error The munge key setup is the same as before so I don't think there is anything wrong with it unless something changes with configless (slurm.conf): AuthType=auth/munge CryptoType=crypto/munge AuthAltTypes=auth/jwt AuthAltParameters=jwt_key=/etc/slurm/jwt-secret.key I found https://groups.google.com/g/slurm-users/c/Q7FVkhx-bOs but this seems unrelated as both can talk fine with each other: worker:~$ nc -zv master 6817 Connection to master (192.168.20.169) 6817 port [tcp/*] succeeded! I tried adding more "-v" to the slurmd start, but that did not give more information. I am unsure how to debug this further. Somehow I think it must be a munge issue, but I am confused as this part hasn't changed. Best regards, Xaver

2 6

Nodes removed from topology after suspend in 25.11.2
by Franky Backeljauw 29 Jan '26

29 Jan '26

Hi all We have upgraded to version 25.11.2 and are now noticing that nodes are removed getting removed from the topology/tree when the node gets suspended. In the slurmctld.log we see things like this: [2026-01-29T23:31:32.517] POWER: power_save: suspending nodes node1 [2026-01-29T23:32:37.536] debug2: _update_node_features_post: nodes node1 active features set to: feature1 [2026-01-29T23:32:37.536] debug2: topology/tree: topology_p_add_rm_node: topology_p_add_rm_node: remove node1 from switch1 If we later on run a job with: srun -w node1 --pty bash then we see the following: [2026-01-30T00:08:04.004] debug2: NodeSet for JobId=1234 [2026-01-30T00:08:04.005] _pick_best_nodes: JobId=1234 never runnable in partition part1 [2026-01-30T00:08:04.005] sched: schedule: JobId=1234 non-runnable: Requested topology configuration is not available [2026-01-30T00:08:04.005] debug2: _schedule: setting JobId=1234 to "BadConstraints" (Requested topology configuration is not available) This worked without any issues in the previous version 25.05.2, so something has change in the newer version 25.11.2. My guess is something with dynamic nodes and/or active versus available features, but it’s just not clear :-( Ideas anyone??? Our setup includes in slurm.conf (trimmed): TopologyPlugin = topology/tree NodeName=node1 Features=feature1 NodeSet=set1 Feature=feature1 PartitionName=part1 Nodes=set1 [...] SuspendProgram = /power_save/suspend.sh ResumeProgram = /power_save/resume.sh ReconfigFlags = KeepPowerSaveSettings In topology.conf: SwitchName=switch1 Nodes=node1 The SuspendProgram and ResumeProgram scripts use tuned-adm to change the power profile (to powersave and hpc-compute resp.). When looking at the node itself, after suspend, it shows (trimmed): NodeName=node1 AvailableFeatures=feature1 ActiveFeatures=feature1 State=IDLE+POWERED_DOWN PARTITION=part1 I hope someone can shed a light on this issue... Thanks in advance for any reply! -- Kind regards Franky

1 0

sacct 25.11.2: --state filter returns no results
by Norman Gaywood 26 Jan '26

26 Jan '26

Hi all, I have recently upgraded my cluster to Slurm 25.11.2 (running on Rocky Linux 9). slurm-25.11.2-1.el9.x86_64 built locally. This time I built accounting in and so I'm new to sacct. When I run sacct without a state filter, I see the jobs correctly marked as COMPLETED: sacct -a -S 2026-01-26 --format=JobID,State | grep COMPLETED | head -10 9 COMPLETED 9.batch COMPLETED 9.extern COMPLETED 9.0 COMPLETED 39 COMPLETED 39.batch COMPLETED 39.extern COMPLETED 39.0 COMPLETED 40 COMPLETED 40.batch COMPLETED When I attempt to filter for those exact states (using either the short or long flag), the output is empty: sacct -a -S 2026-01-26 --format=JobID,State --state=completed JobID State ------------ ---------- (no output beyond the heading) Am I using it wrong? Cheers, Norm.

2 2

opposite of --array
by Herbert Fruchtl 21 Jan '26

21 Jan '26

Folks, When I use sacct, it always shows all tasks of an array job individually. This is supposed to be switched on by --array, but it seems to be the default. How do I switch it off? Thanks in advance, Herbert -- Herbert Fruchtl (he/him) Senior Scientific Computing Officer / HPC Administrator School of Chemistry, IT Services University of St Andrews -- The University of St Andrews is a charity registered in Scotland: No SC013532

2 3

building EL10 packages
by Brian Andrus 17 Jan '26

17 Jan '26

Hopefully someone has already busted through this wall... Currently trying to build slurm for the EL10 distros. Just tried 25.11.2 (been trying .1 as well) straight from tarball to rpms. The issue trying to build so far is that there seems to be no http-parser package in the repos (trying rocky 10 and centos stream 10). Ok, so I disable slurmrestd and rpmbuild runs. It still warns about http-parser missing, but packages come out. Now, when I try to run slurmctld/slurmdbd, I get an error: //usr/sbin/slurmctld: symbol lookup error: /usr/lib64/slurm/auth_slurm.so: undefined symbol: jwt_Base64encode / This is even on the build system... I've tried enabling developer mode even, with no luck. Starting to wonder if there is a breaking change in the EL10 distro line. Anyone have insight here? Thanks in advance, Brian Andrus

3 3

Best practice instructions and packages for Debian/Ubuntu full-upgrade?
by Steffen Grunewald 15 Jan '26

15 Jan '26

Good morning, long story: Every two years, some time after a release of Debian or Ubuntu LTS, when evaluations have ended and configurations adapted or rewritten and some reasonable maintenance date has been agreed upon by users' representatives (and directors!) and admins, we're in the situation to perform a full-upgrade (dist-upgrade) of all the machines that keep important data (while the compute nodes just get a fresh install). With the old 9-month release cycle of Slurm, in the past there was a slim chance that the upgrade from the version provided with the old release to the new release's one was small enough (since 3*9>24), but with the new 6-month cycle (4*6=24) there will always be one version too much in between. For a more or less painless transition, the manager machine running slurmdbd and slurmctld would need an intermediate version - which is not provided by Debian or Ubuntu at the time of the upgrade. Long story shorter: Provided you run Slurm on Debian/Ubuntu: Where do you get that necessary intermediate version from? I'll have to upgrade from 22.05.8-4+deb12u3 (Bookworm) to 24.11.5-4 (Trixie) (while I'm expecting the Slurm developers to urge me to proceed to 25.11.x, I couldn't find *any* deb packages for that, plus it would immediately break Ubuntu LTS integration since Noble is at 23.11.4-1.2ubuntu5). I'd need 23.11 for Debian (Bookworm *or* Trixie) for the half-step upgrade of slurmdbd's databases, and I'd like to bump Ubuntu's version to 24.11 to eventually add our GPU machines to the cluster at a later point. Long story short: Does someone keep those intermediate versions, and maybe builds ahead of the official distro ones? I can't be alone with this problem... Thanks for any pointer, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

4 3

We are pleased to announce the availability of Slurm versions 25.11.2 and 25.05.6.
by Marshall Garey 15 Jan '26

15 Jan '26

We are pleased to announce the availability of Slurm versions 25.11.2 and 25.05.6. Changes in 25.11.2 include fixing potential slurmctld state save file corruption while using the rpc_queue experimental feature, preventing PMIx 5.0.8 and 5.0.9 clients from hanging when connecting to the PMIx server, correctly expanding srun filename format identifiers, and various stability issues. Changes in 25.05.6 include fixing various stability issues. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Build 22.05.8 for Debian Trixie?
by Steffen Grunewald 15 Jan '26

15 Jan '26

Good afternoon, we're still running our HPC cluster on Debian 12 Bookworm, which comes with Slurm 22.05.8 - no issues so far. Yesterday, another machine was added as a submit node (slurm-client only), with Debian 13 Trixie / Slurm 24.11.5. Now on that the commands (sinfo etc.) don't work and the slurmctld logs "Incompatible versions of client and server code". Since upgrading the whole cluster is not an option yet: Has someone attempted (and possibly succeeded) to build 22.05.8 (or 23.11.4? which IIRC would be the last compatible one) for Trixie, and would be willing to share ideas? Thanks, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

4 6

Socket memory binding (NUMA). cpu-bind possible with srun, not sbatch
by Rémy Dernat 14 Jan '26

14 Jan '26

Hi, I am looking at that useful documentation https://slurm.schedmd.com/cpu_management.html We have people complaining about memory performance issues while running highly distributed jobs in a shared HPC cluster environment. After looking into it, we saw that is either because of concurrent access to the memory with other users, or because we have jump over memory from a socket to another (in a Numa architecture). Until now, our "easy" answer was to use "exclusive" mode in sbatch, to ensure avoiding any concurrent memory access, and using numactl to check if the job is bound to the other socket. However, I am looking to a better solution, as exclusive mode took all of a big node (we have only big nodes...), and numactl just check if we will have poor performances or not. I checked the slurm.conf parameters, but all I saw is about modifying selectType. However, I cannot use other selectType as we have GPU, so we are using cons_tres. Moreover, we already have a very complicated Slurm configuration, and I would like to avoid any side effect. I am thinking about talking to our user about the `--cpu-bind` (https://slurm.schedmd.com/srun.html#OPT_cpu-bind) method, but I am not sure how to use it, and it seems to be limited to srun ... ? From a job point of view, I know there are some programs to deal with that, like likwid or placement (but not entirely ?), but that does not seem easy to use, and a slurm would be more suitable and more generic. Do you have any idea on how to deal with these issues ? Thanks, Best regards, Rémy Dernat

1 0

slurmctld hourly: Unexpected missing socket error
by Jason Ellul 13 Jan '26

13 Jan '26

Hi all, I am hoping someone can help with our problem. Every hour after restarting slurmctld the controller becomes unresponsive to commands for 1 sec, reporting errors such as: [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934767]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934760]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934875]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934906]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[939016]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error It occurs consistently at around the hour mark, but generally not at other times, unless we run a reconfigure or restart the controller. We don’t see any issues in the slurmdbd.log and the errors are also always msg type RESPONSE. We have tried building a new server on different infrastructure, but the problem has persisted. Yesterday we even tried updating slurm to v24.05.1 in the hope that may provide a fix. During our troubleshooting we have: Set: * SchedulerParameters = max_rpc_cnt=400,sched_min_interval=50000,sched_max_job_start=300,batch_sched_delay=20,bf_resolution=600,bf_min_prio_reserve=2000,bf_min_age_reserve=600 * SlurmctldPort = 6808-6817 But although the stats in sdiag have improved we still see the errors. On our monitoring software we also see a drop in network and disk activity during this 1 second, always at approx. 1 hour after restarting the controller. Many Thanks in advance Jason Jason Ellul Head - Research Computing Facility Office of Cancer Research Peter MacCallum Cancer Centre

6 8

'sacct -a' missing running jobs 'held' by user
by Lee 08 Jan '26

08 Jan '26

Hello, I am using slurm 23.02.6. I have a strange issue. I periodically use sacct to dump job data. I then generate reports based on the resource allocation of our users. Recently, I noticed some 'missing' jobs from my query. The missing jobs came from a user who had a large array job, who then 'held' all of the array jobs. This included 'holding' the Running array jobs. Now, if I run `sacct -a -S YYYY-MM-DD --format="jobidraw,jobname"`, the job will be missing from that query. However, if I query specifically for that job, i.e. `sacct -j RAWJOBID -S YYYY-MM-DD --format="jobidraw,jobname", the job is present. *Question* : 1. How can I include the 'held' running job when I do my bulk query with `sacct -a`? Finding these outliers and adding them ad-hoc to my dumped file is too laborious and isn't feasible. *Minimum working example *: #. Submit a job : myuser@clusterb01:~$ srun --pty bash # landed on dgx29 #. Hold job myuser@clusterb01:~$ scontrol hold 120918 myuser@clusterb01:~$ scontrol show job=120918 JobId=120918 JobName=bash UserId=myuser(123456) GroupId=rdusers(7000) MCS_label=N/A Priority=0 Nice=0 Account=allusers QOS=normal JobState=*RUNNING* Reason=*JobHeldUser* Dependency=(null) Requeue=0 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:00:29 TimeLimit=7-00:00:00 TimeMin=N/A SubmitTime=2025-12-15T13:31:28 EligibleTime=Unknown AccrueTime=Unknown StartTime=2025-12-15T13:31:28 EndTime=2025-12-22T13:31:28 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-12-15T13:31:28 Scheduler=Main Partition=defq AllocNode:Sid=clusterb01:4145861 ReqNodeList=(null) ExcNodeList=(null) NodeList=dgx29 BatchHost=dgx29 NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=9070M,node=1,billing=1 AllocTRES=cpu=2,mem=18140M,node=1,billing=2 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=9070M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=bash WorkDir=/home/myuser Power= #. Release job myuser@clusterb01:~$ scontrol release 120918 #. Show job again myuser@clusterb01:~$ scontrol show job=120918 JobId=120918 JobName=bash UserId=myuser(123456) GroupId=rdusers(7000) MCS_label=N/A Priority=1741 Nice=0 Account=allusers QOS=normal JobState=*RUNNING* Reason=*None* Dependency=(null) Requeue=0 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:01:39 TimeLimit=7-00:00:00 TimeMin=N/A SubmitTime=2025-12-15T13:31:28 EligibleTime=Unknown AccrueTime=Unknown StartTime=2025-12-15T13:31:28 EndTime=2025-12-22T13:31:28 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-12-15T13:31:28 Scheduler=Main Partition=defq AllocNode:Sid=clusterb01:4145861 ReqNodeList=(null) ExcNodeList=(null) NodeList=dgx29 BatchHost=dgx29 NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=9070M,node=1,billing=1 AllocTRES=cpu=2,mem=18140M,node=1,billing=2 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=9070M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=bash WorkDir=/home/myuser/ Power= #. In slurmctld, I see : root@clusterb01:~# grep 120918 /var/log/slurmctld [2025-12-15T13:31:28.706] sched: _slurm_rpc_allocate_resources JobId=120918 NodeList=dgx29 usec=1269 [2025-12-15T13:31:47.751] sched: _hold_job_rec: hold on JobId=120918 by uid 123456 [2025-12-15T13:31:47.751] sched: _update_job: set priority to 0 for JobId=120918 [2025-12-15T13:31:47.751] _slurm_rpc_update_job: complete JobId=120918 uid=123456 usec=189 [2025-12-15T13:32:48.081] sched: _release_job_rec: release hold on JobId=120918 by uid 123456 [2025-12-15T13:32:48.081] _slurm_rpc_update_job: complete JobId=120918 uid=123456 usec=268 [2025-12-15T13:33:20.552] _job_complete: JobId=120918 WEXITSTATUS 0 [2025-12-15T13:33:20.552] _job_complete: JobId=120918 done #. Job is NOT missing, when identifying it by jobid myuser@clusterb01:~$ sacct -j 120918 --starttime=2025-12-12 -o "jobidraw,jobid,node,start,end,elapsed,state,submitline%30" JobIDRaw JobID NodeList Start End Elapsed State SubmitLine ------------ ------------ --------------- ------------------- ------------------- ---------- ---------- ------------------------------ 120918 120918 dgx29 2025-12-15T13:31:28 2025-12-15T13:33:20 00:01:52 COMPLETED srun --pty bash 120918.0 120918.0 dgx29 2025-12-15T13:31:28 2025-12-15T13:33:20 00:01:52 COMPLETED srun --pty bash #. Job IS *missing* when using -a myuser@clusterb01:~$ sacct -a --starttime=2025-12-12 -o "jobidraw,jobid,node,start,end,elapsed,state,submitline%30" | grep -i 120918 ## *MISSING* Best regards, Lee

3 4

New, faster, simpler makefiles.
by Alex Tingle 08 Jan '26

08 Jan '26

Hello, I've written a set of replacement makefiles for Slurm, which are faster, and simpler than Automake. Faster incremental builds make the code much more pleasant to work with. Try it out here: https://github.com/alex-smartptr/slurm Instructions: 1. Run `./configure` as usual (all settings work as before). 2. `make -s` You can find a full README here: https://github.com/alex-smartptr/slurm/blob/newmake/mk/README.md I'm looking for feedback, especially problems that I've not foreseen, but also suggestions for improvements. -Alex Tingle --

1 0

Announcing slurm-quota: CPU and GPU time quotas for Slurm
by Rémi Palancher 08 Jan '26

08 Jan '26

Hello Slurm users, I would like to announce slurm-quota, an open-source tool providing time-based CPU and GPU quota accounting and enforcement for Slurm users and accounts. Slurm does not currently provide native enforcement of consumption-based quotas (e.g. CPU or GPU minutes). slurm-quota addresses this by integrating with Slurm job submission and job completion mechanisms. Key features include: - Definition and enforcement of CPU and GPU time quotas at user and account levels - Support for heterogeneous GPU clusters using billing factors - Clear command-line reporting of usage, remaining quota, with visual progress bars - Lightweight design with minimal dependencies (Python, SQLite and few common Lua libraries) slurm-quota is released under the MIT license and is available on GitHub, along with documentation describing architecture, deployment, and integration details. GitHub Project: https://github.com/rackslab/slurm-quota Comments, questions, and feedback are very welcome! -- Rémi Palancher Rackslab: Open Source Solutions for HPC Operations https://rackslab.io

2 2

pam_slurm_adopt malfunct after slurm upgrade to 25.11
by taleintervenor＠sjtu.edu.cn 16 Dec '25

16 Dec '25

Hello everyone: We recently upgrade slurm to 25.11 and find the pam_slurm_adopt got broken. This cause the users cannot ssh to the compute node where their jobs is running on. Of course we have make sure all the slurm related packages have been upgraded together. Simplest test process is like: ``` > cat test1.sh #!/bin/bash #SBATCH --job-name=test #SBATCH --partition=debug #SBATCH --nodes=1 #SBATCH --nodelist=cas639 sleep 6000 > sbatch test1.sh Submitted batch job 51047091 > squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 51047091 debug test tester R 0:28 1 cas639 > ssh cas639 (wait for a long time...) Connection closed by 172.16.3.129 port 22 (finally failed to ssh) ``` on the target compute node, we can see the following debug message from pam_slurm_adopt.so: ``` cas639 pam_slurm_adopt[1007301]: debug2: _establish_config_source: using config_file=/etc/slurm/slurm.conf (default) cas639 pam_slurm_adopt[1007301]: debug: slurm_conf_init: using config_file=/etc/slurm/slurm.conf cas639 pam_slurm_adopt[1007301]: debug: Reading slurm.conf file: /etc/slurm/slurm.conf cas639 pam_slurm_adopt[1007301]: PreemptMode=GANG is a cluster-wide option and cannot be set at partition level, option ignored. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/auth_munge.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge authentication plugin type:auth/munge version:0x190b00 cas639 pam_slurm_adopt[1007301]: debug: auth/munge: init: loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/certgen_script.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Certificate generation script plugin type:certgen/script version:0x190b00 cas639 pam_slurm_adopt[1007301]: debug: certgen/script: init: loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/hash_k12.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:KangarooTwelve hash plugin type:hash/k12 version:0x190b00 cas639 pam_slurm_adopt[1007301]: debug: hash/k12: init: init: KangarooTwelve hash plugin loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/tls_none.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Null tls plugin type:tls/none version:0x190b00 cas639 pam_slurm_adopt[1007301]: debug: tls/none: init: tls/none loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/accounting_storage_slurmdbd.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Accounting storage SLURMDBD plugin type:accounting_storage/slurmdbd version:0x190b00 cas639 pam_slurm_adopt[1007301]: accounting_storage/slurmdbd: init: Accounting storage SLURMDBD plugin loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/cred_munge.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x190b00 cas639 pam_slurm_adopt[1007301]: cred/munge: init: Munge credential signature plugin loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug: Reading cgroup.conf file /etc/slurm/cgroup.conf cas639 pam_slurm_adopt[1007301]: debug4: found StepId=51047091.extern cas639 pam_slurm_adopt[1007301]: debug4: found StepId=51047091.batch cas639 pam_slurm_adopt[1007301]: Connection by user tester: user has only one job 51047091 cas639 pam_slurm_adopt[1007301]: debug: _adopt_process: trying to get StepId=51047091.extern to adopt 1007301 cas639 pam_slurm_adopt[1007301]: debug: Leaving stepd_add_extern_pid cas639 pam_slurm_adopt[1007301]: debug: Leaving stepd_get_x11_display cas639 pam_slurm_adopt[1007301]: debug: entering stepd_get_namespace_fd ``` It looks like something block during stepd_get_namespace_fd? And we found nothing in slurmd log even with SlurmdDebug = debug5, so I guess the pam module had not run to the step to talk with slurmd (if it should). Would it be a compatibility problem between slurm25.11 and EL8 system or cgroup/v1? Or can anyone help to give some suggestion on how to further locate the fault point? Best regards, Hermes

4 6

Slurm version 25.11.1 is now available
by Marshall Garey 16 Dec '25

16 Dec '25

We are pleased to announce the availability of Slurm version 25.11.1. This release fixes two critical bugs when upgrading which caused older jobs to be aborted and slurmstepd to crash, a regression in 25.11 that broke pam_slurm_adopt, some stability issues in slurmctld, and various other minor to moderate bugs. The full list of changes are available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

SchedMD joins forces with NVIDIA
by Danny Auble 15 Dec '25

15 Dec '25

Today marks a new journey in the history of SchedMD - we are joining forces with NVIDIA. NVIDIA is committed to developing and supporting Slurm as the open-source vendor-neutral workload manager it has always been. For more details, please see NVIDIA's blog post: https://blogs.nvidia.com/blog/nvidia-acquires-schedmd/ Danny

1 0

Slurm 25.05 and 25.11 release video, slides from recent presentations
by Tim Wickberg 12 Dec '25

12 Dec '25

The release video covering what's new in Slurm 25.05 and 25.11 is online now on the SchedMDSlurm YouTube channel: https://youtu.be/gk42j2_kxFg The is the same content that we presented at the Slurm Community Birds-of-a-Feather session at SC'25 last month, but hopefully in a more broadly accessible location. The slides are in the publication archive as usual, as well as the other recent presentations from SC'25 and KubeCon NA 2025: https://www.schedmd.com/publications/ The direct link to the release sides is: http://slurm.schedmd.com/SC25/Slurm_BoF_SC25.pdf - Tim -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

GPU usage in fair share factor
by Massimo Sgaravatto 12 Dec '25

12 Dec '25

Dear all I have these settings in my slurm.conf: PriorityType=priority/multifactor PriorityDecayHalfLife=7-0 PriorityFavorSmall=NO PriorityMaxAge=10-0 PriorityWeightAge=100000 PriorityWeightFairshare=1000000 PriorityWeightJobSize=100000 PriorityWeightPartition=100000 PriorityWeightQOS=100000 PriorityFlags=ACCRUE_ALWAYS,FAIR_TREE AccountingStorageTRES=gres/gpu,gres/gpu:nvidia-h100,gres/gpu:nvidia-l40s If I have got it right, with these settings, when calculating the Job_priority (I am referring to the formula in https://slurm.schedmd.com/priority_multifactor.html) the fair-share_factor is calculated only considering cores*seconds. So if a job also used some GPUs, this is not taken into account. Am I right ? If I want to take memory and GPUs into account as well (with different weights for different GPU models), my understanding is that I should use the TRESBillingWeights attribute, setting e.g: TRESBillingWeights="CPU=1.0,Mem=0.25G,gres/gpu=nvidia-h100=10.0,gres/gpu=nvidia-l40s=5.0" Is this correct ? Thanks, Massimo

2 4

Problems with salloc
by Gestió Servidors 10 Dec '25

10 Dec '25

Hello, I am getting some problems with "salloc" execution. My cluster is running SLURM v23.11.0. I'm going to explain problems: 1. My first problem is that until some previous version (I can't remember what version exactly), when I ran a "salloc", I got a console inside the assigned host. Then, I could execute whatever I wanted. However, now, after execute "salloc" (without forcing host), SLURM always tries to assign me the last host where a job has been executed. But if that host is now powered off, salloc waits and waits but not tries to connect to other host 2. Second problem is that previous execution gives me a host (the host is powered on), my session continues in the allocation host, not in the assigned host. So if I want to execute some program, I need to run with "srun". Is there anyway to go directly to the assigned host? 3. Third problem is a question. How can I limit the hosts where I (as cluster admin) allow to run a "salloc"? For example, I would like that when a user runs a "salloc", it were execute always in a partition with "x" hosts, avoiding SLURM to assign me a host of the list of hosts. Thanks.

1 0

Invalid generic resource (gres) specification after RMA
by Lee 04 Dec '25

04 Dec '25

Good afternoon, I have a cluster that is managed by Base Command Manager (v10) and it has several Nvidia DGXs. dgx09 is a problem child. The entire node was RMA'd and now it no longer behaves the same as my other DGXs. I think the below symptoms are caused by a single underlying issue. *Symptoms : * 1. When I look at our 8 non-MIG DGXs, via `scontrol show node=dgxXY | grep Gres`, 7/8 DGXs report "Gres=gpu:*H100*:8(S:0-1)" while dgx09 reports " Gres=gpu:*h100*:8(S:0-1)" 2. When I submit a job to this node, I get : $ srun --reservation=g09_test --gres=gpu:2 --pty bash srun: error: Unable to create step for job 105035: Invalid generic resource (gres) specification ### No job is running on the node, yet AllocTRES shows consumed resources... $ scontrol show node=dgx09 | grep -i AllocTRES *AllocTRES=gres/gpu=2* ### dgx09 : /var/log/slurmd contains no information ### slurmctld shows : root@h01:# grep 105035 /var/log/slurmctld [2025-11-13T07:44:56.380] sched: _slurm_rpc_allocate_resources JobId=105035 NodeList=dgx09 usec=3420 [2025-11-13T07:44:56.482] _job_complete: JobId=105035 WTERMSIG 1 [2025-11-13T07:44:56.483] _job_complete: JobId=105035 done *Configuration : * 1. gres.conf : # This section of this file was automatically generated by cmd. Do not edit manually! # BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE AutoDetect=NVML NodeName=dgx[01,02] Name=gpu Type=1g.20gb Count=32 AutoDetect=NVML NodeName=dgx[03-10] Name=gpu Type=h100 Count=8 AutoDetect=NVML # END AUTOGENERATED SECTION -- DO NOT REMOVE 2. grep NodeName slurm.conf root@h01:# grep NodeName slurm.conf NodeName=dgx[01,02] RealMemory=2063937 Boards=1 SocketsPerBoard=2 CoresPerSocket=56 ThreadsPerCore=2 MemSpecLimit=30017 Gres=gpu:1g.20gb:32 Feature=location=local NodeName=dgx03-10] RealMemory=2063937 Boards=1 SocketsPerBoard=2 CoresPerSocket=56 ThreadsPerCore=2 MemSpecLimit=30017 Gres=gpu:H100:8 Feature=location=local 3. What slurmd detects on dgx09 root@dgx09:~# slurmd -C NodeName=dgx09 CPUs=224 Boards=1 SocketsPerBoard=2 CoresPerSocket=56 ThreadsPerCore=2 RealMemory=2063937 UpTime=8-00:39:10 root@dgx09:~# slurmd -G slurmd: gpu/nvml: _get_system_gpu_list_nvml: 8 GPU system device(s) detected slurmd: Gres Name=gpu Type=h100 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-55 CoreCnt=224 Links=-1,0,0,0,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-55 CoreCnt=224 Links=0,-1,0,0,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0-55 CoreCnt=224 Links=0,0,-1,0,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=0-55 CoreCnt=224 Links=0,0,0,-1,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=56-111 CoreCnt=224 Links=0,0,0,0,-1,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=5 ID=7696487 File=/dev/nvidia5 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,-1,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=6 ID=7696487 File=/dev/nvidia6 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,0,-1,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=7 ID=7696487 File=/dev/nvidia7 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,0,0,-1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML *Questions : * 1. As far as I can tell, dgx09 is identical to all my non-MIG DGX nodes in terms of configuration and hardware. Why does scontrol report it having 'h100' with a lower case 'h' unlike the other dgxs which report with an upper case 'H'? 2. Why is dgx09 not accepting GPU jobs and afterwards it artificially thinks that there are GPUs allocated even though no jobs are on the node? 3. Are there additional tests / configurations that I can do to probe the differences between dgx09 and all my other nodes? Best regards, Lee

6 12

Doubts Regarding Definition of Metrics AveRSS and AveDiskWrite
by Manuel G. Marciani 03 Dec '25

03 Dec '25

Hello, fellow users: I have been using Slurm for the past three years, but recently I bumped into a doubt. I am using Slurm's (version 23.02.7) collected metrics (jobacct_gather/linux) to do a performance analysis of an application. I have read the documentation regarding the metrics (https://slurm.schedmd.com/sacct.html) but still find the Ave* metrics confusing, and more specifically the AveRSS and AveDiskWrite. AveDiskWrite is defined as "Average number of bytes written by all tasks in job." So, if I double the workload, which say that it had x avediskwrite, I should observe 2x. So far it is what I observed. But, then, if I double the resources while maintaining the workload I observe again x, and not 2x. So my suspicion is that the metric is the sum of written bytes across time, then divided by the number of nodes. But then with AveRSS, defined as "Average resident set size of all tasks in job," I observe what I expected with AveDiskWrite. That is, that this metric scales with the workload irrespective of the resources it has available. So I am not sure what the "Ave" references here. I would be thankful if someone could clarify the behavior, and even more grateful if someone could point me where in the code these metrics are aggregated and processed to be stored in the database. Many thanks, Manu. -- Manuel G. Marciani - マルシアニ·マヌエル First Stage Reseacher at Computational Earth Sciences (CES) - Earth Sciences Department Barcelona Supercomputing Center - Centro Nacional de Supercomputación Ph.D Student at Departament d'Arquitectura de Computadors (DAC) - Facultat d'Informàtica de Barcelona (FIB) Universitat Politècnica de Catalunya (UPC) BSC building Plaça Eusebi Güell, 1-3, 08034 Barcelona, Spain Desk BSC-PL0-6-20/8 mail to: manuel.gimenez(a)bsc.es mail to: manuel.gimenez.de.castro(a)upc.edu mail to: manuel.marciani(a)a.riken.jp mail to: manuel.gimenez.de.castro.marciani(a)hu-berlin.de

1 0

MIG and eBPF issues (Slurm 24.11.6)
by wk5ng＠uwaterloo.ca 28 Nov '25

28 Nov '25

Hi all, I'm having some trouble getting Slurm 24.11.6 to work with MIG, and the slurmd logs seem to point to an issue with eBPF. For some context, this is an LXD unprivileged container where I'm trying to get MIG to work with Slurm. Other compute nodes without MIG work fine and isolate the GPUs accordingly. What I'm seeing in slurmd logs: [2025-11-24T23:32:50.197] [331.interactive] cgroup/v2: cgroup_p_constrain_apply: CGROUP: EBPF Closing and loading bpf program into /sys/fs/cgroup/system.slice/slurmstepd.scope/job_331 [2025-11-24T23:32:50.197] [331.interactive] error: load_ebpf_prog: BPF load error (Operation not permitted). Please check your system limits (MEMLOCK). I've tried increasing the system limits for MEMLOCK by setting DefaultLimitMEMLOCK=infinity in /etc/systemd/system.conf, and I've copied my slurmd.service file below where I've set Delegate=yes and LimitMEMLOCK=infinity. Previously only Delegate=yes wasn't set (I had rifled through the cgroupv2 documentaton for Slurm and found that setting), but in both cases I see the same BPF load error. Just wondering if this was something that other people had come across before and maybe I'm doing something silly here. I've checked that my slurm.conf has the corresponding parameters set according to Slurm's own documentation for cgroup.conf and my cgroup.conf is also copied below. Some portion of the gres.conf is also copied below, and even though I tried AutoDetect=nvml for this node, it's still doesn't work, which was why I changed to manually setting it based off the output of slurmd -G. Maybe I should try switching back to cgroupv1 and see if that helps fix things, but I'm not sure at this point if MIG and Slurm are compatible using cgroupv1. I can send other parts of logs, configuration files etc. Any help would be greatly appreciated! ###### slurmd.service file [Unit] Description=Slurm node daemon After=network.target munge.service ConditionPathExists=/etc/slurm/slurm.conf [Service] Type=forking EnvironmentFile=-/etc/sysconfig/slurmd ExecStart=/usr/sbin/slurmd -d /usr/sbin/slurmstepd $SLURMD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID PIDFile=/var/run/slurmd.pid KillMode=process LimitNOFILE=51200 Delegate=yes LimitMEMLOCK=infinity LimitSTACK=infinity [Install] WantedBy=multi-user.target ###### cgroup.conf CgroupPlugin=autodetect ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ##### gres.conf NodeName=gpu-3 AutoDetect=nvml Name=gpu NodeName=gpu-4 Name=gpu MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap30,/dev/nvidia-caps/nvidia-cap31

3 4

Slinky version 1.0.0 is now available
by Marrlow Warnicke 20 Nov '25

20 Nov '25

We are pleased to announce the availability of Slinky version 1.0.0! Slinky is SchedMD's set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.0.0: https://github.com/SlinkyProject/slurm-operator/tree/release-1.0 New features include hybrid support, integration points for external tooling, and workload protection and isolation. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge handles the cases where you want Slurm scheduling on your cluster and to be able to run either Kubernetes or Slurm jobs. Release v1.0.0: https://github.com/SlinkyProject/slurm-bridge/tree/release-1.0 New features include support for DRA Extended Resources, support for TaintToleration and VolumeBinding plugins, and integration of new Slurm 25.11 support for granular node resource allocation assignments with GRES. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… The SlinkyProject registry now has containers that support both amd64 (x86_64) and arm64 (aarch64) architectures. You may find these here: https://github.com/orgs/SlinkyProject/packages Apologies for the redundant email-we needed to clarify the release version. -- Marlow Warnicke Principal Cloud Engineer Commercial Slurm - and Slinky - Development and Support

1 0

Slinky version 1.0.0 is now available
by Marrlow Warnicke 20 Nov '25

20 Nov '25

We are pleased to announce the availability of Slinky version 1.0.0-rc1. Slinky is SchedMD’s set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-operator/tree/release-1.0 New features include hybrid support, integration points for external tooling, and workload protection and isolation. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge handles the cases where you want Slurm scheduling on your cluster and to be able to run either Kubernetes or Slurm jobs. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-bridge/tree/release-1.0 New features include support for DRA Extended Resources, support for TaintToleration and VolumeBinding plugins, and integration of new Slurm 25.11 support for granular node resource allocation assignments with GRES. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… The SlinkyProject registry now has containers that support both amd64 (x86_64) and arm64 (aarch64) architectures. You may find these here: https://github.com/orgs/SlinkyProject/packages -- Marlow Warnicke Principal Cloud Engineer, SchedMD LLC Commercial Slurm - and Slinky - Development and Support

1 0

Recommended Stable Slurm Version for >100P Scale Clusters
by KK 18 Nov '25

18 Nov '25

We are currently planning to deploy a new HPC system with a total compute capacity exceeding 100 PF. As part of our preparation, we would like to understand which Slurm versions are considered stable and widely used at this scale. Could you please share your recommendations or experience regarding: 1. Which Slurm version is currently running reliably on very large-scale clusters (>100 PF or >10k nodes)? 2. Whether there are any versions we should avoid due to known issues at large scale. 3. Any best practices or configuration considerations for Slurm deployments of this size.

3 2

Slurm versions 25.05.5 and 24.11.7 are now available
by Marshall Garey 11 Nov '25

11 Nov '25

We are pleased to announce the availability of Slurm versions 25.05.5 and 24.11.7. These releases fix some mild to moderate severity issues and potential crashes. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md https://github.com/SchedMD/slurm/blob/slurm-24.11/CHANGELOG/slurm-24.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Split node into 2 partitions
by Ratnasamy, Fritz 11 Nov '25

11 Nov '25

Hi, Is there a straightforward way to split a gpu node with 8 GPUs into 2 partitions with 4 GPUs on each? How do we edit the slurm.conf and gres.conf in this case? Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Slurm Community Survey 2025
by Tim Wickberg 10 Nov '25

10 Nov '25

With the Slurm 25.11 release out, and the 1.0.0rc1 for the Slinky Project done, we're quickly shifting into conference season. SchedMD staff are presenting at KubeCon North America on Slinky [1] this week. We don't have a booth here, but feel free to say hi if you see any of us, or send me a message on the Kubernetes or CNCF Slack channels (@wickberg) if you have something you'd like to discuss in person. Next week we'll be manning the Slurm Booth at SC25 [2], as well as hosting the annual Slurm Community Birds-of-a-Feather session [3] on Thursday from 12:15-1:15pm. For this year we wanted to to make the Slurm Community BoF survey available ahead of time, and to open it to a wider audience. This'll let us prepare some initial results ahead of the BoF (while still trying to update live data during the BoF), and the results from this are invaluable as we plan for future Slurm releases. The survey is available now: https://schedmd.com/survey For those not at SC25, we'll have a brief set of highlights from this survey included in the Slurm 25.11 release overview video on our YouTube channel in December. - Tim [1] https://kccncna2025.sched.com/event/27FW5/ [2] The Slurm Booth is #1641. We have a new halo banner this year as well. [3] https://sc25.conference-program.com/presentation/?id=bof101&sess=sess471 [4] https://www.youtube.com/SchedMDSlurm -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

sreport and AccountUtilizationByUser
by Gestió Servidors 08 Nov '25

08 Nov '25

Hello, I woud like to get a full report that includes CPU usage of an account (detalied by user). I have run "sreport cluster AccountUtilizationByUser start=9/1 end=11/2 account=mygr" and the output is this: -------------------------------------------------------------------------------- Cluster/Account/User Utilization 2025-09-01T00:00:00 - 2025-11-01T23:59:59 (5360400 secs) Usage reported in CPU Minutes -------------------------------------------------------------------------------- Cluster Account Login Proper Name Used Energy --------- --------------- --------- --------------- --------- -------- mycluster mygr 3979368 0 mycluster mygr mygr-1 3979368 0 mycluster mygr mygr-10 3979368 0 mycluster mygr mygr-11 3979368 0 mycluster mygr mygr-12 3979368 0 mycluster mygr mygr-13 3979368 0 mycluster mygr mygr-14 3979368 0 mycluster mygr mygr-15 3979368 0 mycluster mygr mygr-16 3979368 0 mycluster mygr mygr-17 3979368 0 mycluster mygr mygr-18 3979368 0 mycluster mygr mygr-19 3979368 0 mycluster mygr mygr-2 3979368 0 mycluster mygr mygr-20 3979368 0 mycluster mygr mygr-21 3979368 0 mycluster mygr mygr-22 3979368 0 mycluster mygr mygr-23 3979368 0 mycluster mygr mygr-24 3979368 0 mycluster mygr mygr-25 3979368 0 mycluster mygr mygr-26 3979368 0 mycluster mygr mygr-27 3979368 0 mycluster mygr mygr-28 3979368 0 mycluster mygr mygr-29 3979368 0 mycluster mygr mygr-3 3979368 0 mycluster mygr mygr-30 3979368 0 mycluster mygr mygr-31 3979368 0 mycluster mygr mygr-32 3979368 0 mycluster mygr mygr-33 3979368 0 mycluster mygr mygr-34 3979368 0 mycluster mygr mygr-35 3979368 0 mycluster mygr mygr-36 3979368 0 mycluster mygr mygr-37 3979368 0 mycluster mygr mygr-38 3979368 0 mycluster mygr mygr-39 3979368 0 mycluster mygr mygr-4 3979368 0 mycluster mygr mygr-40 3979368 0 mycluster mygr mygr-5 3979368 0 mycluster mygr mygr-6 3979368 0 mycluster mygr mygr-7 3979368 0 mycluster mygr mygr-8 3979368 0 mycluster mygr mygr-9 3979368 0 I don't understand why column "Used" shows always the same value, because each user "mygr-XX" has used the cluster in differents ways, times and each used has submitted very different number of jobs, as you can see in this "sshare" output: [root@login bin]# sshare -l -A mygr Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS -------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- mygr 1 0.250000 3941705 0.337402 0.961033 0.260137 mygr mygr-1 1 0.024390 32099 0.002748 0.008144 0.095146 2.995008 mygr mygr-10 1 0.024390 98343 0.008418 0.024949 0.067961 0.977586 mygr mygr-11 1 0.024390 213838 0.018304 0.054250 0.052427 0.449588 mygr mygr-12 1 0.024390 134149 0.011483 0.034033 0.056311 0.716657 mygr mygr-13 1 0.024390 89304 0.007644 0.022656 0.071845 1.076537 mygr mygr-14 1 0.024390 524836 0.044925 0.133150 0.040777 0.183179 mygr mygr-15 1 0.024390 303494 0.025979 0.076996 0.044660 0.316774 mygr mygr-16 1 0.024390 99020 0.008476 0.025121 0.066019 0.970897 mygr mygr-17 1 0.024390 296341 0.025366 0.075181 0.046602 0.324421 mygr mygr-18 1 0.024390 37809 0.003236 0.009592 0.085437 2.542696 mygr mygr-19 1 0.024390 72158 0.006177 0.018306 0.077670 1.332337 mygr mygr-2 1 0.024390 225658 0.019316 0.057249 0.050485 0.426039 mygr mygr-20 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-21 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-22 1 0.024390 115042 0.009847 0.029186 0.062136 0.835683 mygr mygr-23 1 0.024390 54169 0.004637 0.013743 0.081553 1.774775 mygr mygr-24 1 0.024390 125767 0.010765 0.031907 0.058252 0.764422 mygr mygr-25 1 0.024390 243088 0.020808 0.061671 0.048544 0.395490 mygr mygr-26 1 0.024390 35516 0.003040 0.009010 0.089320 2.706902 mygr mygr-27 1 0.024390 65281 0.005588 0.016562 0.079612 1.472682 mygr mygr-28 1 0.024390 118203 0.010118 0.029988 0.060194 0.813335 mygr mygr-29 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-3 1 0.024390 107543 0.009206 0.027284 0.064078 0.893956 mygr mygr-30 1 0.024390 33946 0.002906 0.008612 0.091262 2.832099 mygr mygr-31 1 0.024390 183059 0.015670 0.046442 0.054369 0.525180 mygr mygr-32 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-33 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-34 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-35 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-36 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-37 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-38 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-39 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-4 1 0.024390 37605 0.003219 0.009541 0.087379 2.556486 mygr mygr-40 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-5 1 0.024390 81114 0.006943 0.020578 0.073786 1.185232 mygr mygr-6 1 0.024390 91873 0.007864 0.023308 0.069903 1.046428 mygr mygr-7 1 0.024390 80513 0.006892 0.020426 0.075728 1.194082 mygr mygr-8 1 0.024390 367822 0.031485 0.093316 0.042718 0.261374 mygr mygr-9 1 0.024390 33630 0.002879 0.008532 0.093204 2.858713 mygr mygr-tutor 1 0.024390 40472 0.003464 0.010268 0.083495 2.375427 What am I doing wrong? What I need to get is a detalied report from all "mygr" users (mygr account) from CPU usage and, if it is possible, other values. Thanks.

2 1

Slinky version 1.0.0rc1 is now now available
by Marrlow Warnicke 07 Nov '25

07 Nov '25

We are pleased to announce the availability of Slinky version 1.0.0-rc1. Slinky is SchedMD’s set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-operator/tree/release-1.0 New features include hybrid support, integration points for external tooling, and workload protection and isolation. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge handles the cases where you want Slurm scheduling on your cluster and to be able to run either Kubernetes or Slurm jobs. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-bridge/tree/release-1.0 New features include support for DRA Extended Resources, support for TaintToleration and VolumeBinding plugins, and integration of new Slurm 25.11 support for granular node resource allocation assignments with GRES. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… The SlinkyProject registry now has containers that support both amd64 (x86_64) and arm64 (aarch64) architectures. You may find these here: https://github.com/orgs/SlinkyProject/packages -- Marlow Warnicke Principal Cloud Engineer, SchedMD LLC Commercial Slurm - and Slinky - Development and Support

1 0

submit host DNS lookup errors stall slurmctld, fail nodes, kill jobs
by moorehfl＠amazon.com 06 Nov '25

06 Nov '25

As we've scaled up our slurm usage, we've noticed that short, moderate bursts of DNS lookup failures are enough to regularly stall slurmctld: _xgetaddrinfo: getaddrinfo(runner-t35knco5d-project-54-concurrent-0:37251) failed ...this has a cascading effect where, when stalled, the controlled can't always communicate with nodes: error: Error connecting, bad data: family = 0, port = 0 ...and immediately the controller will mark the nodes as unhealthy, and kill jobs: slurmctld: Killing JobId=3120751 on failed node slurm-0f6cacdc1 The reason for the DNS failures is not unreliable DNS server or network, but rather the jobs are submitted by containers that don't have resolvable hostnames. This traditionally hasn't disrupted functionality, but we've noticed if 8-10 jobs all terminate at the same time (submitter container SIGTERMs the srun process) that the controller can be easily overloaded for several seconds, despite having significant free system resources. gdb confirms the process is hanging on DNS. We also can see "Socket timed out on send/recv operation" from clients attempting to interact with the controller during the issue. slurm 24.11.0 RHEL 8.10 kernel 4.18.0-553.58.1.el8_10.x86_64 We're looking into ways to get our ephemeral job submitter containers resolvable in DNS to prevent lookup failures (either by giving them resolvable hostnames, or by blackholing the records to 0.0.0.0 to allow for fast local failure on the slurmctld server). However, it does seem unusual for a handful of bad DNS lookups to cause so much disruption in slurmctld. Is this a known weak point of ctld? The slurmctld host is a single-purpose 16 vCPU 30GB EC2 instance with minimal load. We have ~150 nodes, all nodes have valid IPs in slurm.conf to remove the need for ctld to perform lookups for nodes, but apparently there is still a need to lookup the submit host as well, and we can reliably reproduce these cascading failures. Another possiblity might be to extend SlurmdTimeout to something very long and hope that the controller recovers from its stall in enough time to prevent from marking nodes as unhealthy and killing jobs, but it's not clear if that will have any effect since the first occurrence of "error: Error connecting, bad data: family = 0, port = 0" immediately drains nodes and kills jobs. Thanks

1 0

Slurm version 25.11 is now available
by Marshall Garey 06 Nov '25

06 Nov '25

We are pleased to announce the availability of Slurm 25.11. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md The Slurm documentation has also been updated to the 25.11 release: https://slurm.schedmd.com Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

CPU frequency setting not configured for this node
by Felix 06 Nov '25

06 Nov '25

Hello I get this error: "CPU frequency setting not configured for this node" in some of my node. I do not know what to do, can you please advice. Thank you Felix -- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

1 0

Move to higher qos when cluster idle
by Ratnasamy, Fritz 06 Nov '25

06 Nov '25

Hi, Is there an option to automatically have users move to a higher qos when the cluster is x% idle? I was reading about customizing the job_submit.lua script in order to do that but I was wondering if it makes sense. We have a gpu cluster and many users complain they can not allocate more than their current qos when the cluster is not very busy. Best, *Fritz Ratnasamy*Data Scientist Information Technology

1 0

Slurm release candidate version 25.11.0rc1 is available for testing
by Tim Wickberg 28 Oct '25

28 Oct '25

We are pleased to announce the availability of Slurm release candidate 25.11.0rc1. To highlight some new features coming in 25.11: * Added new "Expedited Requeue" mode for batch jobs. Batch jobs with --requeue=expedite will automatically requeue on node failure, or if the batch script returns a non-zero exit code and one or more Epilog scripts fail. Expedited requeue jobs are eligible to restart immediately, are treated as the highest priority job in the system, and their previously allocated set of nodes will be prevented from launching other work. * Added a new "Mode 3" of operation to Hierarchical Resources. This mode complements the existing Mode 1 and Mode 2 by summing usage from lower levels automatically. This can be used, e.g., to implement a power-capping mode modeling power distribution between the datacenter, local distribution, and individual racks. * Added direct support for exporting OpenMetrics (Prometheus) telemetry from slurmctld. This is accessible on SlurmctldPort on SlurmctldHost by default, or can be disabled if desired. * Added an experimental asynchronous-reply mode to slurmctld. If enabled with "SlurmctldParameters=enable_async_reply", RPC responses are offloaded to the kernel for further processing, freeing individual worker threads for new traffic. This is the first release candidate of the upcoming 25.11 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 25.11.x version and we will address them before the first production 25.11.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/download-slurm/. The changelog for 25.11 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-25.11.md -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

AuthInfo broken in 25.05.1 ?
by michael＠mayer.cx 27 Oct '25

27 Oct '25

I am using the AuthInfo in both slurm.conf and slurmdbd.conf to point to a different location for the munge socket. This works as expected in slurmdbd, but it does not work for slurmctld. In slurmctld it seems to ignore this setting and try to locate the munge socket at the default location and then fail. When using the same approach in 23.11.11, for example, AuthInfo is working as expected for both slurmctld and slurmdbd. Sample config from slurm.conf and slurmdbd.conf AuthType=auth/munge AuthInfo=socket=/my/non-default/location/of/munge.socket I'd be grateful for any pointer whether this is a genuine bug/regression or not. Many thanks, Michael.

3 3

Possible to require jobs to use NVLink-ed pairs of GPUs?
by Marcus Lauer 27 Oct '25

27 Oct '25

One of our researchers asked whether it was possible to require a job to use NVLink-ed pairs of GPUs. I see that there is a support ticket on the SchedMD site which covers this (https://support.schedmd.com/show_bug.cgi?id=15995) That ticket is a few years old though. Does anyone happen to know whether support for this has been added in newer releases of SLURM? The cluster in question does use "AutoDetect=nvml" in its gres.conf and the output of "slurmd -G" shows that SLURM is aware of the NVLink pairs. I assume the scheduler is trying to use that information. What I want to know is whether there is some way for an end-user to add a constraint (for example) to a job such that it only runs on an NVLink-ed pair of GPUs. I do know that there are other ways to implement this such as requiring jobs to run with even numbers of GPUs, perhaps just on some nodes to allow single GPU jobs to run on the remaining nodes. I'm specifically asking about a flag or setting a user could apply to their jobs. If there is such a thing maybe someone here knows about it. If so I'd love to hear about it. Thanks! -- *Mr. Marcus Lauer* Systems Administrator Penn Engineering University of Pennsylvania https://www.seas.upenn.edu/

1 0

Limit number of allocated GPUs
by Gestió Servidors 23 Oct '25

23 Oct '25

Hello, I have three nodes, serving each one 2 GPUs. I would like to limit (qos??) that a user could user only one GPU from earch server, but user could user simultaneously three GPUs if each GPU belongs to different servers. With this QoS "sacctmgr add qos test-limit-GPUs MaxJobsPerUser=3 MaxTRESPerUser=gres/gpu=1" I can limit to one GPU, but then user can't run other job in a GPU from other server. How must I configure QoS (or other method) to allow more than one job requesting GPUs but never in the same server? Thanks.

3 2

Job remains "PENDING" with reason "QOSMaxGRESPerUser"
by Gestió Servidors 23 Oct '25

23 Oct '25

Hello, I have added a new "qos" with these parameters: sacctmgr add qos test-GPUs MaxJobsPerUser=6 MaxTRESPerUser=gres/gpu=1 MaxSubmitJobsPerUser=25. With it, I only allow 6 running jobs per user, a total of 25 pending+running job per user and only 1 GPU. I have applied this qos directly to a partition in slurm.conf. When a user submits to that partition requesting 2 or more GPUs, job remains "PD" (pending) and notifies "QOSMaxGRESPerUser" in NODELIST column, but I would like to know if it would be possible to direcly reject job and avoid that job remains at queue? For example, if I submit 50 jobs, after number 25 I get message "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) sbatch: error: QOSMaxSubmitJobPerUserLimit" 25 times) Thanks.

2 1

dynamic node slurm node list order
by Assmann, Greta Marie 23 Oct '25

23 Oct '25

Hello, we would like to understand how the internal SLURM node list order works. More detailed info: Our setup: We have a Slurm cluster with N dynamic nodes (heterogeneous node types) in a partition and we observe that if all resources are free, jobs always get submitted to one specific node first. This node is also the node that turns up at the top of the list when doing an $ scontrol show nodes . As we do not have a defined node list in slurm.conf as it is done for non-dynamic nodes, we were wondering how the order is set up. Interestingly, when deleting (and unregistering) this node from the cluster and re-registering it again, the node is still at the same position when doing scontrol show nodes. Is there some internal node list order caching or similar? How is the node list order defined ? Thanks , Greta Dr. Greta Assmann Data Analysis and Research Infrastructure OBBA 230 Forschungsstrasse 111 5232 Villigen-PSI email: greta.assmann(a)psi.ch

4 5