- slurm-users - lists.schedmd.com

Configurator tool on Slurm site is not found
by Valerio Bellizzomi 11 Jun '26

11 Jun '26

Please check. VB

1 0

Visit the Slurm & NVIDIA Booths at ISC 2026
by Victoria Hobson 10 Jun '26

10 Jun '26

If you're heading to ISC in Hamburg, Germany (June 22–26), come find us on the floor! Stop by the Slurm booth (Z10) for sessions on HPC workload management, Kubernetes, and more: Tuesday, June 23 12:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 17:00 – What's New: Latest Slurm and Slinky Release 19:00 – Slinky: Slurm for Kubernetes 101 Wednesday, June 24 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 14:00 – Slinky: Slurm for Kubernetes 101 15:00 – What's New: Latest Slurm and Slinky Release 17:00 – Redefining Resiliency for HPC Research and AI Factories: Slurm and NVIDIA Mission Control Thursday, June 25 10:00 – Slinky: Slurm for Kubernetes 101 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement Then head over to the NVIDIA booth (E30) for even more great content and conversations. We hope to see you there! Victoria Hobson | NVIDIA Campaign Manager, CUDA and Slurm

1 0

Use of scontrol update PartitionName
by Richard Lefebvre 09 Jun '26

09 Jun '26

Hi, To adapt to changing users need, we change which nodes are in what partitions. We use the command: scontrol update PartitionName=partition01 Nodes=nd[001-200] and modify slurm.conf at the same time to match. But when I try to just modify slurm.conf and restart the process "slurmctld". The command "scontrol show PartitionName=partition01" still show the old list of nodes. Even after I reboot the slurmctld server completely. I don't know where the cached version of the list is or how it keeps it. Is there a some slurm.conf configuration that dictated that behavior? Any hints? Richard

2 1

We are pleased to announce the availability of Slurm 26.05.1
by Marshall Garey 09 Jun '26

09 Jun '26

We are pleased to announce the availability of Slurm 26.05.1. Some of the changes in 26.05.1 are highlighted below: * Fix a significant performance regression in slurmctld. * Fix a database performance regression handling step completion records. * Fix an incompatibility between 26.05 slurmd and 25.05+ slurmstepd when using a 24.11 sattach that would cause the sattach to hang. The full list of changes are available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Preemptible qos
by Steen Lysgaard 04 Jun '26

04 Jun '26

Hello, We have a recurring situation where some users would like to use idle cluster resources on weekends. Their normal CPU limits prevent this, which is generally desirable during the work week to ensure fair sharing. A pattern we're considering is a special low-priority QOS that allows users to exceed their normal association limits, and that is preemptible by jobs submitted with the normal QOS. The goal is to let heavy users fill otherwise idle nodes over weekends without impacting smaller users when activity picks up again on Monday. The walltime limit (50 hours) is not something we would like to change. Has anyone implemented something similar? If so, I'd be interested in hearing about experiences and any pitfalls to this approach. Thanks, Steen Lysgaard [http://www.dtu.dk/-/media/DTU_Generelt/Andet/mail-signature-logo.png] Steen Lysgaard Scientific Software Developer DTU Energy Mob. +45 51 94 48 86 stly(a)dtu.dk<mailto:stly@dtu.dk> Agnes Nielsens Vej Building 301 2800 Kgs. Lyngby www.dtu.dk/english<https://www.dtu.dk/english>

2 1

Authentication
by Sterner, Douglas E. 02 Jun '26

02 Jun '26

New kubernetes deployment and I can't authenticate via ssh using AD credentials to the login controller. Both id and getent resolve user correctly. I have tried every sssd parameter known to mankind. Can someone provide a working example of an sssd.conf for active directory non ssl that I can compare against. Also do I need to do this in the values.yaml? slurm: config: # Explicitly enable PAM for node environments and pam_slurm_adopt UsePAM: 1 Thanks, Douglas Sterner Johns Hopkins Applied Physics Laboratory NSAD / JAF 11100 Johns Hopkins Road. Laurel, MD 20723-6091 [cid:image001.png@01DCF28D.A50E98A0]

2 1

Packing or spreading single-CPU jobs
by Gerhard Strangar 02 Jun '26

02 Jun '26

Hello, I'd like to somehow achieve this: Group A of nodes runs jobs that are supposed to be packed, i.e. if each node has 16 cpus, pack 16 jobs on the first node before using node 2. But for group B, assign every job their own node until all of them have one job, then put a second one on each node until all of them have 2 ... Ir number of jobs doesn't work, load or available memory would do as well, but it seems there's no way of telling slurm to sort the possible nodes by some criteria. Gerhard

2 1

Purging of job_script and job_env from the database in 26.05
by Ole Holm Nielsen 01 Jun '26

01 Jun '26

If you are using AccountingStoreFlags=job_script and/or job_env in slurm.conf to store job scripts in the Slurm database, you may be interested in a highly useful feature introduced in Slurm 26.05: Previously, the stored job_scripts and job_env entries would accumulate without bounds in the database, potentially causing database performance or size issues, especially while upgrading Slurm. With 26.05 there are some new slurmdbd.conf options which can be used to purge job_script or job_env entries in the database, see ticket_23818 [1]. This new feature isn't mentioned at all in the Release notes [4], and is only inconspicuously mentioned way down in the Changes section [3] as: > Adding new archive/purge options to allow for explicit archiving of job_scripts and job_env without jobs. The safe way to enable the new purge parameters is to introduce them very gradually, for example: PurgeJobScriptAfter=2000days PurgeJobEnvAfter=2000days and lower the values little by little over time (restarting slurmdbd each time). The MaxPurgeLimit [5] parameter may alternatively be used to limit the amount of database work. Best regards, Ole [1] https://support.schedmd.com/show_bug.cgi?id=23818 [2] https://slurm.schedmd.com/slurm.conf.html#OPT_AccountingStoreFlags [3] https://github.com/SchedMD/slurm/releases [4] https://slurm.schedmd.com/release_notes.html [5] https://slurm.schedmd.com/slurmdbd.conf.html#OPT_MaxPurgeLimit -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

2 3

Large overhead for --task-epilog
by Alan O'Cais 29 May '26

29 May '26

Hi all, I've been experimenting with task prolog/epilog and noticed that task epilogs have significant overhead. The overhead seems to be independent of the epilog script content: $ cat test.sh #!/bin/bash $ time srun --overlap --task-prolog=$PWD/test.sh --ntasks=64 sleep 1 real 0m1.730s user 0m0.211s sys 0m0.009s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=2 sleep 1 real 0m3.530s user 0m0.213s sys 0m0.007s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=64 sleep 1 real 1m5.936s user 0m0.216s sys 0m0.005s This test was on one node, with 64 tasks in the allocation. The overhead seems linked only to number of tasks per node, as adding additional nodes did not increase the overhead. I saw this on $ srun --version slurm 23.11.10-BullSequana.1.2.1 and someone else reproduced similar results with |slurm 25.05.7|. I can live with the overhead, but I am curious as to why it happens and to understand the design of the implementation. Does the task epilog script only run when all tasks finish, or does it run immediately at exit for each task? Are the calls serialised? My use case is a fusermount that I want to mount in userspace at task start and clean up at task finish, one task per node is responsible for the mount/unmount and I want to understand if I have a race condition for the unmount. Any advice very much appreciated, Alan

1 0

How to weight nodes properly based on RAM to prevent small instances clogging big instances with backfilling
by Xaver Stiensmeier 28 May '26

28 May '26

Dearl Slurm User List, I am currently reviewing a slurm.conf where the developer set Weight manually to attribute a greater weight to machines that have more RAM to force smaller jobs on smaller instances. However, I feel like there is something already in place or better than manually setting the weights, but I couldn't find it. If I understand correctly Slurm does not schedule jobs to the smallest possible node on default. So small jobs can be scheduled to large instances and a big job might have to wait indefinitely when using backfilling. I thought that Slurm does have mechanisms to prevent this but was unable to find it again in the documentation. Is there really no automatism at place or am I overlooking something? Best, Xaver

2 1

Technical Update: Native Slurm Simulation Environment Successfully Configured
by sami sami 28 May '26

28 May '26

Hi All, I am currently working on a single Linux machine running *Ubuntu 25.10* and am looking to conduct performance testing on the *Slurm backfill algorithm*. My objective is to simulate a cluster environment consisting of *1,024 nodes*. While researching existing solutions, I found that many modern Slurm-simulator implementations (such as various plugin-based designs) rely heavily on *Docker and Docker Compose*. However, I would like to avoid using containerization and instead run the simulation natively on my host OS. I understand that older versions of the Slurm source code included a native sim directory that allowed for simulations without external dependencies like Docker. Given this, I have the following technical questions: - *Native Compatibility:* Is it still feasible to compile and run a Slurm simulator natively on Ubuntu 25.10? - *Version Recommendations:* Would you recommend using a specific legacy version of Slurm that still contains the original simulation files, or is there a way to port that functionality to a more recent release? - *Configuration:* Are there specific build flags or configuration steps required to enable the simulated node environment (1,024 nodes) on a single machine without triggering the overhead of containerized networking? I would appreciate any guidance or documentation you could provide on achieving a high-node-count simulation in a "bare-metal" Linux environment. Best regards,

3 2

GPU fallen of the bus
by Ratnasamy, Fritz 27 May '26

27 May '26

Hello, We are noticing that some of the gpus on a specific node have "fallen of the bus". We would like to remove this specific gpu from the slurm scheduler. For example, let's say GPU0 has fallen off the bus, we would need the rest of the GPU1-8 to be available and make GPU0 not able to be allocated. How can we achieve that? I have read about blacklist on the slurm forum but it seems there is no satisfying solution. Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Slurm version 26.05 is now available
by Marshall Garey 26 May '26

26 May '26

We are pleased to announce the availability of Slurm 26.05. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md The Slurm documentation has also been updated to the 26.05 release: https://slurm.schedmd.com Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Core Specialization Feature
by Dan Healy 21 May '26

21 May '26

I'm curious how to report to our users that the number of available CPU cores on a given node are *(CoresPerSocket * SocketsPerBoard * ThreadsPerCore) - CpuSpecCount*. In a basic example, if the node has 32 cores and CpuSpecCount=1, then there's actually 31 cores available. If a user runs `srun -n32 -N1 bash` he/she will be denied for resources. Then, If a user runs `sinfo -O cpus,nodes` shows there's 32 cores available and the user is confused. How can the user clearly identify that in the above case, the exact number of available CPU cores available are 31 and *not* 32? -- Thanks, Daniel Healy

2 2

cgroup error in SLURM version 25.11.2
by bradford.kerr＠srs.gov 19 May '26

19 May '26

I'm trying to install and run SLURM on a stand-alone (no network access) cluster of four PCs running Red Hat Linux 9. On the first compute node of the cluster, I can not get past the error message "cannot create cgroup context for cgroup/v2". It appears that file cgroup_v2.so does not exist. How do I get past this error?

1 0

We are pleased to announce the availability of Slurm versions 25.11.6 and 25.05.8.
by Marshall Garey 14 May '26

14 May '26

We are pleased to announce the availability of Slurm versions 25.11.6 and 25.05.8. Fixes in these releases primarily cover various stability issues when reconfiguring or shutting down and improved support for newer libraries and compilers. The 25.11.6 release also includes a performance fix for scheduling with topology/block and a regression from 25.11.4 that could cause a "No records archived...but some found" message when archiving job data. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

inconsistent 'Requested node configuration is not available' error
by Paul Raines 14 May '26

14 May '26

In my cluster right now we have: $ sinfo ... pubgpu-req up 7-00:00:00 7 mix cobra,l40s-[01,03],luisa,rtx-03,sevilla,shelob pubgpu-req up 7-00:00:00 5 idle fiber,glaurung,gothmog,l40s-02,leo The following works fine: $ srun -p pubgpu-req -A sysadm --nodelist=cobra,l40s-02 --gres=gpu:1 -N 1 \ --ntasks-per-node=1 --mem=1G --time=1:00:00 --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 cobra[0]:~$ exit exit However just change the order of the nodelist and you get $ srun -p pubgpu-req -A sysadm --nodelist=l40s-02,cobra --gres=gpu:1 -N 1 \ --ntasks-per-node=1 --mem=1G --time=1:00:00 --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 srun: error: Unable to create step for job 8255390: Requested node configuration is not available More experimentation with this and it appears nodelist HAS to be in alphabetical order $ srun -p pubgpu-req -A sysadm --nodelist=l40s-02,leo,fiber \ --gres=gpu:1 -N 1 --ntasks-per-node=1 --mem=1G --time=1:00:00 \ --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 srun: error: Unable to create step for job 8255401: Requested node configuration is not available $ srun -p pubgpu-req -A sysadm --nodelist=fiber,l40s-02,leo \ --gres=gpu:1 -N 1 --ntasks-per-node=1 --mem=1G --time=1:00:00 \ --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 fiber[0]:~$ Surely there is no good reason for this? --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

2 1

SLURM config option to not tie up a host completely.
by Ron Gould 08 May '26

08 May '26

I have a user asking about a SLURM config setting that would allow a job to not tie up a node's unused resources, thereby allowing another job to run concurrently. User: On our cluster, our Fluent GPU jobs each use only 1 of the 2 GPUs on GPUServer[1,2]. Currently the node is fully allocated to whichever job lands on it first, leaving the second GPU, 38 CPU cores, and 270+ GB of RAM idle. Has anyone dealt with this? What options would facilitate this? Are there any gotchas or pitfalls?

4 3

Moving a Slurm database from one slurm database host to another
by James A Allsopp 08 May '26

08 May '26

Hi, We'd like to move our Slurm database from its existing machine running mariadb-server-5.5.68 to another running mariadb-server-10.5.29-3, both of which are running slurm 24.11.7. We dumped the database successfully from the old server, reimported it into the new server using mysqldump and *mysql < db_dump.sql*. The new slurmdbd service was then started. The clusters have different names so there were a few inconsistencies, but when we started the new slurm cluster, we could see the jobs according to sacct (using the -M option) matched those of the original cluster, but none of the associated job steps were shown. I manually logged into the new database and could see the jobs in the job_table but it looked like the foreign key link had been broken in the step table as there were entries in that table. Am I missing some information about how to export and import properly? I was wondering if the import had triggered auto-increment on keys and that was breaking things? Ideally, we'd like to change the name of the cluster as well. Is that just a matter of altering the prefix on the relevant tables, which is something I can do by hand on the database dump before reimporting? For instance, changing the table <old_name>_event_table to <new_cluster_name>_event_table. Thanks James -- Dr James Allsopp | Research Platforms team IT Services | University of Sheffield I do not work Tuesday and Wednesday 3-5pm. I also send emails at odd times, but respond at your convenience.

2 1

Slurm release candidate version 26.05.0rc1 is available for testing
by Tim Wickberg 07 May '26

07 May '26

We are pleased to announce the availability of Slurm release candidate 26.05.0rc1. To highlight some new features coming in 26.05: - New "srun --async" step mode that submits step processes to stepmgr to queue and eventually launch. This avoids issues with keeping a large number of srun processes backgrounded to queue up future step-based workflows. (This was previously described as "mini-batch" in the Slurm roadmap presentations.) - New topology/ring and topology/torus3d topology plugins. These implement a single-dimensional and three-dimensional topology respectively. - Slurm REST API - Add support for creating/updating/removing partitions, and for viewing the active slurmctld and slurmdbd configurations. - Dynamic Memory Resizing. A job can now release memory (and have the cgroup limits updated) using 'scontrol update' while running. A new "sbatch --mem-update=<margin>@<delay>" option can also automatically reduce the memory limit to the current usage plus a given margin percentage after a specified time. - Add topology-based sorting for node ranks when using dynamic nodes with the topology plugins. This is also available generically for topology/flat (no topology) with a new alpha_step_rank option. - Add an optimized single-node path through the scheduling logic for increased performance. - Expanded the openmetrics (Prometheus) nodes, partitions, and jobs endpoints with gpu allocation statistics. This is the first release candidate of the upcoming 26.05 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://support.schedmd.com against the 26.05.x version and we will address them before the first production 26.05.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://github.com/SchedMD/slurm/releases. The changelog for 26.05 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-26.05.md -- Tim Wickberg Director, System Software, Slurm + Slinky, NVIDIA

1 0

cgroup/v2 regression in 25.11.2 — memory.peak read disabled after first task
by Emyr James 06 May '26

06 May '26

Dear all, Heads-up for anyone running 25.11.2 with cgroup/v2 + jobacct_gather/cgroup: there's a regression in the recent series that effectively disables memory.peak accounting for every task after the first one per slurmstepd. The commit is f2d05c08e6 ("cgroup/v2 - Remove static qualifier from variable", ticket 23646, cherry-picked from e1835a1602). It dropped static from memory_peak_interface in cgroup_p_task_get_acct_data while leaving interfaces_checked static. The function now looks like: bool memory_peak_interface = false; static bool interfaces_checked = false; ... if (!interfaces_checked) { memory_peak_interface = cgroup_p_has_feature(CG_MEMCG_PEAK); interfaces_checked = true; } ... if (memory_peak_interface) { common_cgroup_get_param(..., "memory.peak", ...); } interfaces_checked persists across calls; memory_peak_interface no longer does. So the feature-probe block runs exactly once. From the second call onward memory_peak_interface is a fresh stack-local set to false, and the memory.peak read in the second if is skipped. stats->memory_peak ends up INFINITE64 for every task except the first one ever sampled. User-visible effect: MaxRSS in sacct stops reflecting the kernel-tracked peak and falls back to polling memory.current — which, at typical jobacct_gather intervals, routinely misses short-lived peaks. Which is of course exactly the problem memory.peak was added to solve (see the 2024 thread that prompted the original implementation: https://groups.google.com/g/slurm-users/c/BX1_4bztrJA) One-line fix is to put static back on memory_peak_interface. Or, arguably more robust, drop the interfaces_checked cache entirely and run the feature probe on every call — the cost is one access()-equivalent per accounting poll, basically free. he original v2 implementation in ba34862a40 was great, thanks for that — this looks like a small static-qualifier slip on the way through a cleanup. Cheers, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

1 1

slurmdbd and slurmctld prevent alma9 login
by Pols, Maarten 06 May '26

06 May '26

Dear Community, Our Slurm cluster has been running without any issues for several months on version 25.05.3. Last Friday, we experienced a power outage which required us to restart the server. After the restart, we were unable to log in to the master node. Eventually, we managed to access the system via a safe mode workaround. Through a process of elimination, we identified the slurmdbd and slurmctld services as the root cause of the issue. Would you happen to have any idea what might have caused this behavior? We have since upgraded to version 25.11.5, which appears to be running smoothly. However, we would still like to understand the underlying cause of the problem. Thank you in advance for your help. Kind regards, Maarten

3 3

Jobs canceling when nodes become unreachable – need guidance
by Pharthiphan Asokan 05 May '26

05 May '26

Hi, We’re seeing an issue where jobs submitted via salloc are automatically cancelled when a compute node becomes temporarily unreachable. Our goal is to keep jobs pending or requeued instead of being cancelled outright when a node drops offline briefly Slurm sometimes cancels the job rather than requeuing it when the node is marked DOWN/ DRAIN/ DRAINING*. Is there a recommended configuration or additional parameter that ensures jobs remain pending/requeued until the node returns, rather than being cancelled? Any insights or examples from similar setups would be greatly appreciated. Regards, Pharthiphan

3 2

How to track solver usage in Slurm
by fernando.seguro＠rtx.com 23 Apr '26

23 Apr '26

I don't see an easy way of tracking this in slurm. I tried using the --comment="slover" directive, but this gets mapped to the "derived_es" column in the database, which is reserved for job info on errors, from what I read. This would make a good enhancement request. Where would I submit that? Something like: SBATCH --solver=fluent or SBATCH --solver=LSDyna This would be very helpful for tracking solver usage without having to use the job-name. Great for reporting. Please let me know if someone else has requested this previously. If anyone has figured out a way to do it, please share. thanks

4 3

Can't clear the REASON
by Berg, Stephen P CIV USN NRL DET SSC MS (USA) 17 Apr '26

17 Apr '26

Just upgraded my grid to 25.11 a couple days ago. Got two nodes that were down for a bit and when I set them to resume and set the reason="" they are still showing up with the reason I had set while they were down. Nothing I do seems to unset that reason flag for these two nodes. I've set the nodes to down with a reason and then back to idle with the blank reason but it never goes back to "none" like the other nodes. Whatever the last reason set was persists. The behavior seems different since I updated to 25.11. Was running the 20.11 available from the epel-8 repos previously.

2 4