- slurm-users - lists.schedmd.com

Preemptible qos
by Steen Lysgaard 04 Jun '26

04 Jun '26

Hello, We have a recurring situation where some users would like to use idle cluster resources on weekends. Their normal CPU limits prevent this, which is generally desirable during the work week to ensure fair sharing. A pattern we're considering is a special low-priority QOS that allows users to exceed their normal association limits, and that is preemptible by jobs submitted with the normal QOS. The goal is to let heavy users fill otherwise idle nodes over weekends without impacting smaller users when activity picks up again on Monday. The walltime limit (50 hours) is not something we would like to change. Has anyone implemented something similar? If so, I'd be interested in hearing about experiences and any pitfalls to this approach. Thanks, Steen Lysgaard [http://www.dtu.dk/-/media/DTU_Generelt/Andet/mail-signature-logo.png] Steen Lysgaard Scientific Software Developer DTU Energy Mob. +45 51 94 48 86 stly(a)dtu.dk<mailto:stly@dtu.dk> Agnes Nielsens Vej Building 301 2800 Kgs. Lyngby www.dtu.dk/english<https://www.dtu.dk/english>

2 1

Authentication
by Sterner, Douglas E. 02 Jun '26

02 Jun '26

New kubernetes deployment and I can't authenticate via ssh using AD credentials to the login controller. Both id and getent resolve user correctly. I have tried every sssd parameter known to mankind. Can someone provide a working example of an sssd.conf for active directory non ssl that I can compare against. Also do I need to do this in the values.yaml? slurm: config: # Explicitly enable PAM for node environments and pam_slurm_adopt UsePAM: 1 Thanks, Douglas Sterner Johns Hopkins Applied Physics Laboratory NSAD / JAF 11100 Johns Hopkins Road. Laurel, MD 20723-6091 [cid:image001.png@01DCF28D.A50E98A0]

2 1

Packing or spreading single-CPU jobs
by Gerhard Strangar 02 Jun '26

02 Jun '26

Hello, I'd like to somehow achieve this: Group A of nodes runs jobs that are supposed to be packed, i.e. if each node has 16 cpus, pack 16 jobs on the first node before using node 2. But for group B, assign every job their own node until all of them have one job, then put a second one on each node until all of them have 2 ... Ir number of jobs doesn't work, load or available memory would do as well, but it seems there's no way of telling slurm to sort the possible nodes by some criteria. Gerhard

2 1

Purging of job_script and job_env from the database in 26.05
by Ole Holm Nielsen 01 Jun '26

01 Jun '26

If you are using AccountingStoreFlags=job_script and/or job_env in slurm.conf to store job scripts in the Slurm database, you may be interested in a highly useful feature introduced in Slurm 26.05: Previously, the stored job_scripts and job_env entries would accumulate without bounds in the database, potentially causing database performance or size issues, especially while upgrading Slurm. With 26.05 there are some new slurmdbd.conf options which can be used to purge job_script or job_env entries in the database, see ticket_23818 [1]. This new feature isn't mentioned at all in the Release notes [4], and is only inconspicuously mentioned way down in the Changes section [3] as: > Adding new archive/purge options to allow for explicit archiving of job_scripts and job_env without jobs. The safe way to enable the new purge parameters is to introduce them very gradually, for example: PurgeJobScriptAfter=2000days PurgeJobEnvAfter=2000days and lower the values little by little over time (restarting slurmdbd each time). The MaxPurgeLimit [5] parameter may alternatively be used to limit the amount of database work. Best regards, Ole [1] https://support.schedmd.com/show_bug.cgi?id=23818 [2] https://slurm.schedmd.com/slurm.conf.html#OPT_AccountingStoreFlags [3] https://github.com/SchedMD/slurm/releases [4] https://slurm.schedmd.com/release_notes.html [5] https://slurm.schedmd.com/slurmdbd.conf.html#OPT_MaxPurgeLimit -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

2 3

Large overhead for --task-epilog
by Alan O'Cais 29 May '26

29 May '26

Hi all, I've been experimenting with task prolog/epilog and noticed that task epilogs have significant overhead. The overhead seems to be independent of the epilog script content: $ cat test.sh #!/bin/bash $ time srun --overlap --task-prolog=$PWD/test.sh --ntasks=64 sleep 1 real 0m1.730s user 0m0.211s sys 0m0.009s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=2 sleep 1 real 0m3.530s user 0m0.213s sys 0m0.007s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=64 sleep 1 real 1m5.936s user 0m0.216s sys 0m0.005s This test was on one node, with 64 tasks in the allocation. The overhead seems linked only to number of tasks per node, as adding additional nodes did not increase the overhead. I saw this on $ srun --version slurm 23.11.10-BullSequana.1.2.1 and someone else reproduced similar results with |slurm 25.05.7|. I can live with the overhead, but I am curious as to why it happens and to understand the design of the implementation. Does the task epilog script only run when all tasks finish, or does it run immediately at exit for each task? Are the calls serialised? My use case is a fusermount that I want to mount in userspace at task start and clean up at task finish, one task per node is responsible for the mount/unmount and I want to understand if I have a race condition for the unmount. Any advice very much appreciated, Alan

1 0

Technical Update: Native Slurm Simulation Environment Successfully Configured
by sami sami 28 May '26

28 May '26

Hi All, I am currently working on a single Linux machine running *Ubuntu 25.10* and am looking to conduct performance testing on the *Slurm backfill algorithm*. My objective is to simulate a cluster environment consisting of *1,024 nodes*. While researching existing solutions, I found that many modern Slurm-simulator implementations (such as various plugin-based designs) rely heavily on *Docker and Docker Compose*. However, I would like to avoid using containerization and instead run the simulation natively on my host OS. I understand that older versions of the Slurm source code included a native sim directory that allowed for simulations without external dependencies like Docker. Given this, I have the following technical questions: - *Native Compatibility:* Is it still feasible to compile and run a Slurm simulator natively on Ubuntu 25.10? - *Version Recommendations:* Would you recommend using a specific legacy version of Slurm that still contains the original simulation files, or is there a way to port that functionality to a more recent release? - *Configuration:* Are there specific build flags or configuration steps required to enable the simulated node environment (1,024 nodes) on a single machine without triggering the overhead of containerized networking? I would appreciate any guidance or documentation you could provide on achieving a high-node-count simulation in a "bare-metal" Linux environment. Best regards,

3 2

GPU fallen of the bus
by Ratnasamy, Fritz 27 May '26

27 May '26

Hello, We are noticing that some of the gpus on a specific node have "fallen of the bus". We would like to remove this specific gpu from the slurm scheduler. For example, let's say GPU0 has fallen off the bus, we would need the rest of the GPU1-8 to be available and make GPU0 not able to be allocated. How can we achieve that? I have read about blacklist on the slurm forum but it seems there is no satisfying solution. Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Slurm version 26.05 is now available
by Marshall Garey 26 May '26

26 May '26

We are pleased to announce the availability of Slurm 26.05. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md The Slurm documentation has also been updated to the 26.05 release: https://slurm.schedmd.com Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Core Specialization Feature
by Dan Healy 21 May '26

21 May '26

I'm curious how to report to our users that the number of available CPU cores on a given node are *(CoresPerSocket * SocketsPerBoard * ThreadsPerCore) - CpuSpecCount*. In a basic example, if the node has 32 cores and CpuSpecCount=1, then there's actually 31 cores available. If a user runs `srun -n32 -N1 bash` he/she will be denied for resources. Then, If a user runs `sinfo -O cpus,nodes` shows there's 32 cores available and the user is confused. How can the user clearly identify that in the above case, the exact number of available CPU cores available are 31 and *not* 32? -- Thanks, Daniel Healy

2 2

cgroup error in SLURM version 25.11.2
by bradford.kerr＠srs.gov 19 May '26

19 May '26

I'm trying to install and run SLURM on a stand-alone (no network access) cluster of four PCs running Red Hat Linux 9. On the first compute node of the cluster, I can not get past the error message "cannot create cgroup context for cgroup/v2". It appears that file cgroup_v2.so does not exist. How do I get past this error?

1 0