- slurm-users - lists.schedmd.com

Memory requirements: seff vs jobstats
by Loris Bennett 15 Jun '26

15 Jun '26

Hi, For quite a few jobs which run on our system, 'seff' reports higher values for 'Memory Utilized' than 'jobstats' does via 'CPU memory usage' (see below for example of a job running on a single node). I was under the impression that 'seff' reports MaxRSS (from 'sacct'), whereas 'jobstats' reports 'Used' (from cgroups), which, as I understand it, is RSS plus page cache and possibly some kernel memory. Therefore, I would expect 'jobstats' to generally produce *larger* estimates of memory usage. However, the discrepancies I see seem always to be such that 'jobstats' report *lower* memory usage. Any thoughts on this? Cheers, Loris $ seff 25800851 Job ID: 25800851 Cluster: curta User/Group: foo/agbar State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 24 CPU Utilized: 10-09:05:31 CPU Efficiency: 39.05% of 26-13:51:36 core-walltime Job Wall-clock time: 1-02:34:39 Memory Utilized: 18.12 GB Memory Efficiency: 28.64% of 63.28 GB (2.64 GB/core) $ jobstats 25800851 ================================================================================ Slurm Job Statistics ================================================================================ Job ID: 25800851 User/Account: foo/agbar Job Name: foobar State: COMPLETED Nodes: 1 CPU Cores: 24 CPU Memory: 64800MB (2.7GB per CPU-core) QOS/Partition: standard/main Cluster: curta Start Time: Mon Jun 8, 2026 at 3:53 PM Run Time: 1-02:34:39 Time Limit: 5-00:00:00 Overall Utilization ================================================================================ CPU utilization [||||||||||||||||||| 39%] CPU memory usage [||||| 11%] Detailed Utilization ================================================================================ CPU utilization per node (CPU time used/run time) c036.curta.zedat.fu-berlin.de: 10-09:03:12/26-13:51:36 (efficiency=39.0%) CPU memory usage per node - used/allocated c036.curta.zedat.fu-berlin.de: 7.1GB/63.3GB (304.3MB/2.6GB per core of 24) ... -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

1 0

Configurator tool on Slurm site is not found
by Valerio Bellizzomi 11 Jun '26

11 Jun '26

Please check. VB

1 0

Visit the Slurm & NVIDIA Booths at ISC 2026
by Victoria Hobson 10 Jun '26

10 Jun '26

If you're heading to ISC in Hamburg, Germany (June 22–26), come find us on the floor! Stop by the Slurm booth (Z10) for sessions on HPC workload management, Kubernetes, and more: Tuesday, June 23 12:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 17:00 – What's New: Latest Slurm and Slinky Release 19:00 – Slinky: Slurm for Kubernetes 101 Wednesday, June 24 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 14:00 – Slinky: Slurm for Kubernetes 101 15:00 – What's New: Latest Slurm and Slinky Release 17:00 – Redefining Resiliency for HPC Research and AI Factories: Slurm and NVIDIA Mission Control Thursday, June 25 10:00 – Slinky: Slurm for Kubernetes 101 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement Then head over to the NVIDIA booth (E30) for even more great content and conversations. We hope to see you there! Victoria Hobson | NVIDIA Campaign Manager, CUDA and Slurm

1 0

Use of scontrol update PartitionName
by Richard Lefebvre 09 Jun '26

09 Jun '26

Hi, To adapt to changing users need, we change which nodes are in what partitions. We use the command: scontrol update PartitionName=partition01 Nodes=nd[001-200] and modify slurm.conf at the same time to match. But when I try to just modify slurm.conf and restart the process "slurmctld". The command "scontrol show PartitionName=partition01" still show the old list of nodes. Even after I reboot the slurmctld server completely. I don't know where the cached version of the list is or how it keeps it. Is there a some slurm.conf configuration that dictated that behavior? Any hints? Richard

2 1

We are pleased to announce the availability of Slurm 26.05.1
by Marshall Garey 09 Jun '26

09 Jun '26

We are pleased to announce the availability of Slurm 26.05.1. Some of the changes in 26.05.1 are highlighted below: * Fix a significant performance regression in slurmctld. * Fix a database performance regression handling step completion records. * Fix an incompatibility between 26.05 slurmd and 25.05+ slurmstepd when using a 24.11 sattach that would cause the sattach to hang. The full list of changes are available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Preemptible qos
by Steen Lysgaard 04 Jun '26

04 Jun '26

Hello, We have a recurring situation where some users would like to use idle cluster resources on weekends. Their normal CPU limits prevent this, which is generally desirable during the work week to ensure fair sharing. A pattern we're considering is a special low-priority QOS that allows users to exceed their normal association limits, and that is preemptible by jobs submitted with the normal QOS. The goal is to let heavy users fill otherwise idle nodes over weekends without impacting smaller users when activity picks up again on Monday. The walltime limit (50 hours) is not something we would like to change. Has anyone implemented something similar? If so, I'd be interested in hearing about experiences and any pitfalls to this approach. Thanks, Steen Lysgaard [http://www.dtu.dk/-/media/DTU_Generelt/Andet/mail-signature-logo.png] Steen Lysgaard Scientific Software Developer DTU Energy Mob. +45 51 94 48 86 stly(a)dtu.dk<mailto:stly@dtu.dk> Agnes Nielsens Vej Building 301 2800 Kgs. Lyngby www.dtu.dk/english<https://www.dtu.dk/english>

2 1

Authentication
by Sterner, Douglas E. 02 Jun '26

02 Jun '26

New kubernetes deployment and I can't authenticate via ssh using AD credentials to the login controller. Both id and getent resolve user correctly. I have tried every sssd parameter known to mankind. Can someone provide a working example of an sssd.conf for active directory non ssl that I can compare against. Also do I need to do this in the values.yaml? slurm: config: # Explicitly enable PAM for node environments and pam_slurm_adopt UsePAM: 1 Thanks, Douglas Sterner Johns Hopkins Applied Physics Laboratory NSAD / JAF 11100 Johns Hopkins Road. Laurel, MD 20723-6091 [cid:image001.png@01DCF28D.A50E98A0]

2 1

Packing or spreading single-CPU jobs
by Gerhard Strangar 02 Jun '26

02 Jun '26

Hello, I'd like to somehow achieve this: Group A of nodes runs jobs that are supposed to be packed, i.e. if each node has 16 cpus, pack 16 jobs on the first node before using node 2. But for group B, assign every job their own node until all of them have one job, then put a second one on each node until all of them have 2 ... Ir number of jobs doesn't work, load or available memory would do as well, but it seems there's no way of telling slurm to sort the possible nodes by some criteria. Gerhard

2 1

Purging of job_script and job_env from the database in 26.05
by Ole Holm Nielsen 01 Jun '26

01 Jun '26

If you are using AccountingStoreFlags=job_script and/or job_env in slurm.conf to store job scripts in the Slurm database, you may be interested in a highly useful feature introduced in Slurm 26.05: Previously, the stored job_scripts and job_env entries would accumulate without bounds in the database, potentially causing database performance or size issues, especially while upgrading Slurm. With 26.05 there are some new slurmdbd.conf options which can be used to purge job_script or job_env entries in the database, see ticket_23818 [1]. This new feature isn't mentioned at all in the Release notes [4], and is only inconspicuously mentioned way down in the Changes section [3] as: > Adding new archive/purge options to allow for explicit archiving of job_scripts and job_env without jobs. The safe way to enable the new purge parameters is to introduce them very gradually, for example: PurgeJobScriptAfter=2000days PurgeJobEnvAfter=2000days and lower the values little by little over time (restarting slurmdbd each time). The MaxPurgeLimit [5] parameter may alternatively be used to limit the amount of database work. Best regards, Ole [1] https://support.schedmd.com/show_bug.cgi?id=23818 [2] https://slurm.schedmd.com/slurm.conf.html#OPT_AccountingStoreFlags [3] https://github.com/SchedMD/slurm/releases [4] https://slurm.schedmd.com/release_notes.html [5] https://slurm.schedmd.com/slurmdbd.conf.html#OPT_MaxPurgeLimit -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

2 3

Large overhead for --task-epilog
by Alan O'Cais 29 May '26

29 May '26

Hi all, I've been experimenting with task prolog/epilog and noticed that task epilogs have significant overhead. The overhead seems to be independent of the epilog script content: $ cat test.sh #!/bin/bash $ time srun --overlap --task-prolog=$PWD/test.sh --ntasks=64 sleep 1 real 0m1.730s user 0m0.211s sys 0m0.009s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=2 sleep 1 real 0m3.530s user 0m0.213s sys 0m0.007s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=64 sleep 1 real 1m5.936s user 0m0.216s sys 0m0.005s This test was on one node, with 64 tasks in the allocation. The overhead seems linked only to number of tasks per node, as adding additional nodes did not increase the overhead. I saw this on $ srun --version slurm 23.11.10-BullSequana.1.2.1 and someone else reproduced similar results with |slurm 25.05.7|. I can live with the overhead, but I am curious as to why it happens and to understand the design of the implementation. Does the task epilog script only run when all tasks finish, or does it run immediately at exit for each task? Are the calls serialised? My use case is a fusermount that I want to mount in userspace at task start and clean up at task finish, one task per node is responsible for the mount/unmount and I want to understand if I have a race condition for the unmount. Any advice very much appreciated, Alan

1 0