- slurm-users - lists.schedmd.com

Purging of job_script and job_env from the database in 26.05
by Ole Holm Nielsen 01 Jun '26

01 Jun '26

If you are using AccountingStoreFlags=job_script and/or job_env in slurm.conf to store job scripts in the Slurm database, you may be interested in a highly useful feature introduced in Slurm 26.05: Previously, the stored job_scripts and job_env entries would accumulate without bounds in the database, potentially causing database performance or size issues, especially while upgrading Slurm. With 26.05 there are some new slurmdbd.conf options which can be used to purge job_script or job_env entries in the database, see ticket_23818 [1]. This new feature isn't mentioned at all in the Release notes [4], and is only inconspicuously mentioned way down in the Changes section [3] as: > Adding new archive/purge options to allow for explicit archiving of job_scripts and job_env without jobs. The safe way to enable the new purge parameters is to introduce them very gradually, for example: PurgeJobScriptAfter=2000days PurgeJobEnvAfter=2000days and lower the values little by little over time (restarting slurmdbd each time). The MaxPurgeLimit [5] parameter may alternatively be used to limit the amount of database work. Best regards, Ole [1] https://support.schedmd.com/show_bug.cgi?id=23818 [2] https://slurm.schedmd.com/slurm.conf.html#OPT_AccountingStoreFlags [3] https://github.com/SchedMD/slurm/releases [4] https://slurm.schedmd.com/release_notes.html [5] https://slurm.schedmd.com/slurmdbd.conf.html#OPT_MaxPurgeLimit -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

2 3

Large overhead for --task-epilog
by Alan O'Cais 29 May '26

29 May '26

Hi all, I've been experimenting with task prolog/epilog and noticed that task epilogs have significant overhead. The overhead seems to be independent of the epilog script content: $ cat test.sh #!/bin/bash $ time srun --overlap --task-prolog=$PWD/test.sh --ntasks=64 sleep 1 real 0m1.730s user 0m0.211s sys 0m0.009s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=2 sleep 1 real 0m3.530s user 0m0.213s sys 0m0.007s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=64 sleep 1 real 1m5.936s user 0m0.216s sys 0m0.005s This test was on one node, with 64 tasks in the allocation. The overhead seems linked only to number of tasks per node, as adding additional nodes did not increase the overhead. I saw this on $ srun --version slurm 23.11.10-BullSequana.1.2.1 and someone else reproduced similar results with |slurm 25.05.7|. I can live with the overhead, but I am curious as to why it happens and to understand the design of the implementation. Does the task epilog script only run when all tasks finish, or does it run immediately at exit for each task? Are the calls serialised? My use case is a fusermount that I want to mount in userspace at task start and clean up at task finish, one task per node is responsible for the mount/unmount and I want to understand if I have a race condition for the unmount. Any advice very much appreciated, Alan

1 0

Technical Update: Native Slurm Simulation Environment Successfully Configured
by sami sami 28 May '26

28 May '26

Hi All, I am currently working on a single Linux machine running *Ubuntu 25.10* and am looking to conduct performance testing on the *Slurm backfill algorithm*. My objective is to simulate a cluster environment consisting of *1,024 nodes*. While researching existing solutions, I found that many modern Slurm-simulator implementations (such as various plugin-based designs) rely heavily on *Docker and Docker Compose*. However, I would like to avoid using containerization and instead run the simulation natively on my host OS. I understand that older versions of the Slurm source code included a native sim directory that allowed for simulations without external dependencies like Docker. Given this, I have the following technical questions: - *Native Compatibility:* Is it still feasible to compile and run a Slurm simulator natively on Ubuntu 25.10? - *Version Recommendations:* Would you recommend using a specific legacy version of Slurm that still contains the original simulation files, or is there a way to port that functionality to a more recent release? - *Configuration:* Are there specific build flags or configuration steps required to enable the simulated node environment (1,024 nodes) on a single machine without triggering the overhead of containerized networking? I would appreciate any guidance or documentation you could provide on achieving a high-node-count simulation in a "bare-metal" Linux environment. Best regards,

3 2

GPU fallen of the bus
by Ratnasamy, Fritz 27 May '26

27 May '26

Hello, We are noticing that some of the gpus on a specific node have "fallen of the bus". We would like to remove this specific gpu from the slurm scheduler. For example, let's say GPU0 has fallen off the bus, we would need the rest of the GPU1-8 to be available and make GPU0 not able to be allocated. How can we achieve that? I have read about blacklist on the slurm forum but it seems there is no satisfying solution. Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Slurm version 26.05 is now available
by Marshall Garey 26 May '26

26 May '26

We are pleased to announce the availability of Slurm 26.05. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md The Slurm documentation has also been updated to the 26.05 release: https://slurm.schedmd.com Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Core Specialization Feature
by Dan Healy 21 May '26

21 May '26

I'm curious how to report to our users that the number of available CPU cores on a given node are *(CoresPerSocket * SocketsPerBoard * ThreadsPerCore) - CpuSpecCount*. In a basic example, if the node has 32 cores and CpuSpecCount=1, then there's actually 31 cores available. If a user runs `srun -n32 -N1 bash` he/she will be denied for resources. Then, If a user runs `sinfo -O cpus,nodes` shows there's 32 cores available and the user is confused. How can the user clearly identify that in the above case, the exact number of available CPU cores available are 31 and *not* 32? -- Thanks, Daniel Healy

2 2

cgroup error in SLURM version 25.11.2
by bradford.kerr＠srs.gov 19 May '26

19 May '26

I'm trying to install and run SLURM on a stand-alone (no network access) cluster of four PCs running Red Hat Linux 9. On the first compute node of the cluster, I can not get past the error message "cannot create cgroup context for cgroup/v2". It appears that file cgroup_v2.so does not exist. How do I get past this error?

1 0

We are pleased to announce the availability of Slurm versions 25.11.6 and 25.05.8.
by Marshall Garey 14 May '26

14 May '26

We are pleased to announce the availability of Slurm versions 25.11.6 and 25.05.8. Fixes in these releases primarily cover various stability issues when reconfiguring or shutting down and improved support for newer libraries and compilers. The 25.11.6 release also includes a performance fix for scheduling with topology/block and a regression from 25.11.4 that could cause a "No records archived...but some found" message when archiving job data. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

inconsistent 'Requested node configuration is not available' error
by Paul Raines 14 May '26

14 May '26

In my cluster right now we have: $ sinfo ... pubgpu-req up 7-00:00:00 7 mix cobra,l40s-[01,03],luisa,rtx-03,sevilla,shelob pubgpu-req up 7-00:00:00 5 idle fiber,glaurung,gothmog,l40s-02,leo The following works fine: $ srun -p pubgpu-req -A sysadm --nodelist=cobra,l40s-02 --gres=gpu:1 -N 1 \ --ntasks-per-node=1 --mem=1G --time=1:00:00 --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 cobra[0]:~$ exit exit However just change the order of the nodelist and you get $ srun -p pubgpu-req -A sysadm --nodelist=l40s-02,cobra --gres=gpu:1 -N 1 \ --ntasks-per-node=1 --mem=1G --time=1:00:00 --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 srun: error: Unable to create step for job 8255390: Requested node configuration is not available More experimentation with this and it appears nodelist HAS to be in alphabetical order $ srun -p pubgpu-req -A sysadm --nodelist=l40s-02,leo,fiber \ --gres=gpu:1 -N 1 --ntasks-per-node=1 --mem=1G --time=1:00:00 \ --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 srun: error: Unable to create step for job 8255401: Requested node configuration is not available $ srun -p pubgpu-req -A sysadm --nodelist=fiber,l40s-02,leo \ --gres=gpu:1 -N 1 --ntasks-per-node=1 --mem=1G --time=1:00:00 \ --cpus-per-task=4 --pty /bin/bash srun: tres_per_node => gres/gpu:1 fiber[0]:~$ Surely there is no good reason for this? --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

2 1

SLURM config option to not tie up a host completely.
by Ron Gould 08 May '26

08 May '26

I have a user asking about a SLURM config setting that would allow a job to not tie up a node's unused resources, thereby allowing another job to run concurrently. User: On our cluster, our Fluent GPU jobs each use only 1 of the 2 GPUs on GPUServer[1,2]. Currently the node is fully allocated to whichever job lands on it first, leaving the second GPU, 38 CPU cores, and 270+ GB of RAM idle. Has anyone dealt with this? What options would facilitate this? Are there any gotchas or pitfalls?

4 3