- slurm-users - lists.schedmd.com

Slurm release candidate version 26.11.0rc1 is available for testing
by Tim Wickberg 05 Nov '26

05 Nov '26

We are pleased to announce the availability of Slurm release candidate 26.11.0rc1. To highlight some new features coming in 26.11: - Add plugin support to the Hierarchical Resources (HRES) subsystem, and the hres/lua plugin script interface. - Add arbitrary key-value metadata (a JSON document) to jobs, nodes, partitions, and reservations. Set it with "scontrol update KV=<json>", "KVPatch=<json>", or "--kv=<json>". Store it in accounting with AccountingStoreFlags=job_kv and resv_kv. - Add the switch/lua plugin to manage interconnects with a Lua script. - Add step dependencies ("srun --step-dependency=") - Add stepmgr support for heterogeneous jobs. Note that slurmctld, slurmd, and srun all need to be upgraded to 26.11 to support this mode of operation. -- Sites are encouraged to enable stepmgr globally through SlurmctldParameters=enable_stepmgr and report any issues before this becomes the default in Slurm 27.05. - Add swait support for waiting on specific steps, as well as a "--follow" option that reports the completion of individual steps while continuing to block until all steps have completed. - Add gpu/amd plugin (AutoDetect=amd) as an alternative to gpu/rsmi for AMD GPUs. This plugin does not require AMD's RSMI library, and will build on all systems. - Add gres/tpu and job_submit/tpu plugins to support Google TPUs. - Add the experimental task/resctrl plugin to confine job steps to resctrl groups (Arm MPAM, Intel RDT, AMD PQoS). Define the groups in resctrl.yaml. - Add support for draining individual GRES on a compute node as a way to mitigate a single failed accelerator card. - Add the dmem cgroup controller to limit shard GRES device memory (ConstrainDeviceMemory=yes). This is the first release candidate of the upcoming 26.11 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://support.schedmd.com against the 26.11.x version and we will address them before the first production 26.11.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://github.com/SchedMD/slurm/releases. The changelog for 26.11 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-26.11.md -- Tim Wickberg twickberg(a)nvidia.com Director, System Software, Slurm + Slinky, NVIDIA

1 0

SURF slurm tools repo
by Bas van der Vlies 05 Nov '26

05 Nov '26

Hi, Maybe also useful to others we have put our SURF slurm tools on codeberg: https://codeberg.org/SURF/slurm_tools smon & snodes Command line tools for Slurm administrators, using the Slurm REST API (slurmrestd): * smon: rack overview of all nodes with their state or CPU load, a summary per state, or a list of busy nodes with jobs, load and memory. * snodes: list unavailable nodes grouped by reason, and drain or resume nodes with a note and an optional ticket number. Regards Bas -- -- Bas van der Vlies | High Performance Computing & Visualization | SURF| Science Park 140 | 1098 XG Amsterdam | T +31 (0) 20 800 1300 | bas.vandervlies(a)surf.nl | www.surf.nl

1 1

Native Lustre encryption with Slurm: loading the key at job time without storing it on the remote network
by sacawulu 04 Nov '26

04 Nov '26

Hi all, I’m new to this list, so hello to everyone. :-) Goal We want to use native lustre client-side encryption (fscrypt) to protect data on a remote HPC cluster, and run Slurm jobs that can work with that data. Current (not so good) concept: The key would temporarily live on the remote network: Key on NFS: /home/username/enc.key Encrypted data on Lustre: /lustre/project The workflow would be: 1. A Slurm prolog loads the key into the kernel keyring on the compute nodes. 2. The same script deletes enc.key from the remote network. 3. The job runs with access to the decrypted data. 4. A Slurm epilog script removes the key from the node’s keyring. The obvious problem: Until the job is scheduled, the key and the encrypted data are on the same remote network, which defeats much of the purpose. Accepted risks We are aware of other attack vectors: remote admins can read enc.key on NFS, and the key can be read from RAM while the job runs. Strict ACLs are essential, and we run the computes --exclusive We accept these risks for now. Question Does anyone know a way to avoid storing enc.key on the remote network before the job starts? Can the key be 'injected' at job start from our side (pushed or pulled) Does slurm have any native mechanism to deal with this? Is there a better pattern for this? Have you implemented something like this on your site, and how? Note: we have an existing site-to-site ipsec tunnel that we could use. Any tips or experiences are welcome. Thanks!

2 3

Making easy and simple security updates of Slurm compute nodes
by Ole Holm Nielsen 26 Oct '26

26 Oct '26

Dear Slurm users, Keeping Slurm compute nodes up-to-date with security updates is an important task, and since security updates are now released with a high frequency, it is desirable to make the updating process as simple and automatic as possible. The tools documented here in [1] offer a possible solution. It's enlightening to browse OS errata pages such as [2] or [3]. The design used in the present approach [1] is: * Configure all compute nodes to execute a crontab job script at every reboot. This script will usually do nothing, except in case the node has a Slurm "State=down" where it will perform the configured updates. * Making updates simply consists of rebooting compute node lists with the command "scontrol reboot asap nextstate=down <node-list>". Here we have employed (abused?) the Slurm "State=down" as a *trigger* telling the update script to perform some actions. Nodes with a "Reason=Node unexpectedly rebooted" are assumed to be unrelated to the present update system and will therefore be skipped. The page [1] gives details of configuration and the optional handling of mixed-state nodes. We have tested these tools on our 700-node cluster running RockyLinux 8 and 9 for some time now, and they work reliably in our environment when updating several times per week. I would welcome any comments or bug reports regarding these tools. Best regards, Ole [1] https://github.com/OleHolmNielsen/Slurm_tools/tree/master/nodes#compute-nod… [2] RockyLinux errata: https://errata.build.resf.org/ [3] AlmaLinux errata: https://errata.almalinux.org/ -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

2 4

Jobs not getting assigned the requested GPU card
by Dj Merrill 19 Oct '26

19 Oct '26

We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo. Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour. We have a node with 5 GPU cards, of three types: % nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) /etc/slurm/gres.conf matches the above output (and has been this way for years): % cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4 The /etc/slurm/slurm.conf entry for this node is the same as it has been for years: NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 When submitting a job such as: $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources $ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) Any suggestions as to where to look? Thank you, -Dj

2 1

Re: Jobs not getting assigned the requested GPU card
by Dj Merrill 16 Oct '26

16 Oct '26

Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked. We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either. As you surmised, the minor does not match the ID: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1 Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4 # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0 # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago. -Dj On 10/15/26 19:14, Brian Haymore wrote: > The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, > and 4, are not certain to map to /dev/nvidia$X. > > Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i > $i | grep "Minor Number"'; done > > If each matches then what I'm asking about is not the issue. If they > do not match then you will need to adjust the gres.conf file. > > Another command that would be good to see is 'nvidia-smi topo -m'. > > -- > Brian D. Haymore > University of Utah > Center for High Performance Computing > 155 South 1452 East RM 405 > Salt Lake City, Ut 84112 > Phone: 801-558-1150 > http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> > ------------------------------------------------------------------------ > *From:* Dj Merrill via slurm-users <slurm-users(a)lists.schedmd.com> > *Sent:* Thursday, October 15, 2026 1:42 PM > *To:* slurm-users(a)lists.schedmd.com <slurm-users(a)lists.schedmd.com> > *Subject:* [slurm-users] Jobs not getting assigned the requested GPU card > We have a Slurm Cluster that has been working fine for many years. A > few months ago it was upgraded to Slurm version 25.11.65, and has been > working properly running that version. The systems are running Rocky > Linux 9.8 and are patched/updated on a regular basis. They are currently > running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from > Nvidia's repo. > > Today it was reported that jobs are not getting assigned the GPU card > they requested, so for example they may request a 5090 card but are > assigned an A100 card. This was working fine, so we are struggling to > figure out what has changed to cause this behaviour. > > We have a node with 5 GPU cards, of three types: > > % nvidia-smi -L > > GPU 0: NVIDIA GeForce RTX 5090 (UUID: > GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) > GPU 1: NVIDIA GeForce RTX 3080 (UUID: > GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) > GPU 2: NVIDIA GeForce RTX 3080 (UUID: > GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) > GPU 3: NVIDIA A100 80GB PCIe (UUID: > GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) > GPU 4: NVIDIA GeForce RTX 3080 (UUID: > GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) > > /etc/slurm/gres.conf matches the above output (and has been this way for > years): > > % cat /etc/slurm/gres.conf > Name=gpu Type=rtx5090 File=/dev/nvidia0 > Name=gpu Type=rtx3080 File=/dev/nvidia1 > Name=gpu Type=rtx3080 File=/dev/nvidia2 > Name=gpu Type=a100 File=/dev/nvidia3 > Name=gpu Type=rtx3080 File=/dev/nvidia4 > > > The /etc/slurm/slurm.conf entry for this node is the same as it has been > for years: > > NodeName=moose[63] CPUs=16 RealMemory=380000 > Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 > > > When submitting a job such as: > > $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash > srun: job 71048 queued and waiting for resources > srun: job 71048 has been allocated resources > > $ nvidia-smi -L > GPU 0: NVIDIA A100 80GB PCIe (UUID: > GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) > > Any suggestions as to where to look? > > Thank you, > > -Dj > > > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com

5 16

Slurm 26.05.4: idle~ nodes go to unknown after reconfigure
by Stefan Staeglich 14 Oct '26

14 Oct '26

Hi, since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running scontrol reconfigure. This state is not cleaned by Slurm itself. Is this a known issue of Slurm 26.05? Best, Stefan

4 8

Slurm version 26.05.5 is now available
by Marshall Garey 13 Oct '26

13 Oct '26

We are pleased to announce the availability of Slurm version 26.05.5. These releases fix some mild to moderate severity issues and potential crashes. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ Marshall Garey Senior System Software Engineer NVIDIA

1 0

Need configuration help
by Steffen Grunewald 13 Oct '26

13 Oct '26

Good afternoon all, after having run a Slurm cluster for a long time with exclusive node usage (the DEFAULT partition set to Shared=EXCLUSIVE, for historical reasons), I have now received a request to provide individual CPU cores as well, so "shared mode" would be it. I set up a new partition with OverSubscribe=NO, changed the DEFAULT stanza to OverSubscribe=EXCLUSIVE, and things seemed to work (after I changed SelectTypeParameters to CR_Core_Memory from CR_Core - apparently nobody ever hit a memory limit before...). But - now there are jobs that don't run anymore. After staring at them for hours, I found that they would request CPUs=32 (they always did, I suspect). Our compute nodes are 2s/16c/2t, "slurmd -C" on them reports CPUs=64 Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 Moving those jobs (that claim "never to be runnable") to the new partition now appears to work, and they claim enough memory not to share their node with other jobs, but I'd like to keep them in their original partition. Where's my mistake? Am I trying to mix things that can't be mixed? Thanks, S -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

3 2

SlurmDB Duplicate
by Paul Edmon 28 Sep '26

28 Sep '26

Currently we have a slurmdbd that is our main production, and thus under heavy traffic with query restrictions (1 week at a time currently). We also archive data that is older than 6 months so that the database stays lean and mean. This serves the purpose well for general day to day but limits us in terms of doing larger queries for the sake of summary statistics and billing. What I would like to stand up is a second slurmdbd that has no query restrictions and has all my historical data, while also being kept abreast of the live information coming off the cluster. Straight replication (slurmdbd or mysql) won't suffice as when I do my monthly purges, it would then propagate that purge. Likewise reimporting the archived data into an external database would be missing the current 6 months. My question is is there a clever way to run a duplicate slurmdbd that gets all the cluster traffic so it stays up to date, while also not purging data? Also in such as way that I could say have the duplicate slurmdbd offsite and not drag down cluster performance (as really the cluster should lean on the active database that is more proximate)? Looking at the documentation I can see an obvious method for HA, and AccountingStorageExternalHost which looks like it might work but may drag down performance due to needing to wait for DB operations to complete there. I'm curious if other sites have solved this. Summary programs like XDMod are useful but having the raw data available to cross check and process in new ways is what we are really looking for. Thanks in advance. -Paul Edmon-

4 4