- slurm-users - lists.schedmd.com

Slurm and Slinky at SC'26 - Booth 1711
by Tim Wickberg 10 Nov '26

10 Nov '26

Hey folks - We'll be at SC'26 providing updates on both Slurm and Slinky at the Slurm Booth (1711). We are sharing the presentation space with other NVIDIA software this year, and wanted to highlight a few of the Slurm and Slinky presentations ahead of the conference. Note that we'll be hosting the annual BoF within the booth this year. It wasn't accepted into the conference track, so we'll need to make the best use of the space we have available. We'll be covering the Slurm 26.11 release itself earlier in the week - as well as having the same content available at one of the pods within the booth - and we'll focus the BoF itself on presenting the longer-term strategic roadmap for Slurm, alongside results from the annual community survey. The community survey itself is live now: https://forms.gle/u9YyVrdgFkzHyQu9A Slurm and Slinky Presentations at the Slurm Booth: Tuesday, November 17 > 10:30 - 10:50: Bringing Quantum Computing to Scientific Supercomputing Workflows with Slurm (Stefano Mensa, NVIDIA) > 11:00 - 11:20: Slurm on NVIDIA Vera Rubin: Mastering GPU Topology and Multi-Node Workload Placement (Ben Glines, NVIDIA) > 13:00 - 13:50: Slurm 26.11 and Beyond (Tim Wickberg, NVIDIA) > 14:30 - 14:50: Slinky 101: Slurm for Kubernetes (Vivian Hafener, NVIDIA) Wednesday, Nov 18 > 11:30 - 11:50: Unify Slurm and Kubernetes Scheduling with Slurm Bridge (Skyler Malinowski, NVIDIA) > 13:30 - 13:50: Using Slurm to enable Research Computing in a heterogeneous HPC and AI cluster (Ryan Daniels, University of Cambridge) > 15:00 - 15:20: Keep Nodes Productive with GPU-Level Draining in Slurm (Scott Hilton, NVIDIA) Thursday, Nov 19 > 12:15 - 13:15: Slurm Community Birds of a Feather -- Tim Wickberg twickberg(a)nvidia.com Director, System Software, Slurm + Slinky

1 0

We are pleased to announce the availability of Slurm versions 26.05.6 and 25.11.9.
by Marshall Garey 10 Nov '26

10 Nov '26

We are pleased to announce the availability of Slurm versions 26.05.6 and 25.11.9. A variety of mild to moderate issues were fixed in 26.05.6, including several regressions in 26.05.0 related to accounting limits for heterogeneous jobs. Several stability issues were fixed in 25.11.9. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ Marshall Garey Senior System Software Engineer NVIDIA

1 0

Slurm release candidate version 26.11.0rc1 is available for testing
by Tim Wickberg 05 Nov '26

05 Nov '26

We are pleased to announce the availability of Slurm release candidate 26.11.0rc1. To highlight some new features coming in 26.11: - Add plugin support to the Hierarchical Resources (HRES) subsystem, and the hres/lua plugin script interface. - Add arbitrary key-value metadata (a JSON document) to jobs, nodes, partitions, and reservations. Set it with "scontrol update KV=<json>", "KVPatch=<json>", or "--kv=<json>". Store it in accounting with AccountingStoreFlags=job_kv and resv_kv. - Add the switch/lua plugin to manage interconnects with a Lua script. - Add step dependencies ("srun --step-dependency=") - Add stepmgr support for heterogeneous jobs. Note that slurmctld, slurmd, and srun all need to be upgraded to 26.11 to support this mode of operation. -- Sites are encouraged to enable stepmgr globally through SlurmctldParameters=enable_stepmgr and report any issues before this becomes the default in Slurm 27.05. - Add swait support for waiting on specific steps, as well as a "--follow" option that reports the completion of individual steps while continuing to block until all steps have completed. - Add gpu/amd plugin (AutoDetect=amd) as an alternative to gpu/rsmi for AMD GPUs. This plugin does not require AMD's RSMI library, and will build on all systems. - Add gres/tpu and job_submit/tpu plugins to support Google TPUs. - Add the experimental task/resctrl plugin to confine job steps to resctrl groups (Arm MPAM, Intel RDT, AMD PQoS). Define the groups in resctrl.yaml. - Add support for draining individual GRES on a compute node as a way to mitigate a single failed accelerator card. - Add the dmem cgroup controller to limit shard GRES device memory (ConstrainDeviceMemory=yes). This is the first release candidate of the upcoming 26.11 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://support.schedmd.com against the 26.11.x version and we will address them before the first production 26.11.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://github.com/SchedMD/slurm/releases. The changelog for 26.11 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-26.11.md -- Tim Wickberg twickberg(a)nvidia.com Director, System Software, Slurm + Slinky, NVIDIA

1 0

SURF slurm tools repo
by Bas van der Vlies 05 Nov '26

05 Nov '26

Hi, Maybe also useful to others we have put our SURF slurm tools on codeberg: https://codeberg.org/SURF/slurm_tools smon & snodes Command line tools for Slurm administrators, using the Slurm REST API (slurmrestd): * smon: rack overview of all nodes with their state or CPU load, a summary per state, or a list of busy nodes with jobs, load and memory. * snodes: list unavailable nodes grouped by reason, and drain or resume nodes with a note and an optional ticket number. Regards Bas -- -- Bas van der Vlies | High Performance Computing & Visualization | SURF| Science Park 140 | 1098 XG Amsterdam | T +31 (0) 20 800 1300 | bas.vandervlies(a)surf.nl | www.surf.nl

1 1

Native Lustre encryption with Slurm: loading the key at job time without storing it on the remote network
by sacawulu 04 Nov '26

04 Nov '26

Hi all, I’m new to this list, so hello to everyone. :-) Goal We want to use native lustre client-side encryption (fscrypt) to protect data on a remote HPC cluster, and run Slurm jobs that can work with that data. Current (not so good) concept: The key would temporarily live on the remote network: Key on NFS: /home/username/enc.key Encrypted data on Lustre: /lustre/project The workflow would be: 1. A Slurm prolog loads the key into the kernel keyring on the compute nodes. 2. The same script deletes enc.key from the remote network. 3. The job runs with access to the decrypted data. 4. A Slurm epilog script removes the key from the node’s keyring. The obvious problem: Until the job is scheduled, the key and the encrypted data are on the same remote network, which defeats much of the purpose. Accepted risks We are aware of other attack vectors: remote admins can read enc.key on NFS, and the key can be read from RAM while the job runs. Strict ACLs are essential, and we run the computes --exclusive We accept these risks for now. Question Does anyone know a way to avoid storing enc.key on the remote network before the job starts? Can the key be 'injected' at job start from our side (pushed or pulled) Does slurm have any native mechanism to deal with this? Is there a better pattern for this? Have you implemented something like this on your site, and how? Note: we have an existing site-to-site ipsec tunnel that we could use. Any tips or experiences are welcome. Thanks!

2 3

Making easy and simple security updates of Slurm compute nodes
by Ole Holm Nielsen 26 Oct '26

26 Oct '26

Dear Slurm users, Keeping Slurm compute nodes up-to-date with security updates is an important task, and since security updates are now released with a high frequency, it is desirable to make the updating process as simple and automatic as possible. The tools documented here in [1] offer a possible solution. It's enlightening to browse OS errata pages such as [2] or [3]. The design used in the present approach [1] is: * Configure all compute nodes to execute a crontab job script at every reboot. This script will usually do nothing, except in case the node has a Slurm "State=down" where it will perform the configured updates. * Making updates simply consists of rebooting compute node lists with the command "scontrol reboot asap nextstate=down <node-list>". Here we have employed (abused?) the Slurm "State=down" as a *trigger* telling the update script to perform some actions. Nodes with a "Reason=Node unexpectedly rebooted" are assumed to be unrelated to the present update system and will therefore be skipped. The page [1] gives details of configuration and the optional handling of mixed-state nodes. We have tested these tools on our 700-node cluster running RockyLinux 8 and 9 for some time now, and they work reliably in our environment when updating several times per week. I would welcome any comments or bug reports regarding these tools. Best regards, Ole [1] https://github.com/OleHolmNielsen/Slurm_tools/tree/master/nodes#compute-nod… [2] RockyLinux errata: https://errata.build.resf.org/ [3] AlmaLinux errata: https://errata.almalinux.org/ -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

2 4

Jobs not getting assigned the requested GPU card
by Dj Merrill 19 Oct '26

19 Oct '26

We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo. Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour. We have a node with 5 GPU cards, of three types: % nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) /etc/slurm/gres.conf matches the above output (and has been this way for years): % cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4 The /etc/slurm/slurm.conf entry for this node is the same as it has been for years: NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 When submitting a job such as: $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources $ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) Any suggestions as to where to look? Thank you, -Dj

2 1

Re: Jobs not getting assigned the requested GPU card
by Dj Merrill 16 Oct '26

16 Oct '26

Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked. We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either. As you surmised, the minor does not match the ID: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1 Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4 # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0 # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago. -Dj On 10/15/26 19:14, Brian Haymore wrote: > The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, > and 4, are not certain to map to /dev/nvidia$X. > > Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i > $i | grep "Minor Number"'; done > > If each matches then what I'm asking about is not the issue. If they > do not match then you will need to adjust the gres.conf file. > > Another command that would be good to see is 'nvidia-smi topo -m'. > > -- > Brian D. Haymore > University of Utah > Center for High Performance Computing > 155 South 1452 East RM 405 > Salt Lake City, Ut 84112 > Phone: 801-558-1150 > http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> > ------------------------------------------------------------------------ > *From:* Dj Merrill via slurm-users <slurm-users(a)lists.schedmd.com> > *Sent:* Thursday, October 15, 2026 1:42 PM > *To:* slurm-users(a)lists.schedmd.com <slurm-users(a)lists.schedmd.com> > *Subject:* [slurm-users] Jobs not getting assigned the requested GPU card > We have a Slurm Cluster that has been working fine for many years. A > few months ago it was upgraded to Slurm version 25.11.65, and has been > working properly running that version. The systems are running Rocky > Linux 9.8 and are patched/updated on a regular basis. They are currently > running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from > Nvidia's repo. > > Today it was reported that jobs are not getting assigned the GPU card > they requested, so for example they may request a 5090 card but are > assigned an A100 card. This was working fine, so we are struggling to > figure out what has changed to cause this behaviour. > > We have a node with 5 GPU cards, of three types: > > % nvidia-smi -L > > GPU 0: NVIDIA GeForce RTX 5090 (UUID: > GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) > GPU 1: NVIDIA GeForce RTX 3080 (UUID: > GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) > GPU 2: NVIDIA GeForce RTX 3080 (UUID: > GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) > GPU 3: NVIDIA A100 80GB PCIe (UUID: > GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) > GPU 4: NVIDIA GeForce RTX 3080 (UUID: > GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) > > /etc/slurm/gres.conf matches the above output (and has been this way for > years): > > % cat /etc/slurm/gres.conf > Name=gpu Type=rtx5090 File=/dev/nvidia0 > Name=gpu Type=rtx3080 File=/dev/nvidia1 > Name=gpu Type=rtx3080 File=/dev/nvidia2 > Name=gpu Type=a100 File=/dev/nvidia3 > Name=gpu Type=rtx3080 File=/dev/nvidia4 > > > The /etc/slurm/slurm.conf entry for this node is the same as it has been > for years: > > NodeName=moose[63] CPUs=16 RealMemory=380000 > Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 > > > When submitting a job such as: > > $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash > srun: job 71048 queued and waiting for resources > srun: job 71048 has been allocated resources > > $ nvidia-smi -L > GPU 0: NVIDIA A100 80GB PCIe (UUID: > GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) > > Any suggestions as to where to look? > > Thank you, > > -Dj > > > > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com

5 16

Slurm 26.05.4: idle~ nodes go to unknown after reconfigure
by Stefan Staeglich 14 Oct '26

14 Oct '26

Hi, since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running scontrol reconfigure. This state is not cleaned by Slurm itself. Is this a known issue of Slurm 26.05? Best, Stefan

4 8

Slurm version 26.05.5 is now available
by Marshall Garey 13 Oct '26

13 Oct '26

We are pleased to announce the availability of Slurm version 26.05.5. These releases fix some mild to moderate severity issues and potential crashes. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ Marshall Garey Senior System Software Engineer NVIDIA

1 0