- slurm-users - lists.schedmd.com

How do you handle GPU node failures during long jobs?
by Antonio Jose Alonso-Stepanov 26 Mar '26

26 Mar '26

Hi all, I'm a Stanford CS student looking into how sites handle GPU node failures during long-running jobs. A couple questions: When a GPU node goes down mid-job, do most sites use Slurm's requeue or --no-kill to handle it, or is it mostly manual drain and resubmit? Is anyone using HealthCheckProgram to catch GPU issues (like ECC errors via DCGM), or do you handle GPU health monitoring outside of Slurm? Curious what's worked and what hasn't. Thanks. Antonio

3 2

_node_config_validate: gres/gpu: Count changed on node (0 != 2)
by Xaver Stiensmeier 23 Mar '26

23 Mar '26

Hey Slurm-users list, while our regular gpu nodes are working fine, our on demand gpu nodes have a weird issue. They power up, I can ssh onto them and execute nvidia-smi on them without issue, but they are marked invalid and slurmctld logs _node_config_validate: gres/gpu: Count changed on node (0 != 2) however, node show shows that the gpus are recognized and the gres.conf are stored on the worker nodes as expected and the node entries in the slurm.conf are fine, too: # slurm.conf NodeName=my_worker_node SocketsPerBoard=16 CoresPerSocket=1 RealMemory=64075 MemSpecLimit=4000 State=CLOUD Gres=gpu:L4:2 # openstack # gres.conf on my_worker_node ubuntu@my_node:~$ cat /etc/slurm/gres.conf # GRES CONFIG Name=gpu Type=L4 File=/dev/nvidia0 Name=gpu Type=L4 File=/dev/nvidia1 Thankful for any ideas and debugging ideas. Best, Xaver PS: By executing: sudo scontrol update NodeName=$(bibiname 0) Gres= sudo scontrol reconfigure sudo scontrol update NodeName=$(bibiname 0) state=RESUME reason=None the node can be resumed. However, this is not a real solution.

2 3

We are pleased to announce the availability of Slurm versions 25.11.4 and 25.05.7.
by Marshall Garey 12 Mar '26

12 Mar '26

We are pleased to announce the availability of Slurm versions 25.11.4 and 25.05.7. Changes in both versions include preventing SIGHUP from terminating slurmrestd. Changes in 25.11.4 include fixing several issues with the switch/nvidia_imex plugin, several memory leaks, a race that could cause jobs to be stuck in the completing state or be requeued and held if the --get-user-env option was used, flex reservations taking nodes from other reservations that do not request full nodes, and a variety of other issues. Changes in 25.05.7 include fixing the gpu/nrt plugin, and fixing some other regressions and stability issues. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Server with two GPUs sharing each GPU to different partitions
by Gestió Servidors 09 Mar '26

09 Mar '26

Hello, First of all, sorry if my question is about something easy, but in my environment, this is the first time I need to do this. I have one server that have 2 identical GPUs. I have two different partitions: one is accessible from some allocation nodes and the other from other allocation nodes. My question is if it's possible to share one GPU with one partition and the other GPU with the other partition. For example, something like this: * gres.conf: NodeName=gpu-node AutoDetect=off Name=gpu Type=RTX5070 File=/dev/nvidia0 Cores=0-127 NodeName=gpu-node AutoDetect=off Name=gpu Type=RTX5070 File=/dev/nvidia1 Cores=128-255 * slurm.conf: NodeName=gpu-node CPUs=256 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=260000 TmpDisk=47000 Gres= RTX5070:2 PartitionName=gpu-int.q Nodes=gpu-node OverSubscribe=No State=UP AllocNodes=submit-node-1 PriorityTier=3 GraceTime=900 PartitionName=gpu-ext.q Nodes=gpu-node OverSubscribe=No State=UP AllocNodes=submit-node-1 PriorityTier=3 GraceTime=900 With this configuration, if simultaneously two different users tries to execute in one of the RTX5070 (one in gpu-int.q and the other in gpu-ext.q), could both of them execute with no problems? If user from gpu-int.q uses device #0 and user from gpu-ext.q uses device #1, could SLURM work without problems if a second simultaneously exection assigns device #1 to user from gpu-int.q and device #0 to user from gpu-ext.q? Thanks.

1 0

Optimizing CPU allocation in Slurm with hyperthreading enabled
by Guillaume COCHARD 02 Mar '26

02 Mar '26

Hello, The Slurm documentation states that "The default allocation method within a node is cyclic allocation (allocate available CPUs in a round-robin fashion across the sockets within a node)." In our case, we are using hyperthreading, which means that if two jobs requesting one CPU each arrive at the same time on an empty node, they will be assigned to the two threads of the same physical core. To improve performance, we would like Slurm to allocate CPUs (in our case, threads) in a way that prioritizes using one thread per physical core first, and only once all cores are occupied, assign the second thread of each core. This would be similar to the Least Loaded Node (LLN) approach, but instead of distributing jobs across nodes, it would distribute them across physical cores (Least Loaded Core). We do not want to disable hyperthreading; we simply want to optimize core usage when a machine is not fully loaded. We also want this to be a cluster-wide setting, and not a user and/or job setting. Our current slurm.conf contains: ConstrainCores = no SelectType = select/cons_tres SelectTypeParameters = CR_CPU_MEMORY And our cgroup.conf contains: ConstrainDevices=yes ConstrainCores=yes Does anyone know how to achieve this? Thanks, Guillaume

3 2

Missing mounts.h in Ubuntu Bionic build
by Michael Gutteridge 25 Feb '26

25 Feb '26

Hi We're running into a problem with building 25.11.3 for Ubuntu Bionic. When the cgroup/v2 plugin is building, ebpf throws an error: ``` ../../../../../src/plugins/cgroup/v2/ebpf.c:38:10: fatal error: linux/mount.h: No such file or directory #include <linux/mount.h> ^~~~~~~~~~~~~~~ compilation terminated. Makefile:638: recipe for target 'ebpf.lo' failed make[6]: *** [ebpf.lo] Error 1 make[6]: Leaving directory '/build/slurm-25.11.3/obj-x86_64-linux-gnu/src/plugins/cgroup/v2' ``` I think this is because on Ubuntu, the `mounts.h` header is in the path `sys`- the manpage for mounts(2) describes including this file via `#include <sys/mounts.h>`. `ebpf.c` seems to be the only place where the `linux/mounts.h` path is used, everywhere else using `sys`. Curiously it seems to build OK on Noble (Ubuntu 24). I have no explanation for the difference. Has anyone run into this problem building on Ubuntu? This include looks to have been a recent add so might be a bug, but I feel like someone else would have run into this already if it were. Thanks - Michael

1 0

Issue with coordinator increasing limits
by Groner, Rob 24 Feb '26

24 Feb '26

I'm seeing a weird issue when I try to increase limits on a child account that a coordinator has created. Here is the setup: 1. Create an account named slurm_parent, set coordinator1 as a coordinator of that account, and give that account a grptresmins=cpu=100 limit. 2. As coordinator1, create a child account named slurm_child whose parent is slurm_parent. 3. As coordinator1, set the grptresmins=cpu=10 limit on the child account. That all works fine. The problem is when I try to INCREASE the cpu limit on the child account by any amount. $coordinator1> sacctmgr modify account test_crch_slurmaccessch set grptresmins=cpu=11 sacctmgr: error: Coordinators can not increase job limits beyond the parent ones Modified account associations... Error with request: Coordinators can not increase job limits beyond the parent ones The first issue is that the limit of cpu=11 is NOT beyond the parent limit of cpu=100. The second issue is that (as a workaround), I can remove the cpu=10 limit on the child account as coordinator (by setting it to -1), and then I can set grptresmins=cpu=1000. That command works. So, I can very clearly set a grptresmins=cpu limit that is beyond the parent limit (I understand that the child account in reality won't be able to access more than the parent account, no matter what the limit is). Note that as coordinator, I can REDUCE the limit on the child account all I want. So I can set it to cpu=9 just fine. But if I try to set it back to 10, I'll again get the error saying I can't go above parent limits. So here are my issues: 1. Inability of coordinator to increase the child account grptresmins limits once set. 2. Incorrect error message about exceeding the parent limit. I originally saw this issue on 24.11.6, and built and tested on 25.11.3, and still see the issue. Thanks for any help figuring out what I'm doing wrong. Rob

1 1

Reliable, Atomic, Idempotent, or Transactional Job Submission?
by Adam Novak 24 Feb '26

24 Feb '26

I'm working on the Slurm integration in our Toil workflow runner project. I'm having a problem where an `sbatch` command to submit a job to Slurm can fail (with exit code 1 and message "sbatch: error: Batch job submission failed: Socket timed out on send/recv operation", in my case, but possibly in other ways), but the job can still actually have been submitted, and can still execute. This causes a problem for Toil because right now, when it sees a submission attempt fail, it backs off and submits the job again a little later. But Toil can't handle multiple copies of the same job running at once, and if a submission appears to the client to have failed but actually succeeded, it's possible to get into that situation if you just submit again. When an sbatch command fails, is it possible to detect cases where the cluster will still execute the job? (For example, is it guaranteed that the job ID will be available on standard output on the client if the job is going to execute on the cluster, no matter when in the client process it might get a socket disconnection, allowing the job to be inquired about later?) Do I maybe need to tag my jobs with unique identifiers myself so I can poll for them in the queue after a supposedly-failed submission? Is it possible to write an idempotent sbatch command, where it can be run any number of times but will only actually submit one copy of the job? Is the Slurm REST API somehow more transactional, or able to promise somehow that a job will not actually go into the queue without the client having acknowledged receipt of the job's assigned ID? Thanks, -Adam -- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code." Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5

4 7

Get the list of added dynamic nodes
by Edmundo Carmona Antoranz 23 Feb '26

23 Feb '26

Hi! Is there a way to get the list of dynamic nodes that have been created but that are pending to get slurmd to run on them? I fail to see them with with either scontrol or sinfo: root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol create NodeName=node004 .... state=FUTURE feature=dyn1 root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol version slurm 25.11.2 Only when I start slurmd from node004 does it become visible (through scontrol only): root@ea-u24-dynamic-2511:~# ssh node004 /cm/local/apps/slurm/current/sbin/slurmd -Z -f /cm/shared/apps/slurm/etc/slurm/slurm.conf root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 NodeName=node004 Arch=x86_64 CoresPerSocket=1 Thanks in advance. Edmundo Carmona Antoranz Software Engineer NVIDIA Corporation

3 4

We are pleased to announce the availability of Slurm version 25.11.3
by Tim Wickberg 19 Feb '26

19 Feb '26

We are pleased to announce the availability of Slurm version 25.11.3. 25.11.3 fixes a number of stability issues, regressions, and a variety of other bugs, including the following: * Fix multiple issues with dynamic topology when power saving is enabled * Fix a backfill performance regression when using bf_licenses * Fix race conditions when shutting down slurmctld or slurmdbd that cause crashes or hangs * Fix slurmctld crashing when using enable_async_reply and TLS * Fix regression from 25.05 which caused usercpu and systemcpu to be missing for job steps * Fix regression that caused thread limits in slurmd to not be respected for incoming RPCs * Prevent the double-removal of accounting usage for jobs being requeued that are in the COMPLETED or COMPLETING state The full list of changes is available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases One note on some updates to our release process - we've started publishing the releases directly on GitHub, in parallel to their usual location on the SchedMD web server. If you 'star' the Slurm repository you can automatically receive notices when new releases are published there. You can also subscribe to the RSS/Atom feed through: https://github.com/SchedMD/slurm/releases.atom - Tim -- Tim Wickberg Director, System Software, Slurm and Slinky, NVIDIA

1 0

2026

2025

2024

slurm-users