- slurm-users - lists.schedmd.com

Jobs aborting after slurmctld reload on Intel nodes - AMD unaffected
by Pharthiphan Asokan 16 Apr '26

16 Apr '26

Hi team, We’re observing job aborts on Intel-based nodes immediately after a slurmctld reload. AMD nodes remain stable and jobs continue unaffected. No system or Slurm configuration changes were made before the issue started. Error observed: error: Aborting JobID=1288 due to change in socket/core configuration of allocated nodes Relevant node configuration (Intel node example): NodeName=smc-h4-u19 CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=1 RealMemory=510976 State=UNKNOWN Feature=model_SSG-1228-B From logs: error: valid_job_resources: smc-h4-u19 sockets:2:2, cores 64,32 error: Node configuration differs from hardware: CPUs=128:128(hw) Boards=1:1(hw) SocketsPerBoard=2:2(hw) What we’ve verified: * No changes in BIOS, firmware, or hardware topology * No edits to slurm.conf or slurm_nodes.conf * Reload (scontrol reconfigure) triggers job aborts only on Intel nodes * AMD nodes remain intact through reloads

3 2

We are pleased to announce the availability of Slurm version 25.11.5
by Marshall Garey 14 Apr '26

14 Apr '26

We are pleased to announce the availability of Slurm version 25.11.5. Some of the changes in 25.11.0 are highlighted below: * Fix a regression in 25.11.0 that could cause daemons to hang after reaching conmgr_max_connections. * Fix a regression added in 25.05.0 concerning how the slurmctld inherits /run/slurmctld/sack.socket when using AuthType=auth/slurm to prevent clients that connected during a reconfigure from hanging indefinitely. * Fix a race in tree forwarding that could cause a crash during shutdown * Fix several bugs in the namespace/linux plugin that caused instability The full list of changes are available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Why my job can't start (backfill reservation issue)
by Massimo Sgaravatto 14 Apr '26

14 Apr '26

Dear all I (try to) manage a slurm cluster composed by some CPU-only nodes and some worker nodes which have also GPUs: NodeName=cld-ter-[01-06] Sockets=2 CoresPerSocket=96 ThreadsPerCore=2 RealMemory=1536000 State=UNKNOWN NodeName=cld-ter-gpu-[01-05] Sockets=2 CoresPerSocket=96 ThreadsPerCore=2 Gres=gpu:nvidia-h100:4 RealMemory=1536000 State=UNKNOWN The GPU nodes are exposed through multiple partitions: PartitionName=gpus Nodes=cld-ter-gpu-[01-02] State=UP PriorityTier=20 PartitionName=sparch Nodes=cld-ter-gpu-03 AllowAccounts=sparch,operators QoS=sparch State=UP PriorityTier=20 PartitionName=geant4 Nodes=cld-ter-gpu-03 AllowAccounts=geant4,operators QoS=geant4 State=UP PriorityTier=20 PartitionName=enipred Nodes=cld-ter-gpu-04 AllowAccounts=enipred,operators QoS=enipred State=UP PriorityTier=20 PartitionName=enipiml Nodes=cld-ter-gpu-05 AllowAccounts=enipiml,operators QoS=enipiml State=UP PriorityTier=20 We also set a partition to allow cpu-only jobs on the GPU nodes, but these jobs should be preempted (killed and requeued) if jobs submitted to partitions with higher priorities require those resources: PreemptType=preempt/partition_prio PreemptMode=REQUEUE PartitionName=onlycpus-opp Nodes=cld-ter-gpu-[01-05],cld-dfa-gpu-06,btc-dfa-gpu-02 State=UP PriorityTier=10 Now, I don't understand why this job [*] submitted on the onlycpus-opp partition can't start running e.g. on the cld-ter-gpu-01, since it has a lot of free resources: [sgaravat@cld-ter-ui-01 ~]$ scontrol show node cld-ter-gpu-01 NodeName=cld-ter-gpu-01 Arch=x86_64 CoresPerSocket=96 CPUAlloc=8 CPUEfctv=384 CPUTot=384 CPULoad=5.93 AvailableFeatures=(null) ActiveFeatures=(null) Gres=gpu:nvidia-h100:4 NodeAddr=cld-ter-gpu-01 NodeHostName=cld-ter-gpu-01 Version=25.11.3 OS=Linux 5.14.0-611.45.1.el9_7.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 1 05:56:53 EDT 2026 RealMemory=1536000 AllocMem=560000 FreeMem=1192357 Sockets=2 Boards=1 State=MIXED+PLANNED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpus,onlycpus-opp BootTime=2026-04-09T10:39:35 SlurmdStartTime=2026-04-09T10:40:01 LastBusyTime=2026-04-09T11:54:46 ResumeAfterTime=None CfgTRES=cpu=384,mem=1500G,billing=839,gres/gpu=4,gres/gpu:nvidia-h100=4 AllocTRES=cpu=8,mem=560000M,gres/gpu=4,gres/gpu:nvidia-h100=4 CurrentWatts=0 AveWatts=0 I guess the "MIXED+PLANNED" is the answer, but as far as I can see only a job (283469) is planned for this worker node: sgaravat@cld-ter-ui-01 ~]$ squeue --start | grep ter-gpu-01 JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON) 283469 gpus vllm-pod ciangott PD 2026-04-13T14:31:40 1 cld-ter-gpu-01 (Resources) But job 283469 doesn't require too many resources [**], so the 2 jobs could run together. Why job 283534 can't start ? Any hints ? Thanks, Massimo [*] [sgaravat@cld-ter-ui-01 ~]$ scontrol show job=283534 JobId=283534 JobName=myscript.sh UserId=sgaravat(5008) GroupId=tbadmin(5001) MCS_label=N/A Priority=542954 Nice=0 Account=operators QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:41 TimeLimit=1-00:00:00 TimeMin=N/A SubmitTime=2026-04-13T11:10:13 EligibleTime=2026-04-13T11:10:13 AccrueTime=2026-04-13T11:10:13 StartTime=2026-04-13T11:58:39 EndTime=2026-04-14T11:58:39 Deadline=N/A PreemptEligibleTime=2026-04-13T11:58:39 PreemptTime=None SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-13T11:58:39 Scheduler=Backfill Partition=onlycpus-opp AllocNode:Sid=cld-ter-ui-01:3035857 ReqNodeList=(null) ExcNodeList=(null) NodeList=btc-dfa-gpu-02 BatchHost=btc-dfa-gpu-02 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=100G,node=1,billing=26 AllocTRES=cpu=1,mem=100G,node=1,billing=26 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null) Command=/shared/home/sgaravat/myscript.sh SubmitLine=sbatch myscript.sh WorkDir=/shared/home/sgaravat StdErr=/shared/home/sgaravat/JOB-myscript.sh.283534.4294967294.err StdIn=/dev/null StdOut=/shared/home/sgaravat/JOB-myscript.sh.283534.4294967294.out MailUser=massimo.sgaravatto(a)pd.infn.it MailType=INVALID_DEPEND,BEGIN,END,FAIL,REQUEUE,STAGE_OUT [**] sgaravat@cld-ter-ui-01 ~]$ scontrol show job=283469 JobId=283469 JobName=vllm-pod UserId=ciangott(6054) GroupId=tbuser(6000) MCS_label=N/A Priority=499703 Nice=0 Account=cms QOS=normal JobState=PENDING Reason=Resources Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A SubmitTime=2026-04-13T06:48:37 EligibleTime=2026-04-13T06:48:37 AccrueTime=2026-04-13T06:48:37 StartTime=2026-04-13T14:31:40 EndTime=2026-04-14T14:31:40 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-13T11:59:48 Scheduler=Main Partition=gpus AllocNode:Sid=cld-ter-ui-01:3015801 ReqNodeList=(null) ExcNodeList=(null) NodeList= SchedNodeList=cld-ter-gpu-01 NumNodes=1-1 NumCPUs=32 NumTasks=1 CPUs/Task=32 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=32,mem=190734M,node=1,billing=118,gres/gpu=2,gres/gpu:nvidia-h100=2 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=32 MinMemoryNode=190734M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null) Command=.interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.slurm SubmitLine=sbatch .interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.slurm WorkDir=/shared/home/ciangott StdErr= StdIn=/dev/null StdOut=/shared/home/ciangott/.interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.out TresPerNode=gres/gpu:nvidia-h100:2 TresPerTask=cpu=32

5 10

pam_slurm_adopt
by Ratnasamy, Fritz 09 Apr '26

09 Apr '26

Hi, We have blocked ssh to nodes where users do not have a running job. However, when there is a running job, a user can ssh to that node and use more than the slurm resources allocated on that node. Am I missing something? I usually request an interactive session of 2GB. I then open another terminal and ssh to the node where the interactive session runs. I allocated a 4GB R variable on the node where I just ssh but this does not error out. Additionally, if I end my interactive session, the ssh connection still lives which I believe should not happen. Best, *Fritz Ratnasamy*Data Scientist Information Technology

7 6

How to delete my defaultwckey ?
by Faraz Hussain 07 Apr '26

07 Apr '26

I want every submitted job to have some value for the wckey, i.e: #SBATCH --wckey=myproject I made the appropriate changes to slurm.conf and slurmdb.conf and it works great. I can track how many hours people are using with those wckeys. But now I want to make it mandatory to use a wckey. To do that I need to delete the default wckey associated with the user's account. I tried doing it as follows, but it still lets me submit jobs without a wckey. It probably thinks I have an "empty" default wckey. sacctmgr mod user fhussa set defaultwckey= [root@mas01 ~]# sacctmgr list user fhussa format=user,defaultwckey User Def WCKey ---------- ---------- fhussa

1 0

Difference between sreport and sacct
by hany.ibrahim＠kit.edu 07 Apr '26

07 Apr '26

Hello, I am trying to understand how sreport work. So I run sreport and obtain th estatistics about speciic project. On teh other side, I collected the jobs related to this account using sacct. When I try to sum the numbers, sometimes it don't sum up. I can't see any reservations. This is the sreport I used: sreport --tres cpu cluster AccountUtilizationByUser Start=2021-01-01 account=<account> -t hours -np This is jobs I collected by using sacct: sudo -u slurm sacct -a -S "2021-01-01" -E NOW --format=JobID%100,User,Account%100,State,AllocCPUs,CPUTime%100,CPUTIMERAW%100,ElapsedRaw%100,AllocTres%100,AllocNodes,NodeList,NNodes --noheader --parsable2 | grep <account> I tried to exclude the steps from the main Jobs, so I summed only the main jobs whatever its state (canceled, completed, timeout, fail..). However the sreport occasionally is larger. What I am missing exactly? Warm regards, Hany.

1 0

Shared queue: how to request full node with all resources?
by Dustin Lang 31 Mar '26

31 Mar '26

Hi, With a partition with "OverSubscribe=FORCE" set, is there a way to request all the node resources? I see "--mem=0" does that for memory. But I do not see an option to request all the CPUs and GRES/TRES such as GPUs. I tried "--nodes=1 --ntasks=1 --cpus-per-task=0", but "--cpus-per-task=0" does not do the same thing as "--mem=0". In other words, is it possible to have a queue where nodes can be shared between jobs, but have a simple way for an sbatch script to request the full node with all its memory, cpus, and other resources? We have a heterogeneous cluster, so telling users (or jupyterhub scripts) to list exactly the resources they want doesn't really work. thanks, dustin

3 4

How do you handle GPU node failures during long jobs?
by Antonio Jose Alonso-Stepanov 26 Mar '26

26 Mar '26

Hi all, I'm a Stanford CS student looking into how sites handle GPU node failures during long-running jobs. A couple questions: When a GPU node goes down mid-job, do most sites use Slurm's requeue or --no-kill to handle it, or is it mostly manual drain and resubmit? Is anyone using HealthCheckProgram to catch GPU issues (like ECC errors via DCGM), or do you handle GPU health monitoring outside of Slurm? Curious what's worked and what hasn't. Thanks. Antonio

3 2

_node_config_validate: gres/gpu: Count changed on node (0 != 2)
by Xaver Stiensmeier 23 Mar '26

23 Mar '26

Hey Slurm-users list, while our regular gpu nodes are working fine, our on demand gpu nodes have a weird issue. They power up, I can ssh onto them and execute nvidia-smi on them without issue, but they are marked invalid and slurmctld logs _node_config_validate: gres/gpu: Count changed on node (0 != 2) however, node show shows that the gpus are recognized and the gres.conf are stored on the worker nodes as expected and the node entries in the slurm.conf are fine, too: # slurm.conf NodeName=my_worker_node SocketsPerBoard=16 CoresPerSocket=1 RealMemory=64075 MemSpecLimit=4000 State=CLOUD Gres=gpu:L4:2 # openstack # gres.conf on my_worker_node ubuntu@my_node:~$ cat /etc/slurm/gres.conf # GRES CONFIG Name=gpu Type=L4 File=/dev/nvidia0 Name=gpu Type=L4 File=/dev/nvidia1 Thankful for any ideas and debugging ideas. Best, Xaver PS: By executing: sudo scontrol update NodeName=$(bibiname 0) Gres= sudo scontrol reconfigure sudo scontrol update NodeName=$(bibiname 0) state=RESUME reason=None the node can be resumed. However, this is not a real solution.

2 3

We are pleased to announce the availability of Slurm versions 25.11.4 and 25.05.7.
by Marshall Garey 12 Mar '26

12 Mar '26

We are pleased to announce the availability of Slurm versions 25.11.4 and 25.05.7. Changes in both versions include preventing SIGHUP from terminating slurmrestd. Changes in 25.11.4 include fixing several issues with the switch/nvidia_imex plugin, several memory leaks, a race that could cause jobs to be stuck in the completing state or be requeued and held if the --get-user-env option was used, flex reservations taking nodes from other reservations that do not request full nodes, and a variety of other issues. Changes in 25.05.7 include fixing the gpu/nrt plugin, and fixing some other regressions and stability issues. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Server with two GPUs sharing each GPU to different partitions
by Gestió Servidors 09 Mar '26

09 Mar '26

Hello, First of all, sorry if my question is about something easy, but in my environment, this is the first time I need to do this. I have one server that have 2 identical GPUs. I have two different partitions: one is accessible from some allocation nodes and the other from other allocation nodes. My question is if it's possible to share one GPU with one partition and the other GPU with the other partition. For example, something like this: * gres.conf: NodeName=gpu-node AutoDetect=off Name=gpu Type=RTX5070 File=/dev/nvidia0 Cores=0-127 NodeName=gpu-node AutoDetect=off Name=gpu Type=RTX5070 File=/dev/nvidia1 Cores=128-255 * slurm.conf: NodeName=gpu-node CPUs=256 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=260000 TmpDisk=47000 Gres= RTX5070:2 PartitionName=gpu-int.q Nodes=gpu-node OverSubscribe=No State=UP AllocNodes=submit-node-1 PriorityTier=3 GraceTime=900 PartitionName=gpu-ext.q Nodes=gpu-node OverSubscribe=No State=UP AllocNodes=submit-node-1 PriorityTier=3 GraceTime=900 With this configuration, if simultaneously two different users tries to execute in one of the RTX5070 (one in gpu-int.q and the other in gpu-ext.q), could both of them execute with no problems? If user from gpu-int.q uses device #0 and user from gpu-ext.q uses device #1, could SLURM work without problems if a second simultaneously exection assigns device #1 to user from gpu-int.q and device #0 to user from gpu-ext.q? Thanks.

1 0

Optimizing CPU allocation in Slurm with hyperthreading enabled
by Guillaume COCHARD 02 Mar '26

02 Mar '26

Hello, The Slurm documentation states that "The default allocation method within a node is cyclic allocation (allocate available CPUs in a round-robin fashion across the sockets within a node)." In our case, we are using hyperthreading, which means that if two jobs requesting one CPU each arrive at the same time on an empty node, they will be assigned to the two threads of the same physical core. To improve performance, we would like Slurm to allocate CPUs (in our case, threads) in a way that prioritizes using one thread per physical core first, and only once all cores are occupied, assign the second thread of each core. This would be similar to the Least Loaded Node (LLN) approach, but instead of distributing jobs across nodes, it would distribute them across physical cores (Least Loaded Core). We do not want to disable hyperthreading; we simply want to optimize core usage when a machine is not fully loaded. We also want this to be a cluster-wide setting, and not a user and/or job setting. Our current slurm.conf contains: ConstrainCores = no SelectType = select/cons_tres SelectTypeParameters = CR_CPU_MEMORY And our cgroup.conf contains: ConstrainDevices=yes ConstrainCores=yes Does anyone know how to achieve this? Thanks, Guillaume

3 2

Missing mounts.h in Ubuntu Bionic build
by Michael Gutteridge 25 Feb '26

25 Feb '26

Hi We're running into a problem with building 25.11.3 for Ubuntu Bionic. When the cgroup/v2 plugin is building, ebpf throws an error: ``` ../../../../../src/plugins/cgroup/v2/ebpf.c:38:10: fatal error: linux/mount.h: No such file or directory #include <linux/mount.h> ^~~~~~~~~~~~~~~ compilation terminated. Makefile:638: recipe for target 'ebpf.lo' failed make[6]: *** [ebpf.lo] Error 1 make[6]: Leaving directory '/build/slurm-25.11.3/obj-x86_64-linux-gnu/src/plugins/cgroup/v2' ``` I think this is because on Ubuntu, the `mounts.h` header is in the path `sys`- the manpage for mounts(2) describes including this file via `#include <sys/mounts.h>`. `ebpf.c` seems to be the only place where the `linux/mounts.h` path is used, everywhere else using `sys`. Curiously it seems to build OK on Noble (Ubuntu 24). I have no explanation for the difference. Has anyone run into this problem building on Ubuntu? This include looks to have been a recent add so might be a bug, but I feel like someone else would have run into this already if it were. Thanks - Michael

1 0

Issue with coordinator increasing limits
by Groner, Rob 24 Feb '26

24 Feb '26

I'm seeing a weird issue when I try to increase limits on a child account that a coordinator has created. Here is the setup: 1. Create an account named slurm_parent, set coordinator1 as a coordinator of that account, and give that account a grptresmins=cpu=100 limit. 2. As coordinator1, create a child account named slurm_child whose parent is slurm_parent. 3. As coordinator1, set the grptresmins=cpu=10 limit on the child account. That all works fine. The problem is when I try to INCREASE the cpu limit on the child account by any amount. $coordinator1> sacctmgr modify account test_crch_slurmaccessch set grptresmins=cpu=11 sacctmgr: error: Coordinators can not increase job limits beyond the parent ones Modified account associations... Error with request: Coordinators can not increase job limits beyond the parent ones The first issue is that the limit of cpu=11 is NOT beyond the parent limit of cpu=100. The second issue is that (as a workaround), I can remove the cpu=10 limit on the child account as coordinator (by setting it to -1), and then I can set grptresmins=cpu=1000. That command works. So, I can very clearly set a grptresmins=cpu limit that is beyond the parent limit (I understand that the child account in reality won't be able to access more than the parent account, no matter what the limit is). Note that as coordinator, I can REDUCE the limit on the child account all I want. So I can set it to cpu=9 just fine. But if I try to set it back to 10, I'll again get the error saying I can't go above parent limits. So here are my issues: 1. Inability of coordinator to increase the child account grptresmins limits once set. 2. Incorrect error message about exceeding the parent limit. I originally saw this issue on 24.11.6, and built and tested on 25.11.3, and still see the issue. Thanks for any help figuring out what I'm doing wrong. Rob

1 1

Reliable, Atomic, Idempotent, or Transactional Job Submission?
by Adam Novak 24 Feb '26

24 Feb '26

I'm working on the Slurm integration in our Toil workflow runner project. I'm having a problem where an `sbatch` command to submit a job to Slurm can fail (with exit code 1 and message "sbatch: error: Batch job submission failed: Socket timed out on send/recv operation", in my case, but possibly in other ways), but the job can still actually have been submitted, and can still execute. This causes a problem for Toil because right now, when it sees a submission attempt fail, it backs off and submits the job again a little later. But Toil can't handle multiple copies of the same job running at once, and if a submission appears to the client to have failed but actually succeeded, it's possible to get into that situation if you just submit again. When an sbatch command fails, is it possible to detect cases where the cluster will still execute the job? (For example, is it guaranteed that the job ID will be available on standard output on the client if the job is going to execute on the cluster, no matter when in the client process it might get a socket disconnection, allowing the job to be inquired about later?) Do I maybe need to tag my jobs with unique identifiers myself so I can poll for them in the queue after a supposedly-failed submission? Is it possible to write an idempotent sbatch command, where it can be run any number of times but will only actually submit one copy of the job? Is the Slurm REST API somehow more transactional, or able to promise somehow that a job will not actually go into the queue without the client having acknowledged receipt of the job's assigned ID? Thanks, -Adam -- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code." Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5

4 7

Get the list of added dynamic nodes
by Edmundo Carmona Antoranz 23 Feb '26

23 Feb '26

Hi! Is there a way to get the list of dynamic nodes that have been created but that are pending to get slurmd to run on them? I fail to see them with with either scontrol or sinfo: root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol create NodeName=node004 .... state=FUTURE feature=dyn1 root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol version slurm 25.11.2 Only when I start slurmd from node004 does it become visible (through scontrol only): root@ea-u24-dynamic-2511:~# ssh node004 /cm/local/apps/slurm/current/sbin/slurmd -Z -f /cm/shared/apps/slurm/etc/slurm/slurm.conf root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 NodeName=node004 Arch=x86_64 CoresPerSocket=1 Thanks in advance. Edmundo Carmona Antoranz Software Engineer NVIDIA Corporation

3 4

We are pleased to announce the availability of Slurm version 25.11.3
by Tim Wickberg 19 Feb '26

19 Feb '26

We are pleased to announce the availability of Slurm version 25.11.3. 25.11.3 fixes a number of stability issues, regressions, and a variety of other bugs, including the following: * Fix multiple issues with dynamic topology when power saving is enabled * Fix a backfill performance regression when using bf_licenses * Fix race conditions when shutting down slurmctld or slurmdbd that cause crashes or hangs * Fix slurmctld crashing when using enable_async_reply and TLS * Fix regression from 25.05 which caused usercpu and systemcpu to be missing for job steps * Fix regression that caused thread limits in slurmd to not be respected for incoming RPCs * Prevent the double-removal of accounting usage for jobs being requeued that are in the COMPLETED or COMPLETING state The full list of changes is available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases One note on some updates to our release process - we've started publishing the releases directly on GitHub, in parallel to their usual location on the SchedMD web server. If you 'star' the Slurm repository you can automatically receive notices when new releases are published there. You can also subscribe to the RSS/Atom feed through: https://github.com/SchedMD/slurm/releases.atom - Tim -- Tim Wickberg Director, System Software, Slurm and Slinky, NVIDIA

1 0

preempt/qos is not working as expected
by Marchand Aurélia 19 Feb '26

19 Feb '26

Hello, I want too use QOS preemption. my configuration is : preemptMode = GANG,SUSPEND PreemptParameters = (null) PreemptType = preempt/qos PriorityType = priority/multifactor SelectType = select/cons_tres SelectTypeParameters = CR_CPU I define 3 partitions : PartitionName=DEFAULT Nodes=ALL Default=NO MaxTime=15-0 State=UP OverSubscribe=FORCE:1 PreemptMode=suspend PartitionName=veryhi PriorityTier=30 AllowQos=veryhi_short,gaia_veryhi_short PartitionName=hi PriorityTier=20 AllowQos=hi_short,gaia_hi_short PartitionName=def PriorityTier=10 OverSubscribe=No Default=YES AllowQos=def_short I define 5 QOS : sacctmgr show qos format=name%20,Priority,Preempt%40,PreemptMode Name Priority Preempt PreemptMode ----------------------- ------------ ---------------------------------- --------------- normal 0 cluster veryhi_short 20000 def_short suspend hi_short 10000 def_short suspend def_short 0 normal suspend gaia_veryhi_short 20000 def_short,gaia_hi_short suspend gaia_hi_short 10000 def_short suspend I define account : sacctmgr show ass format=account%20,user,qos%60 Account User QOS -------------------- -------- -------------------------------------------- gaia_hi user1 gaia_hi_short gaia_veryhi user1 gaia_veryhi_short mis_hi user1 hi_short def user1 def_short I submit 2 jobs : JOBID PARTITION QOS ACCOUNT NAME USER STATE PRIORITY 159 hi hi_short mis_hi test user1 SUSPENDE 637 160 veryhi gaia_veryhi_short gaia_veryhi test user1 RUNNING 1150 I don't understand why job 160 suspend job 159, the partition priority is higher, but I want gaia_veryhi_short QOS preempt only def_short and gaia_hi_short QOS. The QOS of job 159 is hi_short. Where is my configuration error ? Thanks for your time, Aurélia

2 4

for srun jobs, if we try to get UserCPU time by sacct. it always shows 0.
by squallabc＠126.com 16 Feb '26

16 Feb '26

Hi we have a slurm v25.05.2 setup. we found any srun jobs are always shows 0 on UserCPU . for sbatch jobs shoiws normal. is this expected or it is a bug ? Thanks. sacct -j 2948 --format=AveCPU,UserCPU AveCPU UserCPU ---------- ---------- 00:00:00 00:02:00 00:00:00

4 4

Sharding GPUs
by alessandr.dauria＠gmail.com 13 Feb '26

13 Feb '26

Dear Community, We are trying to activate sharding. Our Compute Node are configured with 64 cores, 4 phisical GPU MI250x ( 8 logical ) 4 Numa Domain. 1 Phisical Gpu / 2 logical GPU for each Numa Domain. 1 Logical GPU for each l3 cache domain gres.conf AutoDetect=rsmi NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD128 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD129 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD130 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD131 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD132 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD133 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD134 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD135 Count=4 If I ask 2 cores with block:cyclic I have the expected result srun -N1 -n2 -c1 --cpu-bind=cores -m block:cyclic --pty bash cpuset cgroup is 1,17 But if I add 2 shard in the request I don't expect this result srun -N1 -n2 -c1 --cpu-bind=cores --gres=shard:2 -m block:cyclic --pty bash cpuset cgroup is 1-2 ROCR_VISIBILE_DEVICES=0 Is it possibile request 2 sharding in round robin fashion, in order to run a multigpu job on different GPUs? srun -N1 -n2 -c1 --cpu-bind=cores --gres=shard:2 -m block:cyclic --pty bash Practically, I would to have this result cpuset cgroup is 1-17 ROCR_VISIBILE_DEVICES=0,1 Thank you in advance, Alessandro

1 1

Restarting slurmd kills still-running jobs
by Griebel, Christian 13 Feb '26

13 Feb '26

Dear community, Trying to implement the latest fix/patch for munged, we restarted the updated munged locally on the compute nodes with "systemctl restart munged", resulting in the sudden death of a lot of compute nodes' slurmd. Checking the jobs on the affected nodes, we saw a lot of user processes/jobs still running, which was good - yet "systemctl restart slurmd" cancelled all of them, eg. [2026-02-12T17:08:00.325] Cleaning up stray StepId=49695760.extern [2026-02-12T17:08:00.325] [49695760.extern] Sent signal 9 to StepId=49695760.extern [2026-02-12T17:08:00.325] slurmd version 25.05.5 started and all affected user jobs (even though having survived the death of their parent slurmd) were killed and re-queued... We have cgroups v2 (only, no hybrid), "Delegate=yes" in the slurmd unit and "ProctrackType=proctrack/cgroup" configured. Other sites do not see the same behavior (their user jobs survive a slurmd restart without issues), so now we are at a loss figuring out why the h.... this happens within our setup. Anyone experienced similar problems and got them solved...? Thanks in advance - -- ___________________________ Christian Griebel/HPC

6 5

sstat is not able check others job.
by squallabc＠126.com 12 Feb '26

12 Feb '26

Hi slurm version is v25.05.2. when I tried to query others job by sstat, it returns error. is there any configuration we can setup to fix this ? Thanks. sstat -j 3023.batch JobID AllocTRES MaxVMSize MaxVMSizeNode MaxVMSizeTask AveVMSize MaxRSS MaxRSSNode MaxRSSTask AveRSS MaxPages MaxPagesNode MaxPagesTask AvePages MinCPU MinCPUNode MinCPUTask AveCPU NTasks AveCPUFreq ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov ConsumedEnergy MaxDiskRead MaxDiskReadNode MaxDiskReadTask AveDiskRead MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask AveDiskWrite TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask TRESUsageInMin TRESUsageInMinNode TRESUsageInMinTask TRESUsageInTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot ------------ ---------- ---------- -------------- -------------- ---------- ---------- ---------- ---------- ---------- -------- ------------ -------------- ---------- ---------- ---------- ---------- ---------- -------- ---------- ------------- ------------- ------------- -------------- ------------ --------------- --------------- ------------ ------------ ---------------- ---------------- ------------ -------------- -------------- ------------------ ------------------ -------------- ------------------ ------------------ -------------- --------------- --------------- ------------------- ------------------- --------------- ------------------- ------------------- --------------- sstat: error: slurm_job_step_stat: there was an error with the request to atlsde8-reg0001 rc = Invalid user id sstat: error: problem getting step_layout for StepId=3023.batch: Invalid user id

3 2

MUNGE security issue (CVE-2026-25506)
by Tim Wickberg 12 Feb '26

12 Feb '26

MUNGE just announced a serious security issue as CVE-2026-25506: https://github.com/dun/munge/security/advisories/GHSA-r9cr-jf4v-75gh The issue is entirely within MUNGE - specifically in the munged daemon. However, as most Slurm installations explicitly trust MUNGE as the source of authentication, Slurm's security is immediately undermined if an attacker were to exploit that issue and gain access to the MUNGE key. On Slurm clusters you should treat that security issue as if it were local-root exploit. If you have switched to Slurm's built-in authentication ("auth/slurm"), you are not impacted by this issue. (However, we do not recommend switching over as a mitigation - you cannot safely change authentication types in Slurm while jobs are currently running.) I strongly encourage sites to get install the fixed packages from the upstream distros and restart the munged daemon on all nodes. If you install and restart munged promptly then Slurm should be unaffected. You do not need to rebuild Slurm for this issue. The MUNGE ABI/API remain unchanged, so there is no need to recompile Slurm, and there are no patches to apply to Slurm related to this issue. If you are concerned that an attacker could have compromised the MUNGE key itself, you may want to rotate that key. Unfortunately there's no mechanism to safely handle this key rotation with the cluster operational, and I do not recommend key rotation as long as you're installing and restarting MUNGE ASAP. - Tim -- Tim Wickberg Director, System Software, Slurm and Slinky, NVIDIA

11 16

Can I update when jobs are running?
by Gould, Ron (GRC-VBA0)[AEGIS] 11 Feb '26

11 Feb '26

The tl;dr is “This is my first upgrade since inheriting this Cluster, so I’m not sure what can or can’t be running during the upgrades.”. My Cluster is running an old version, 22.05.3. This is my first upgrade since inheriting the Cluster. As such, I’d like to install 22.05.4 because it’s a short jump, and it fixes the bug my users are seeing. The Cluster is composed of mostly Oracle Linux 8. I’m aware that I can upgrade within the two release compatibility window. I’ve read through the Upgrade guide and I’m unclear if downtime is required. Perhaps I’m unifying downtime requirements across different SLURM services where I should be interpreting that certain services have their own downtime requirements. https://slurm.schedmd.com/upgrades.html<https://slurm.schedmd.com/upgrades.html#procedure> In the Upgrade Procedure section<https://slurm.schedmd.com/upgrades.html#procedure>, there’re a couple questionable things. 1. Is downtime required? Does downtime == “all jobs must be halted”? “Downtime”, to me, seems like nothing should be running. This statement indicates that jobs can be running during the upgrade. Before considering the upgrade complete, wait for all jobs that were already running to finish. Any jobs started before the slurmd system was upgraded will be running with the old version of slurmstepd, so starting another upgrade or trying to use new features in the new version may cause problems. within a few paragraphs<https://slurm.schedmd.com/upgrades.html#downtime>, this message indicates I will need downtime: Refer to the expected downtime guidance in the following sections for each relevant Slurm daemon Further in the guide, in SLURMD (COMPUTE NODES)<https://slurm.schedmd.com/upgrades.html#slurmd>, I read Upgrades will not interrupt running jobs as long as SlurmdTimeout is not reached during the process This implies, at least, that existing running jobs can stay running. 2. There’re conflicting suggestions of using “rpm” to install the RPMs I built with “rpmbuild". Should I use “dnf localinstall ./*.rpm”? I’m inferring that dependencies will (not) be handled correctly. NOTE: If RPM/DEB packages are used, all packages present on each system must be upgraded together instead of piecewise. … Avoid using low-level package managers like rpm or dpkg as they may not properly enforce these dependencies However, in SLURMDBD (ACCOUNTING)<https://slurm.schedmd.com/upgrades.html#slurmdbd>, this statement Upgrade the slurmdbd daemon binaries, libraries, and its systemd unit file (if used). If using RPM/DEB packages, the package manager will take care of these indicates I should be using RPM packages. Lastly, to get to a current install, I need to step through multiple versions, with the condition that jobs started with a specific major version must finish within the compatibility window. GitLab has a tool where you plug in your current and intended versions and it tells you explicitly which versions are required along the upgrade path. I’d like a similarly explicit tool for SLURM, but I infer from the Compatibility Window<https://slurm.schedmd.com/upgrades.html#compatibility_window> that I can update like so: 1. Current = 22.05.3 2. 23.11 3. 25.05 4. 26.05 That feels like a big leapfrog between versions. I’d like the practice of upgrading. Is there any detriment to upgrading at a slower pace: 1. Current = 22.05.3 2. 22.05.11 3. 23.02.8 4. 23.11.11 5. 24.05.8 6. 24.11.7 7. 25.05.6 8. 25.11.2

9 26

Adding FPGAs for Slurm Tracking
by Raven Hallsby 08 Feb '26

08 Feb '26

Hey all, We have a very heterogeneous cluster with a mix of Intel & AMD machines, GPUs, and FPGAs. We use Slurm to allocate time on the CPUs right now, but do not manage the GPU or FPGAs with Slurm yet. Some of our work involves hardware/SoC design, and we have been using Firesim (https://fires.im/) for a good amount of our work. Firesim is a bunch of Python that sits on top of invoking programs to provide "seamless" functionality and orchestration. However, Firesim was designed for AWS where a single user controls all of the resources (FPGAs in our case) used for the simulation. We have gone through the effort to pull out the important bits from Firesim to write a shell script that we can execute locally to achieve the same thing as Firesim' orchestration, without the need for an orchestrator. We have two FPGAs in two separate but identical machines and multiple people that want to use them for very long periods of time (days or longer) and want to let people freely move between the two FPGAs. All of our other machines in the cluster use Slurm, so we are looking to have Slurm multiplex these FPGA jobs. We already have the NFS stuff set up, so shared disk is fine. The important thing in our desired Slurm setup is that no user is allocated the same FPGA at the same time for the same job (you cannot have multiple FPGA jobs running the same design in parallel, partly because of bitstreams, partly because of disk images). Slurm does not track FPGAs by default. I have looked into how to teach Slurm about the FPGAs and know that I need to tell Slurm about the FPGAs as a GRES through gres.conf. I just want to confirm that I have the correct steps below: 1. Mark the 2 FPGAs as GRES with gres.conf. What should they be marked as? Do I need anything special? 2. Then enable tracking on the FPGA GRES to make a TRES in slurm.conf. What should this be labeled as? I see other groups using GresTypes=gpu in their configuration. Would that apply here? Since FPGA usage is exclusive already, I could see the GPU GRES type applying. 3. Then the TRES can be used by a QoS to enforce limits? The other important question, will I need to write a Slurm plugin (GRES or otherwise) to manage and detect the FPGAs? I saw the files stuff for GRES-es, but want to be sure. Writing a plugin is not a big deal for us, but it is future maintenance that I would want to document.

1 0