- slurm-users - lists.schedmd.com

Missing mounts.h in Ubuntu Bionic build
by Michael Gutteridge 25 Feb '26

25 Feb '26

Hi We're running into a problem with building 25.11.3 for Ubuntu Bionic. When the cgroup/v2 plugin is building, ebpf throws an error: ``` ../../../../../src/plugins/cgroup/v2/ebpf.c:38:10: fatal error: linux/mount.h: No such file or directory #include <linux/mount.h> ^~~~~~~~~~~~~~~ compilation terminated. Makefile:638: recipe for target 'ebpf.lo' failed make[6]: *** [ebpf.lo] Error 1 make[6]: Leaving directory '/build/slurm-25.11.3/obj-x86_64-linux-gnu/src/plugins/cgroup/v2' ``` I think this is because on Ubuntu, the `mounts.h` header is in the path `sys`- the manpage for mounts(2) describes including this file via `#include <sys/mounts.h>`. `ebpf.c` seems to be the only place where the `linux/mounts.h` path is used, everywhere else using `sys`. Curiously it seems to build OK on Noble (Ubuntu 24). I have no explanation for the difference. Has anyone run into this problem building on Ubuntu? This include looks to have been a recent add so might be a bug, but I feel like someone else would have run into this already if it were. Thanks - Michael

1 0

Issue with coordinator increasing limits
by Groner, Rob 24 Feb '26

24 Feb '26

I'm seeing a weird issue when I try to increase limits on a child account that a coordinator has created. Here is the setup: 1. Create an account named slurm_parent, set coordinator1 as a coordinator of that account, and give that account a grptresmins=cpu=100 limit. 2. As coordinator1, create a child account named slurm_child whose parent is slurm_parent. 3. As coordinator1, set the grptresmins=cpu=10 limit on the child account. That all works fine. The problem is when I try to INCREASE the cpu limit on the child account by any amount. $coordinator1> sacctmgr modify account test_crch_slurmaccessch set grptresmins=cpu=11 sacctmgr: error: Coordinators can not increase job limits beyond the parent ones Modified account associations... Error with request: Coordinators can not increase job limits beyond the parent ones The first issue is that the limit of cpu=11 is NOT beyond the parent limit of cpu=100. The second issue is that (as a workaround), I can remove the cpu=10 limit on the child account as coordinator (by setting it to -1), and then I can set grptresmins=cpu=1000. That command works. So, I can very clearly set a grptresmins=cpu limit that is beyond the parent limit (I understand that the child account in reality won't be able to access more than the parent account, no matter what the limit is). Note that as coordinator, I can REDUCE the limit on the child account all I want. So I can set it to cpu=9 just fine. But if I try to set it back to 10, I'll again get the error saying I can't go above parent limits. So here are my issues: 1. Inability of coordinator to increase the child account grptresmins limits once set. 2. Incorrect error message about exceeding the parent limit. I originally saw this issue on 24.11.6, and built and tested on 25.11.3, and still see the issue. Thanks for any help figuring out what I'm doing wrong. Rob

1 1

Reliable, Atomic, Idempotent, or Transactional Job Submission?
by Adam Novak 24 Feb '26

24 Feb '26

I'm working on the Slurm integration in our Toil workflow runner project. I'm having a problem where an `sbatch` command to submit a job to Slurm can fail (with exit code 1 and message "sbatch: error: Batch job submission failed: Socket timed out on send/recv operation", in my case, but possibly in other ways), but the job can still actually have been submitted, and can still execute. This causes a problem for Toil because right now, when it sees a submission attempt fail, it backs off and submits the job again a little later. But Toil can't handle multiple copies of the same job running at once, and if a submission appears to the client to have failed but actually succeeded, it's possible to get into that situation if you just submit again. When an sbatch command fails, is it possible to detect cases where the cluster will still execute the job? (For example, is it guaranteed that the job ID will be available on standard output on the client if the job is going to execute on the cluster, no matter when in the client process it might get a socket disconnection, allowing the job to be inquired about later?) Do I maybe need to tag my jobs with unique identifiers myself so I can poll for them in the queue after a supposedly-failed submission? Is it possible to write an idempotent sbatch command, where it can be run any number of times but will only actually submit one copy of the job? Is the Slurm REST API somehow more transactional, or able to promise somehow that a job will not actually go into the queue without the client having acknowledged receipt of the job's assigned ID? Thanks, -Adam -- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code." Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5

4 7

Get the list of added dynamic nodes
by Edmundo Carmona Antoranz 24 Feb '26

24 Feb '26

Hi! Is there a way to get the list of dynamic nodes that have been created but that are pending to get slurmd to run on them? I fail to see them with with either scontrol or sinfo: root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol create NodeName=node004 .... state=FUTURE feature=dyn1 root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol version slurm 25.11.2 Only when I start slurmd from node004 does it become visible (through scontrol only): root@ea-u24-dynamic-2511:~# ssh node004 /cm/local/apps/slurm/current/sbin/slurmd -Z -f /cm/shared/apps/slurm/etc/slurm/slurm.conf root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 NodeName=node004 Arch=x86_64 CoresPerSocket=1 Thanks in advance. Edmundo Carmona Antoranz Software Engineer NVIDIA Corporation

3 4

We are pleased to announce the availability of Slurm version 25.11.3
by Tim Wickberg 19 Feb '26

19 Feb '26

We are pleased to announce the availability of Slurm version 25.11.3. 25.11.3 fixes a number of stability issues, regressions, and a variety of other bugs, including the following: * Fix multiple issues with dynamic topology when power saving is enabled * Fix a backfill performance regression when using bf_licenses * Fix race conditions when shutting down slurmctld or slurmdbd that cause crashes or hangs * Fix slurmctld crashing when using enable_async_reply and TLS * Fix regression from 25.05 which caused usercpu and systemcpu to be missing for job steps * Fix regression that caused thread limits in slurmd to not be respected for incoming RPCs * Prevent the double-removal of accounting usage for jobs being requeued that are in the COMPLETED or COMPLETING state The full list of changes is available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases One note on some updates to our release process - we've started publishing the releases directly on GitHub, in parallel to their usual location on the SchedMD web server. If you 'star' the Slurm repository you can automatically receive notices when new releases are published there. You can also subscribe to the RSS/Atom feed through: https://github.com/SchedMD/slurm/releases.atom - Tim -- Tim Wickberg Director, System Software, Slurm and Slinky, NVIDIA

1 0

preempt/qos is not working as expected
by Marchand Aurélia 19 Feb '26

19 Feb '26

Hello, I want too use QOS preemption. my configuration is : preemptMode = GANG,SUSPEND PreemptParameters = (null) PreemptType = preempt/qos PriorityType = priority/multifactor SelectType = select/cons_tres SelectTypeParameters = CR_CPU I define 3 partitions : PartitionName=DEFAULT Nodes=ALL Default=NO MaxTime=15-0 State=UP OverSubscribe=FORCE:1 PreemptMode=suspend PartitionName=veryhi PriorityTier=30 AllowQos=veryhi_short,gaia_veryhi_short PartitionName=hi PriorityTier=20 AllowQos=hi_short,gaia_hi_short PartitionName=def PriorityTier=10 OverSubscribe=No Default=YES AllowQos=def_short I define 5 QOS : sacctmgr show qos format=name%20,Priority,Preempt%40,PreemptMode Name Priority Preempt PreemptMode ----------------------- ------------ ---------------------------------- --------------- normal 0 cluster veryhi_short 20000 def_short suspend hi_short 10000 def_short suspend def_short 0 normal suspend gaia_veryhi_short 20000 def_short,gaia_hi_short suspend gaia_hi_short 10000 def_short suspend I define account : sacctmgr show ass format=account%20,user,qos%60 Account User QOS -------------------- -------- -------------------------------------------- gaia_hi user1 gaia_hi_short gaia_veryhi user1 gaia_veryhi_short mis_hi user1 hi_short def user1 def_short I submit 2 jobs : JOBID PARTITION QOS ACCOUNT NAME USER STATE PRIORITY 159 hi hi_short mis_hi test user1 SUSPENDE 637 160 veryhi gaia_veryhi_short gaia_veryhi test user1 RUNNING 1150 I don't understand why job 160 suspend job 159, the partition priority is higher, but I want gaia_veryhi_short QOS preempt only def_short and gaia_hi_short QOS. The QOS of job 159 is hi_short. Where is my configuration error ? Thanks for your time, Aurélia

2 4

for srun jobs, if we try to get UserCPU time by sacct. it always shows 0.
by squallabc＠126.com 16 Feb '26

16 Feb '26

Hi we have a slurm v25.05.2 setup. we found any srun jobs are always shows 0 on UserCPU . for sbatch jobs shoiws normal. is this expected or it is a bug ? Thanks. sacct -j 2948 --format=AveCPU,UserCPU AveCPU UserCPU ---------- ---------- 00:00:00 00:02:00 00:00:00

4 4

Sharding GPUs
by alessandr.dauria＠gmail.com 13 Feb '26

13 Feb '26

Dear Community, We are trying to activate sharding. Our Compute Node are configured with 64 cores, 4 phisical GPU MI250x ( 8 logical ) 4 Numa Domain. 1 Phisical Gpu / 2 logical GPU for each Numa Domain. 1 Logical GPU for each l3 cache domain gres.conf AutoDetect=rsmi NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD128 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD129 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD130 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD131 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD132 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD133 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD134 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD135 Count=4 If I ask 2 cores with block:cyclic I have the expected result srun -N1 -n2 -c1 --cpu-bind=cores -m block:cyclic --pty bash cpuset cgroup is 1,17 But if I add 2 shard in the request I don't expect this result srun -N1 -n2 -c1 --cpu-bind=cores --gres=shard:2 -m block:cyclic --pty bash cpuset cgroup is 1-2 ROCR_VISIBILE_DEVICES=0 Is it possibile request 2 sharding in round robin fashion, in order to run a multigpu job on different GPUs? srun -N1 -n2 -c1 --cpu-bind=cores --gres=shard:2 -m block:cyclic --pty bash Practically, I would to have this result cpuset cgroup is 1-17 ROCR_VISIBILE_DEVICES=0,1 Thank you in advance, Alessandro

1 1

Restarting slurmd kills still-running jobs
by Griebel, Christian 13 Feb '26

13 Feb '26

Dear community, Trying to implement the latest fix/patch for munged, we restarted the updated munged locally on the compute nodes with "systemctl restart munged", resulting in the sudden death of a lot of compute nodes' slurmd. Checking the jobs on the affected nodes, we saw a lot of user processes/jobs still running, which was good - yet "systemctl restart slurmd" cancelled all of them, eg. [2026-02-12T17:08:00.325] Cleaning up stray StepId=49695760.extern [2026-02-12T17:08:00.325] [49695760.extern] Sent signal 9 to StepId=49695760.extern [2026-02-12T17:08:00.325] slurmd version 25.05.5 started and all affected user jobs (even though having survived the death of their parent slurmd) were killed and re-queued... We have cgroups v2 (only, no hybrid), "Delegate=yes" in the slurmd unit and "ProctrackType=proctrack/cgroup" configured. Other sites do not see the same behavior (their user jobs survive a slurmd restart without issues), so now we are at a loss figuring out why the h.... this happens within our setup. Anyone experienced similar problems and got them solved...? Thanks in advance - -- ___________________________ Christian Griebel/HPC

6 5

sstat is not able check others job.
by squallabc＠126.com 13 Feb '26

13 Feb '26

Hi slurm version is v25.05.2. when I tried to query others job by sstat, it returns error. is there any configuration we can setup to fix this ? Thanks. sstat -j 3023.batch JobID AllocTRES MaxVMSize MaxVMSizeNode MaxVMSizeTask AveVMSize MaxRSS MaxRSSNode MaxRSSTask AveRSS MaxPages MaxPagesNode MaxPagesTask AvePages MinCPU MinCPUNode MinCPUTask AveCPU NTasks AveCPUFreq ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov ConsumedEnergy MaxDiskRead MaxDiskReadNode MaxDiskReadTask AveDiskRead MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask AveDiskWrite TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask TRESUsageInMin TRESUsageInMinNode TRESUsageInMinTask TRESUsageInTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot ------------ ---------- ---------- -------------- -------------- ---------- ---------- ---------- ---------- ---------- -------- ------------ -------------- ---------- ---------- ---------- ---------- ---------- -------- ---------- ------------- ------------- ------------- -------------- ------------ --------------- --------------- ------------ ------------ ---------------- ---------------- ------------ -------------- -------------- ------------------ ------------------ -------------- ------------------ ------------------ -------------- --------------- --------------- ------------------- ------------------- --------------- ------------------- ------------------- --------------- sstat: error: slurm_job_step_stat: there was an error with the request to atlsde8-reg0001 rc = Invalid user id sstat: error: problem getting step_layout for StepId=3023.batch: Invalid user id

3 2