- slurm-users - lists.schedmd.com

cgroup/v2 regression in 25.11.2 — memory.peak read disabled after first task
by Emyr James 06 May '26

06 May '26

Dear all, Heads-up for anyone running 25.11.2 with cgroup/v2 + jobacct_gather/cgroup: there's a regression in the recent series that effectively disables memory.peak accounting for every task after the first one per slurmstepd. The commit is f2d05c08e6 ("cgroup/v2 - Remove static qualifier from variable", ticket 23646, cherry-picked from e1835a1602). It dropped static from memory_peak_interface in cgroup_p_task_get_acct_data while leaving interfaces_checked static. The function now looks like: bool memory_peak_interface = false; static bool interfaces_checked = false; ... if (!interfaces_checked) { memory_peak_interface = cgroup_p_has_feature(CG_MEMCG_PEAK); interfaces_checked = true; } ... if (memory_peak_interface) { common_cgroup_get_param(..., "memory.peak", ...); } interfaces_checked persists across calls; memory_peak_interface no longer does. So the feature-probe block runs exactly once. From the second call onward memory_peak_interface is a fresh stack-local set to false, and the memory.peak read in the second if is skipped. stats->memory_peak ends up INFINITE64 for every task except the first one ever sampled. User-visible effect: MaxRSS in sacct stops reflecting the kernel-tracked peak and falls back to polling memory.current — which, at typical jobacct_gather intervals, routinely misses short-lived peaks. Which is of course exactly the problem memory.peak was added to solve (see the 2024 thread that prompted the original implementation: https://groups.google.com/g/slurm-users/c/BX1_4bztrJA) One-line fix is to put static back on memory_peak_interface. Or, arguably more robust, drop the interfaces_checked cache entirely and run the feature probe on every call — the cost is one access()-equivalent per accounting poll, basically free. he original v2 implementation in ba34862a40 was great, thanks for that — this looks like a small static-qualifier slip on the way through a cleanup. Cheers, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

1 1

slurmdbd and slurmctld prevent alma9 login
by Pols, Maarten 06 May '26

06 May '26

Dear Community, Our Slurm cluster has been running without any issues for several months on version 25.05.3. Last Friday, we experienced a power outage which required us to restart the server. After the restart, we were unable to log in to the master node. Eventually, we managed to access the system via a safe mode workaround. Through a process of elimination, we identified the slurmdbd and slurmctld services as the root cause of the issue. Would you happen to have any idea what might have caused this behavior? We have since upgraded to version 25.11.5, which appears to be running smoothly. However, we would still like to understand the underlying cause of the problem. Thank you in advance for your help. Kind regards, Maarten

3 3

Jobs canceling when nodes become unreachable – need guidance
by Pharthiphan Asokan 05 May '26

05 May '26

Hi, We’re seeing an issue where jobs submitted via salloc are automatically cancelled when a compute node becomes temporarily unreachable. Our goal is to keep jobs pending or requeued instead of being cancelled outright when a node drops offline briefly Slurm sometimes cancels the job rather than requeuing it when the node is marked DOWN/ DRAIN/ DRAINING*. Is there a recommended configuration or additional parameter that ensures jobs remain pending/requeued until the node returns, rather than being cancelled? Any insights or examples from similar setups would be greatly appreciated. Regards, Pharthiphan

3 2

How to track solver usage in Slurm
by fernando.seguro＠rtx.com 23 Apr '26

23 Apr '26

I don't see an easy way of tracking this in slurm. I tried using the --comment="slover" directive, but this gets mapped to the "derived_es" column in the database, which is reserved for job info on errors, from what I read. This would make a good enhancement request. Where would I submit that? Something like: SBATCH --solver=fluent or SBATCH --solver=LSDyna This would be very helpful for tracking solver usage without having to use the job-name. Great for reporting. Please let me know if someone else has requested this previously. If anyone has figured out a way to do it, please share. thanks

4 3

Can't clear the REASON
by Berg, Stephen P CIV USN NRL DET SSC MS (USA) 17 Apr '26

17 Apr '26

Just upgraded my grid to 25.11 a couple days ago. Got two nodes that were down for a bit and when I set them to resume and set the reason="" they are still showing up with the reason I had set while they were down. Nothing I do seems to unset that reason flag for these two nodes. I've set the nodes to down with a reason and then back to idle with the blank reason but it never goes back to "none" like the other nodes. Whatever the last reason set was persists. The behavior seems different since I updated to 25.11. Was running the 20.11 available from the epel-8 repos previously.

2 4

Jobs aborting after slurmctld reload on Intel nodes - AMD unaffected
by Pharthiphan Asokan 16 Apr '26

16 Apr '26

Hi team, We’re observing job aborts on Intel-based nodes immediately after a slurmctld reload. AMD nodes remain stable and jobs continue unaffected. No system or Slurm configuration changes were made before the issue started. Error observed: error: Aborting JobID=1288 due to change in socket/core configuration of allocated nodes Relevant node configuration (Intel node example): NodeName=smc-h4-u19 CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=1 RealMemory=510976 State=UNKNOWN Feature=model_SSG-1228-B From logs: error: valid_job_resources: smc-h4-u19 sockets:2:2, cores 64,32 error: Node configuration differs from hardware: CPUs=128:128(hw) Boards=1:1(hw) SocketsPerBoard=2:2(hw) What we’ve verified: * No changes in BIOS, firmware, or hardware topology * No edits to slurm.conf or slurm_nodes.conf * Reload (scontrol reconfigure) triggers job aborts only on Intel nodes * AMD nodes remain intact through reloads

3 2

We are pleased to announce the availability of Slurm version 25.11.5
by Marshall Garey 14 Apr '26

14 Apr '26

We are pleased to announce the availability of Slurm version 25.11.5. Some of the changes in 25.11.0 are highlighted below: * Fix a regression in 25.11.0 that could cause daemons to hang after reaching conmgr_max_connections. * Fix a regression added in 25.05.0 concerning how the slurmctld inherits /run/slurmctld/sack.socket when using AuthType=auth/slurm to prevent clients that connected during a reconfigure from hanging indefinitely. * Fix a race in tree forwarding that could cause a crash during shutdown * Fix several bugs in the namespace/linux plugin that caused instability The full list of changes are available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Why my job can't start (backfill reservation issue)
by Massimo Sgaravatto 14 Apr '26

14 Apr '26

Dear all I (try to) manage a slurm cluster composed by some CPU-only nodes and some worker nodes which have also GPUs: NodeName=cld-ter-[01-06] Sockets=2 CoresPerSocket=96 ThreadsPerCore=2 RealMemory=1536000 State=UNKNOWN NodeName=cld-ter-gpu-[01-05] Sockets=2 CoresPerSocket=96 ThreadsPerCore=2 Gres=gpu:nvidia-h100:4 RealMemory=1536000 State=UNKNOWN The GPU nodes are exposed through multiple partitions: PartitionName=gpus Nodes=cld-ter-gpu-[01-02] State=UP PriorityTier=20 PartitionName=sparch Nodes=cld-ter-gpu-03 AllowAccounts=sparch,operators QoS=sparch State=UP PriorityTier=20 PartitionName=geant4 Nodes=cld-ter-gpu-03 AllowAccounts=geant4,operators QoS=geant4 State=UP PriorityTier=20 PartitionName=enipred Nodes=cld-ter-gpu-04 AllowAccounts=enipred,operators QoS=enipred State=UP PriorityTier=20 PartitionName=enipiml Nodes=cld-ter-gpu-05 AllowAccounts=enipiml,operators QoS=enipiml State=UP PriorityTier=20 We also set a partition to allow cpu-only jobs on the GPU nodes, but these jobs should be preempted (killed and requeued) if jobs submitted to partitions with higher priorities require those resources: PreemptType=preempt/partition_prio PreemptMode=REQUEUE PartitionName=onlycpus-opp Nodes=cld-ter-gpu-[01-05],cld-dfa-gpu-06,btc-dfa-gpu-02 State=UP PriorityTier=10 Now, I don't understand why this job [*] submitted on the onlycpus-opp partition can't start running e.g. on the cld-ter-gpu-01, since it has a lot of free resources: [sgaravat@cld-ter-ui-01 ~]$ scontrol show node cld-ter-gpu-01 NodeName=cld-ter-gpu-01 Arch=x86_64 CoresPerSocket=96 CPUAlloc=8 CPUEfctv=384 CPUTot=384 CPULoad=5.93 AvailableFeatures=(null) ActiveFeatures=(null) Gres=gpu:nvidia-h100:4 NodeAddr=cld-ter-gpu-01 NodeHostName=cld-ter-gpu-01 Version=25.11.3 OS=Linux 5.14.0-611.45.1.el9_7.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 1 05:56:53 EDT 2026 RealMemory=1536000 AllocMem=560000 FreeMem=1192357 Sockets=2 Boards=1 State=MIXED+PLANNED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpus,onlycpus-opp BootTime=2026-04-09T10:39:35 SlurmdStartTime=2026-04-09T10:40:01 LastBusyTime=2026-04-09T11:54:46 ResumeAfterTime=None CfgTRES=cpu=384,mem=1500G,billing=839,gres/gpu=4,gres/gpu:nvidia-h100=4 AllocTRES=cpu=8,mem=560000M,gres/gpu=4,gres/gpu:nvidia-h100=4 CurrentWatts=0 AveWatts=0 I guess the "MIXED+PLANNED" is the answer, but as far as I can see only a job (283469) is planned for this worker node: sgaravat@cld-ter-ui-01 ~]$ squeue --start | grep ter-gpu-01 JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON) 283469 gpus vllm-pod ciangott PD 2026-04-13T14:31:40 1 cld-ter-gpu-01 (Resources) But job 283469 doesn't require too many resources [**], so the 2 jobs could run together. Why job 283534 can't start ? Any hints ? Thanks, Massimo [*] [sgaravat@cld-ter-ui-01 ~]$ scontrol show job=283534 JobId=283534 JobName=myscript.sh UserId=sgaravat(5008) GroupId=tbadmin(5001) MCS_label=N/A Priority=542954 Nice=0 Account=operators QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:41 TimeLimit=1-00:00:00 TimeMin=N/A SubmitTime=2026-04-13T11:10:13 EligibleTime=2026-04-13T11:10:13 AccrueTime=2026-04-13T11:10:13 StartTime=2026-04-13T11:58:39 EndTime=2026-04-14T11:58:39 Deadline=N/A PreemptEligibleTime=2026-04-13T11:58:39 PreemptTime=None SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-13T11:58:39 Scheduler=Backfill Partition=onlycpus-opp AllocNode:Sid=cld-ter-ui-01:3035857 ReqNodeList=(null) ExcNodeList=(null) NodeList=btc-dfa-gpu-02 BatchHost=btc-dfa-gpu-02 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=100G,node=1,billing=26 AllocTRES=cpu=1,mem=100G,node=1,billing=26 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null) Command=/shared/home/sgaravat/myscript.sh SubmitLine=sbatch myscript.sh WorkDir=/shared/home/sgaravat StdErr=/shared/home/sgaravat/JOB-myscript.sh.283534.4294967294.err StdIn=/dev/null StdOut=/shared/home/sgaravat/JOB-myscript.sh.283534.4294967294.out MailUser=massimo.sgaravatto(a)pd.infn.it MailType=INVALID_DEPEND,BEGIN,END,FAIL,REQUEUE,STAGE_OUT [**] sgaravat@cld-ter-ui-01 ~]$ scontrol show job=283469 JobId=283469 JobName=vllm-pod UserId=ciangott(6054) GroupId=tbuser(6000) MCS_label=N/A Priority=499703 Nice=0 Account=cms QOS=normal JobState=PENDING Reason=Resources Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A SubmitTime=2026-04-13T06:48:37 EligibleTime=2026-04-13T06:48:37 AccrueTime=2026-04-13T06:48:37 StartTime=2026-04-13T14:31:40 EndTime=2026-04-14T14:31:40 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-13T11:59:48 Scheduler=Main Partition=gpus AllocNode:Sid=cld-ter-ui-01:3015801 ReqNodeList=(null) ExcNodeList=(null) NodeList= SchedNodeList=cld-ter-gpu-01 NumNodes=1-1 NumCPUs=32 NumTasks=1 CPUs/Task=32 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=32,mem=190734M,node=1,billing=118,gres/gpu=2,gres/gpu:nvidia-h100=2 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=32 MinMemoryNode=190734M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null) Command=.interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.slurm SubmitLine=sbatch .interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.slurm WorkDir=/shared/home/ciangott StdErr= StdIn=/dev/null StdOut=/shared/home/ciangott/.interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.out TresPerNode=gres/gpu:nvidia-h100:2 TresPerTask=cpu=32

5 10

pam_slurm_adopt
by Ratnasamy, Fritz 10 Apr '26

10 Apr '26

Hi, We have blocked ssh to nodes where users do not have a running job. However, when there is a running job, a user can ssh to that node and use more than the slurm resources allocated on that node. Am I missing something? I usually request an interactive session of 2GB. I then open another terminal and ssh to the node where the interactive session runs. I allocated a 4GB R variable on the node where I just ssh but this does not error out. Additionally, if I end my interactive session, the ssh connection still lives which I believe should not happen. Best, *Fritz Ratnasamy*Data Scientist Information Technology

7 6

Moving a Slurm database from one slurm database host to another
by James A Allsopp 09 Apr '26

09 Apr '26

Hi, We'd like to move our Slurm database from its existing machine running mariadb-server-5.5.68 to another running mariadb-server-10.5.29-3, both of which are running slurm 24.11.7. We dumped the database successfully from the old server, reimported it into the new server using mysqldump and *mysql < db_dump.sql*. The new slurmdbd service was then started. The clusters have different names so there were a few inconsistencies, but when we started the new slurm cluster, we could see the jobs according to sacct (using the -M option) matched those of the original cluster, but none of the associated job steps were shown. I manually logged into the new database and could see the jobs in the job_table but it looked like the foreign key link had been broken in the step table as there were entries in that table. Am I missing some information about how to export and import properly? I was wondering if the import had triggered auto-increment on keys and that was breaking things? Ideally, we'd like to change the name of the cluster as well. Is that just a matter of altering the prefix on the relevant tables, which is something I can do by hand on the database dump before reimporting? For instance, changing the table <old_name>_event_table to <new_cluster_name>_event_table. Thanks James -- Dr James Allsopp | Research Platforms team IT Services | University of Sheffield I do not work Tuesday and Wednesday 3-5pm. I also send emails at odd times, but respond at your convenience.

1 0