January 2026 - slurm-users - lists.schedmd.com

Build 22.05.8 for Debian Trixie?
by Steffen Grunewald 15 Jan '26

15 Jan '26

Good afternoon, we're still running our HPC cluster on Debian 12 Bookworm, which comes with Slurm 22.05.8 - no issues so far. Yesterday, another machine was added as a submit node (slurm-client only), with Debian 13 Trixie / Slurm 24.11.5. Now on that the commands (sinfo etc.) don't work and the slurmctld logs "Incompatible versions of client and server code". Since upgrading the whole cluster is not an option yet: Has someone attempted (and possibly succeeded) to build 22.05.8 (or 23.11.4? which IIRC would be the last compatible one) for Trixie, and would be willing to share ideas? Thanks, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

4 6

Socket memory binding (NUMA). cpu-bind possible with srun, not sbatch
by Rémy Dernat 14 Jan '26

14 Jan '26

Hi, I am looking at that useful documentation https://slurm.schedmd.com/cpu_management.html We have people complaining about memory performance issues while running highly distributed jobs in a shared HPC cluster environment. After looking into it, we saw that is either because of concurrent access to the memory with other users, or because we have jump over memory from a socket to another (in a Numa architecture). Until now, our "easy" answer was to use "exclusive" mode in sbatch, to ensure avoiding any concurrent memory access, and using numactl to check if the job is bound to the other socket. However, I am looking to a better solution, as exclusive mode took all of a big node (we have only big nodes...), and numactl just check if we will have poor performances or not. I checked the slurm.conf parameters, but all I saw is about modifying selectType. However, I cannot use other selectType as we have GPU, so we are using cons_tres. Moreover, we already have a very complicated Slurm configuration, and I would like to avoid any side effect. I am thinking about talking to our user about the `--cpu-bind` (https://slurm.schedmd.com/srun.html#OPT_cpu-bind) method, but I am not sure how to use it, and it seems to be limited to srun ... ? From a job point of view, I know there are some programs to deal with that, like likwid or placement (but not entirely ?), but that does not seem easy to use, and a slurm would be more suitable and more generic. Do you have any idea on how to deal with these issues ? Thanks, Best regards, Rémy Dernat

1 0

slurmctld hourly: Unexpected missing socket error
by Jason Ellul 13 Jan '26

13 Jan '26

Hi all, I am hoping someone can help with our problem. Every hour after restarting slurmctld the controller becomes unresponsive to commands for 1 sec, reporting errors such as: [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934767]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934760]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934875]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934906]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[939016]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error It occurs consistently at around the hour mark, but generally not at other times, unless we run a reconfigure or restart the controller. We don’t see any issues in the slurmdbd.log and the errors are also always msg type RESPONSE. We have tried building a new server on different infrastructure, but the problem has persisted. Yesterday we even tried updating slurm to v24.05.1 in the hope that may provide a fix. During our troubleshooting we have: Set: * SchedulerParameters = max_rpc_cnt=400,sched_min_interval=50000,sched_max_job_start=300,batch_sched_delay=20,bf_resolution=600,bf_min_prio_reserve=2000,bf_min_age_reserve=600 * SlurmctldPort = 6808-6817 But although the stats in sdiag have improved we still see the errors. On our monitoring software we also see a drop in network and disk activity during this 1 second, always at approx. 1 hour after restarting the controller. Many Thanks in advance Jason Jason Ellul Head - Research Computing Facility Office of Cancer Research Peter MacCallum Cancer Centre

6 8

'sacct -a' missing running jobs 'held' by user
by Lee 08 Jan '26

08 Jan '26

Hello, I am using slurm 23.02.6. I have a strange issue. I periodically use sacct to dump job data. I then generate reports based on the resource allocation of our users. Recently, I noticed some 'missing' jobs from my query. The missing jobs came from a user who had a large array job, who then 'held' all of the array jobs. This included 'holding' the Running array jobs. Now, if I run `sacct -a -S YYYY-MM-DD --format="jobidraw,jobname"`, the job will be missing from that query. However, if I query specifically for that job, i.e. `sacct -j RAWJOBID -S YYYY-MM-DD --format="jobidraw,jobname", the job is present. *Question* : 1. How can I include the 'held' running job when I do my bulk query with `sacct -a`? Finding these outliers and adding them ad-hoc to my dumped file is too laborious and isn't feasible. *Minimum working example *: #. Submit a job : myuser@clusterb01:~$ srun --pty bash # landed on dgx29 #. Hold job myuser@clusterb01:~$ scontrol hold 120918 myuser@clusterb01:~$ scontrol show job=120918 JobId=120918 JobName=bash UserId=myuser(123456) GroupId=rdusers(7000) MCS_label=N/A Priority=0 Nice=0 Account=allusers QOS=normal JobState=*RUNNING* Reason=*JobHeldUser* Dependency=(null) Requeue=0 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:00:29 TimeLimit=7-00:00:00 TimeMin=N/A SubmitTime=2025-12-15T13:31:28 EligibleTime=Unknown AccrueTime=Unknown StartTime=2025-12-15T13:31:28 EndTime=2025-12-22T13:31:28 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-12-15T13:31:28 Scheduler=Main Partition=defq AllocNode:Sid=clusterb01:4145861 ReqNodeList=(null) ExcNodeList=(null) NodeList=dgx29 BatchHost=dgx29 NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=9070M,node=1,billing=1 AllocTRES=cpu=2,mem=18140M,node=1,billing=2 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=9070M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=bash WorkDir=/home/myuser Power= #. Release job myuser@clusterb01:~$ scontrol release 120918 #. Show job again myuser@clusterb01:~$ scontrol show job=120918 JobId=120918 JobName=bash UserId=myuser(123456) GroupId=rdusers(7000) MCS_label=N/A Priority=1741 Nice=0 Account=allusers QOS=normal JobState=*RUNNING* Reason=*None* Dependency=(null) Requeue=0 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:01:39 TimeLimit=7-00:00:00 TimeMin=N/A SubmitTime=2025-12-15T13:31:28 EligibleTime=Unknown AccrueTime=Unknown StartTime=2025-12-15T13:31:28 EndTime=2025-12-22T13:31:28 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-12-15T13:31:28 Scheduler=Main Partition=defq AllocNode:Sid=clusterb01:4145861 ReqNodeList=(null) ExcNodeList=(null) NodeList=dgx29 BatchHost=dgx29 NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=9070M,node=1,billing=1 AllocTRES=cpu=2,mem=18140M,node=1,billing=2 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=9070M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=bash WorkDir=/home/myuser/ Power= #. In slurmctld, I see : root@clusterb01:~# grep 120918 /var/log/slurmctld [2025-12-15T13:31:28.706] sched: _slurm_rpc_allocate_resources JobId=120918 NodeList=dgx29 usec=1269 [2025-12-15T13:31:47.751] sched: _hold_job_rec: hold on JobId=120918 by uid 123456 [2025-12-15T13:31:47.751] sched: _update_job: set priority to 0 for JobId=120918 [2025-12-15T13:31:47.751] _slurm_rpc_update_job: complete JobId=120918 uid=123456 usec=189 [2025-12-15T13:32:48.081] sched: _release_job_rec: release hold on JobId=120918 by uid 123456 [2025-12-15T13:32:48.081] _slurm_rpc_update_job: complete JobId=120918 uid=123456 usec=268 [2025-12-15T13:33:20.552] _job_complete: JobId=120918 WEXITSTATUS 0 [2025-12-15T13:33:20.552] _job_complete: JobId=120918 done #. Job is NOT missing, when identifying it by jobid myuser@clusterb01:~$ sacct -j 120918 --starttime=2025-12-12 -o "jobidraw,jobid,node,start,end,elapsed,state,submitline%30" JobIDRaw JobID NodeList Start End Elapsed State SubmitLine ------------ ------------ --------------- ------------------- ------------------- ---------- ---------- ------------------------------ 120918 120918 dgx29 2025-12-15T13:31:28 2025-12-15T13:33:20 00:01:52 COMPLETED srun --pty bash 120918.0 120918.0 dgx29 2025-12-15T13:31:28 2025-12-15T13:33:20 00:01:52 COMPLETED srun --pty bash #. Job IS *missing* when using -a myuser@clusterb01:~$ sacct -a --starttime=2025-12-12 -o "jobidraw,jobid,node,start,end,elapsed,state,submitline%30" | grep -i 120918 ## *MISSING* Best regards, Lee

3 4

New, faster, simpler makefiles.
by Alex Tingle 08 Jan '26

08 Jan '26

Hello, I've written a set of replacement makefiles for Slurm, which are faster, and simpler than Automake. Faster incremental builds make the code much more pleasant to work with. Try it out here: https://github.com/alex-smartptr/slurm Instructions: 1. Run `./configure` as usual (all settings work as before). 2. `make -s` You can find a full README here: https://github.com/alex-smartptr/slurm/blob/newmake/mk/README.md I'm looking for feedback, especially problems that I've not foreseen, but also suggestions for improvements. -Alex Tingle --

1 0

Announcing slurm-quota: CPU and GPU time quotas for Slurm
by Rémi Palancher 08 Jan '26

08 Jan '26

Hello Slurm users, I would like to announce slurm-quota, an open-source tool providing time-based CPU and GPU quota accounting and enforcement for Slurm users and accounts. Slurm does not currently provide native enforcement of consumption-based quotas (e.g. CPU or GPU minutes). slurm-quota addresses this by integrating with Slurm job submission and job completion mechanisms. Key features include: - Definition and enforcement of CPU and GPU time quotas at user and account levels - Support for heterogeneous GPU clusters using billing factors - Clear command-line reporting of usage, remaining quota, with visual progress bars - Lightweight design with minimal dependencies (Python, SQLite and few common Lua libraries) slurm-quota is released under the MIT license and is available on GitHub, along with documentation describing architecture, deployment, and integration details. GitHub Project: https://github.com/rackslab/slurm-quota Comments, questions, and feedback are very welcome! -- Rémi Palancher Rackslab: Open Source Solutions for HPC Operations https://rackslab.io

2 2