July 2025 - slurm-users - lists.schedmd.com

slurmctld hourly: Unexpected missing socket error
by Jason Ellul 13 Jan '26

13 Jan '26

Hi all, I am hoping someone can help with our problem. Every hour after restarting slurmctld the controller becomes unresponsive to commands for 1 sec, reporting errors such as: [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934767]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934760]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934875]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934906]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[939016]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error It occurs consistently at around the hour mark, but generally not at other times, unless we run a reconfigure or restart the controller. We don’t see any issues in the slurmdbd.log and the errors are also always msg type RESPONSE. We have tried building a new server on different infrastructure, but the problem has persisted. Yesterday we even tried updating slurm to v24.05.1 in the hope that may provide a fix. During our troubleshooting we have: Set: * SchedulerParameters = max_rpc_cnt=400,sched_min_interval=50000,sched_max_job_start=300,batch_sched_delay=20,bf_resolution=600,bf_min_prio_reserve=2000,bf_min_age_reserve=600 * SlurmctldPort = 6808-6817 But although the stats in sdiag have improved we still see the errors. On our monitoring software we also see a drop in network and disk activity during this 1 second, always at approx. 1 hour after restarting the controller. Many Thanks in advance Jason Jason Ellul Head - Research Computing Facility Office of Cancer Research Peter MacCallum Cancer Centre

6 8

Assistance with Node Restrictions and Priority for Users in Floating Partition
by Manisha Yadav 28 Aug '25

28 Aug '25

Dear Team, I have a scenario where I need to provide priority access to multiple users from different projects for only 3 nodes. This means that, at any given time, only 3 nodes can be used in that partition, and if one user is utilizing all 3 nodes, no other user should be able to submit jobs to that partition, or their jobs should remain in the queue. To achieve this, I attempted to use QoS by creating a floating partition with some of the nodes and configuring a QoS with priority. I also set a limit with GrpTRES=gres/gpu=24, given that each node has 8 GPUs, and there are 3 nodes in total. I then attached the QoS to the partition and assigned it to the users who need access. I Also tried MaxTRES=gres/gpu=24 While this setup works as expected in the testing environment for CPUs, it is not functioning as intended in production, and it is not effectively restricting node usage in the partition. Could anyone provide suggestions or guidance on how to properly implement node restrictions along with priority? Thank you for your assistance. Best regards, Manisha Yadav ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

4 8

sinfo history
by Steve Kirk 11 Aug '25

11 Aug '25

Hi, Am I correct in thinking that the history of a *node* as shown by sinfo isn't stored anywhere by Slurm? Interested to know if slurm can tell me historically when a node was draining,drained etc. Regards, Steve

5 6

Building Debian packages - problem
by John Hearns 05 Aug '25

05 Aug '25

I am trying to build Slurm version 24.11.6 on an Ubuntu 22.04 system. I download and unpack the source mk-build-deps -i debian/control (not run as root) debuild -b -uc -us I get this error repeated many times: dpkg-shlibdeps: warning: can't extract name and version from library name 'libslurmfull.so' Has anyone seen similar issues? For information, there is an existing SLurm installation on this syste, with version 21 dpkgs from the Ubuntu repository.

4 4

X11 forwarding broken in 25.05.1
by Patryk Bełzak 24 Jul '25

24 Jul '25

Hi, we've recentry upgraded our slurm from 24.11.3 to 25.05.1 and it seems that since the upgrade the ssh X11 forwaring is broken. Quick recap - * on Monday 14'th I performed slurdbd and slurmctld upgrades - X forwarding was still working * on Tuesday 15'th I performed slurmd upgrades - X forwarding stopped working The issue is very hard to determine and it looks like it sits somhere in slurm code. You can submit a job with --x11 and it starts corretly. Xauthority is created, you have all the magic cookies needed, but when you try to start any application, you get error related to permissions I guess, see for yourself: ``` me@sand ~ ssh -X -Y ui [wcss] me@ui.wcss.pl:~ > srun -p lem-cpu-short -A kdm-staff --gres=storage:local:50G -c 12 --mem 12G -t 1:0:0 --x11 --pty /bin/bash [wcss] me@r17ch05b01 ~ > xauth list r17ch05b01.lem.kdm.wcss.pl/unix:91 MIT-MAGIC-COOKIE-1 d82a2efd [wcss] me@r17ch05b01 ~ > xterm xterm: Xt error: Can't open display: localhost:91.0 [wcss] me@r17ch05b01 ~ > date && telnet -4 localhost 6091 || date Wed Jul 23 12:02:39 CEST 2025 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Connection closed by foreign host. Wed Jul 23 12:02:41 CEST 2025 ``` As you can see the connection to port is being dropped/killed after a second or two. Now, it doesn't really matter which flags for ssh you pick (-X or -Y or both). X forwarding is working when you log in as a regular user outside of slurm job. Also if I do ssh localhost inside a job, then I can perform connection to port assigned to $DISPLAY and it isn't dropped - but it doesn't work since $DISPLAY and cookies are being messed up when you perform triple jump and one within same host. Our worker nodes are mostly on el9.5 AlmaLinux. Some are on el8.10 - and there acutally you can do some X forwarding but you must use both -X and -Y (which wasn't the case before slurm upgrade). TLS is disabled in slurm.conf. I am 100% sure that both SSHD and Xorg are properly configured. Has anyone encountered similiar issue? Or any comment from slurm dev team? Best regards Patryk -- Wroclaw Centre for Networking and Supercomputing

3 2

New SchedMD Bike Kits Are Here - Preorder Now Open!
by Victoria Hobson 23 Jul '25

23 Jul '25

We’re excited to announce that SchedMD-branded bike kits are officially available for preorder! Each kit features both the SchedMD and Slurm logos—designed for comfort, performance, and pride.Pick between thoughtfully styled designs for mountain and road biking in each of the following categories: • Jerseys (zip-up or v-neck short sleeve) • Bibs (gravel bib with side and back pockets or switch bib) • Gloves (tech or summer) • Cycling socks with the SchedMD or Slurm logo (the right answer here is to just order both!) Preorders are open now through July 30, 2025. After the store closes, kits will be handcrafted and shipped directly to your door. Please note: delivery will take 8–10 weeks after preorder ends. Want 20% off your order? Reach out to Jess directly (jess(a)schedmd.com) to get an exclusive promo code! Ready to ride in style? Check out the SchedMD bike kits here: https://store.dnacycling.com/schedmd — Thanks for being part of the SchedMD community!

1 0

strange GPU "no resources selected" error
by Paul Raines 23 Jul '25

23 Jul '25

Recently (past 3 days) on our NVIDIA DGX A100 systems running Ubuntu 22.04.5 and slurm 24.11.5 we have had jobs that ask for a gpu, get started by Slurm, but fail to be given a GPU and then fail. In the slurmctld log we see a line like: [2025-07-22T02:46:29.697] error: gres/gpu: job 6919154 node A100-04 no resources selected on the slurmd log I see no errors for the job but there is a line like [2025-07-22T02:46:29.757] [6919154.extern] task/cgroup: _handle_device_access: GRES: job devices.deny: adding c 195:0 rwm(/dev/nvidia0) for all 8 of the GPUs on the node. Other jobs still seem to start up and get a GPU fine. If you look at the job stats one sees: ReqTRES : billing=7,cpu=1,gres/gpu=1,mem=96G,node=1 AllocTRES : billing=3,cpu=1,mem=96G,node=1 showing that even though the gpu was requested, it was not allocated. Occasionly on these boxes (and only these -- my Dell Rocky 8 boxes with GPUS have no problem) we see the nodes go into drain mode with the "res/gpu GRES core specification ... doesn't match socket boundaries." message as per https://support.schedmd.com/show_bug.cgi?id=22498 It does seem to happen after slurmctld restart. I then restart slurmd on the nodes and can resume SLURM on the nodes whenever that happens. Otherwise nothing has changed on these nodes with SLURM config or the OS in over a month. Definition of the nodes are NodeName=A100-[01-04] \ CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 \ ThreadsPerCore=1 RealMemory=1031000 MemSpecLimit=2048 \ TmpDisk=1400000 Feature=amd,epyc,a100 \ Gres=gpu:a100-sxm4-40gb:8 and gres.conf on the nodes is simply AutoDetect=nvml --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

slurm 25.05.0 : allocates less GPUs than requested by --gres=gpu:count
by Jan Gmys 23 Jul '25

23 Jul '25

Since upgrading slurm to 25.05.0 (22.05.9 -> 23.11.11 -> 25.05.0) some jobs requesting --gres=gpu:reqcount GPUs are allocated less than reqcount GPUs if some of the node's GPUs are already in use by other jobs. We have a node - let's call it ares-c02-06 - with 2 GPUs. Consider the following test script: > #!/bin/bash > #SBATCH --nodes=1 > #SBATCH --ntasks-per-node=1 > #SBATCH --time=1-00:00:00 > > echo "CUDA_VISIBLE_DEVICES: " $CUDA_VISIBLE_DEVICES > echo "SLURM_JOB_GPUS: " $SLURM_JOB_GPUS > echo "SLURM_GPUS_ON_NODE: " $SLURM_GPUS_ON_NODE > sleep 10d Submit a job to the node: > > sbatch *--gres=gpu:1* --nodelist=ares-c02-06 job.sh > > Submitted batch job 1950559 > The job starts. Now submit the script again, asking for 2 GPUs > sbatch*--gres=gpu:2* --nodelist=ares-c02-06 job.sh > > Submitted batch job 1950567 This second job should not start as the ressources are not available. Surprisingly, _both jobs are running_ > $ squeue -w ares-c02-06 > JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) > 1950567 gpuai job.sh jan.gmys R 1:55 1 ares-c02-06 > 1950559 gpuai job.sh jan.gmys R 2:58 1 ares-c02-06 In the second job - 1950567 - AllocTRES shows gres/gpu=1, instead of the requested gres/gpu=2. > # sacct -j 1950567,1950559 -X -o jobid%10,reqtres%45,alloctres%60 > JobID ReqTRES AllocTRES > ---------- --------------------------------------------- ------------------------------------------------------------ > 1950559 billing=1,cpu=1,gres/gpu=1,mem=4G,node=1 billing=1,cpu=1,gres/gpu:l40s=1,gres/gpu=1,mem=4G,node=1 > 1950567 billing=1,cpu=1,*gres/gpu=2*,mem=4G,node=1 billing=1,cpu=1,gres/gpu:l40s=1,*gres/gpu=1*,mem=4G,node=1 The output of both jobs > $ cat slurm-1950559.out > CUDA_VISIBLE_DEVICES: 0 > SLURM_JOB_GPUS: 0 > SLURM_GPUS_ON_NODE: 1 > > $ cat slurm-1950567.out > CUDA_VISIBLE_DEVICES: 0 > SLURM_JOB_GPUS: 1 > SLURM_GPUS_ON_NODE: 1 CUDA_VISIBLE_DEVICES is set to 0 for both jobs. SLURM_JOB_GPUS is 0 resp. 1. *Environment: * - RHEL 9.1 - slurm 25.05.0 - The GRES configuration seems fine, AutoDetect is off : > # /usr/sbin/slurmd -G --conf-server hpc-slurm.cluster.hpc -v > [2025-07-22T16:44:05.548] GRES: Global*AutoDetect=off*(4) > [2025-07-22T16:44:05.548] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, > minor 0 > [2025-07-22T16:44:05.548] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, > minor 1 > [2025-07-22T16:44:05.548] GRES: gpu device number 0(/dev/nvidia0):c 195:0 rwm > [2025-07-22T16:44:05.548] GRES: gpu device number 1(/dev/nvidia1):c 195:1 rwm > [2025-07-22T16:44:05.548] Gres Name=gpu Type=L40S Count=2 Index=0 ID=7696487 > File=/dev/nvidia[0-1] Links=(null) > Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT and 'scontrol show node' > NodeName=ares-c02-06 Arch=x86_64 CoresPerSocket=24 > CPUAlloc=0 CPUEfctv=48 CPUTot=48 CPULoad=0.00 > AvailableFeatures=(null) > ActiveFeatures=(null) > Gres=gpu:L40S:2 > NodeAddr=ares-c02-06 NodeHostName=ares-c02-06 Version=25.05.0 > OS=Linux 5.14.0-162.6.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 30 > 07:36:03 EDT 2022 > RealMemory=386000 AllocMem=0 FreeMem=363069 Sockets=2 Boards=1 > State=IDLE+RESERVED+PLANNED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A > MCS_label=N/A > Partitions=gpuai > BootTime=2025-07-04T10:26:27 SlurmdStartTime=2025-07-22T14:48:54 > LastBusyTime=2025-07-22T16:26:36 ResumeAfterTime=None > CfgTRES=cpu=48,mem=386000M,billing=48,gres/gpu=2 > AllocTRES= *Debug notes :* - When using the --gpus option instead of --gres everything works as expected : the second job is PENDING (Ressources) - Tried both, ConstrainDevices=on/off in cgroup.conf, same result - The same is happening on other multi-GPU nodes of the cluster - When the --gres=gpu:2 job is submitted first, i.e. when all GPUs are taken, the second (--gres=gpu:1) job waits correctly. - When both GPUs are free, the --gres=gpu:2 job, correctly gets both GPUs : CUDA_VISIBLE_DEVICES: 0,1 - It worked in slurm 22.05.9 (we recently upgraded in two steps -> 23.11.11 -> 25.05.0) - The only viable workaround I see for the moment is to intercept --gres and --gpus-per-node (don't even have it in job_desc I think!! :-/) options in job_submit.lua and force users to use the --gpus option, which seems to works fine. Anyone experienced similar issues? Any idea how to solve this would be highly appreciated. Jan == -- Jan Gmys Ingénieur de recherche Support HPC/IA pour la plateforme MesoNET Mésocentre de Calcul Scientifique Intensif de l'Université de Lille

1 0

Reserve memory for OS
by John Hearns 17 Jul '25

17 Jul '25

Someone remind me please what is the SLurm node parameter to reserve an amount of memory for the OS? Thanks

3 2

Integrating Fabric Modules with SLURM – Best Practices for HPC Networking?
by xoyeyor178＠coderdir.com 15 Jul '25

15 Jul '25

Our team is exploring ways to optimize our HPC cluster’s network performance, particularly for multi-node SLURM workloads. We’re considering Network Devices Expansion Modules Fabric Modules https://serverorbit.com/network-devices/network-expansion-modules/fabric-mo… to enhance scalability and reduce latency between compute nodes. Has anyone successfully deployed Fabric Modules (e.g., Cisco Nexus, Arista, or Mellanox solutions) in a SLURM environment? Specifically: Interconnect Strategies – Any tips for configuring Fabric Modules to handle SLURM’s bursty traffic patterns? Performance Gains – Measurable improvements in job throughput or MPI communication? Troubleshooting – Known conflicts with SLURM’s network topology detection?

1 0

2026

2025

2024

slurm-users July 2025