- slurm-users - lists.schedmd.com

X11 forwarding broken in 25.05.1
by Patryk Bełzak 23 Jul '25

23 Jul '25

Hi, we've recentry upgraded our slurm from 24.11.3 to 25.05.1 and it seems that since the upgrade the ssh X11 forwaring is broken. Quick recap - * on Monday 14'th I performed slurdbd and slurmctld upgrades - X forwarding was still working * on Tuesday 15'th I performed slurmd upgrades - X forwarding stopped working The issue is very hard to determine and it looks like it sits somhere in slurm code. You can submit a job with --x11 and it starts corretly. Xauthority is created, you have all … [View More]

1 0

slurm 25.05.0 : allocates less GPUs than requested by --gres=gpu:count
by Jan Gmys 23 Jul '25

23 Jul '25

Since upgrading slurm to 25.05.0 (22.05.9 -> 23.11.11 -> 25.05.0) some jobs requesting --gres=gpu:reqcount GPUs are allocated less than reqcount GPUs if some of the node's GPUs are already in use by other jobs. We have a node - let's call it ares-c02-06 - with 2 GPUs. Consider the following test script: > #!/bin/bash > #SBATCH --nodes=1 > #SBATCH --ntasks-per-node=1 > #SBATCH --time=1-00:00:00 > > echo "CUDA_VISIBLE_DEVICES: " $CUDA_VISIBLE_DEVICES > echo "… [View More]SLURM_JOB_GPUS: " $SLURM_JOB_GPUS > echo "SLURM_GPUS_ON_NODE: " $SLURM_GPUS_ON_NODE > sleep 10d Submit a job to the node: > > sbatch *--gres=gpu:1* --nodelist=ares-c02-06 job.sh > > Submitted batch job 1950559 > The job starts. Now submit the script again, asking for 2 GPUs > sbatch*--gres=gpu:2* --nodelist=ares-c02-06 job.sh > > Submitted batch job 1950567 This second job should not start as the ressources are not available. Surprisingly, _both jobs are running_ > $ squeue -w ares-c02-06 > JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) > 1950567 gpuai job.sh jan.gmys R 1:55 1 ares-c02-06 > 1950559 gpuai job.sh jan.gmys R 2:58 1 ares-c02-06 In the second job - 1950567 - AllocTRES shows gres/gpu=1, instead of the requested gres/gpu=2. > # sacct -j 1950567,1950559 -X -o jobid%10,reqtres%45,alloctres%60 > JobID ReqTRES AllocTRES > ---------- --------------------------------------------- ------------------------------------------------------------ > 1950559 billing=1,cpu=1,gres/gpu=1,mem=4G,node=1 billing=1,cpu=1,gres/gpu:l40s=1,gres/gpu=1,mem=4G,node=1 > 1950567 billing=1,cpu=1,*gres/gpu=2*,mem=4G,node=1 billing=1,cpu=1,gres/gpu:l40s=1,*gres/gpu=1*,mem=4G,node=1 The output of both jobs > $ cat slurm-1950559.out > CUDA_VISIBLE_DEVICES: 0 > SLURM_JOB_GPUS: 0 > SLURM_GPUS_ON_NODE: 1 > > $ cat slurm-1950567.out > CUDA_VISIBLE_DEVICES: 0 > SLURM_JOB_GPUS: 1 > SLURM_GPUS_ON_NODE: 1 CUDA_VISIBLE_DEVICES is set to 0 for both jobs. SLURM_JOB_GPUS is 0 resp. 1. *Environment: * - RHEL 9.1 - slurm 25.05.0 - The GRES configuration seems fine, AutoDetect is off : > # /usr/sbin/slurmd -G --conf-server hpc-slurm.cluster.hpc -v > [2025-07-22T16:44:05.548] GRES: Global*AutoDetect=off*(4) > [2025-07-22T16:44:05.548] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, > minor 0 > [2025-07-22T16:44:05.548] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, > minor 1 > [2025-07-22T16:44:05.548] GRES: gpu device number 0(/dev/nvidia0):c 195:0 rwm > [2025-07-22T16:44:05.548] GRES: gpu device number 1(/dev/nvidia1):c 195:1 rwm > [2025-07-22T16:44:05.548] Gres Name=gpu Type=L40S Count=2 Index=0 ID=7696487 > File=/dev/nvidia[0-1] Links=(null) > Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT and 'scontrol show node' > NodeName=ares-c02-06 Arch=x86_64 CoresPerSocket=24 > CPUAlloc=0 CPUEfctv=48 CPUTot=48 CPULoad=0.00 > AvailableFeatures=(null) > ActiveFeatures=(null) > Gres=gpu:L40S:2 > NodeAddr=ares-c02-06 NodeHostName=ares-c02-06 Version=25.05.0 > OS=Linux 5.14.0-162.6.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 30 > 07:36:03 EDT 2022 > RealMemory=386000 AllocMem=0 FreeMem=363069 Sockets=2 Boards=1 > State=IDLE+RESERVED+PLANNED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A > MCS_label=N/A > Partitions=gpuai > BootTime=2025-07-04T10:26:27 SlurmdStartTime=2025-07-22T14:48:54 > LastBusyTime=2025-07-22T16:26:36 ResumeAfterTime=None > CfgTRES=cpu=48,mem=386000M,billing=48,gres/gpu=2 > AllocTRES= *Debug notes :* - When using the --gpus option instead of --gres everything works as expected : the second job is PENDING (Ressources) - Tried both, ConstrainDevices=on/off in cgroup.conf, same result - The same is happening on other multi-GPU nodes of the cluster - When the --gres=gpu:2 job is submitted first, i.e. when all GPUs are taken, the second (--gres=gpu:1) job waits correctly. - When both GPUs are free, the --gres=gpu:2 job, correctly gets both GPUs : CUDA_VISIBLE_DEVICES: 0,1 - It worked in slurm 22.05.9 (we recently upgraded in two steps -> 23.11.11 -> 25.05.0) - The only viable workaround I see for the moment is to intercept --gres and --gpus-per-node (don't even have it in job_desc I think!! :-/) options in job_submit.lua and force users to use the --gpus option, which seems to works fine. Anyone experienced similar issues? Any idea how to solve this would be highly appreciated. Jan == -- Jan Gmys Ingénieur de recherche Support HPC/IA pour la plateforme MesoNET Mésocentre de Calcul Scientifique Intensif de l'Université de Lille [View Less]

1 0

Building Debian packages - problem
by John Hearns 23 Jul '25

23 Jul '25

I am trying to build Slurm version 24.11.6 on an Ubuntu 22.04 system. I download and unpack the source mk-build-deps -i debian/control (not run as root) debuild -b -uc -us I get this error repeated many times: dpkg-shlibdeps: warning: can't extract name and version from library name 'libslurmfull.so' Has anyone seen similar issues? For information, there is an existing SLurm installation on this syste, with version 21 dpkgs from the Ubuntu repository.

3 3

Reserve memory for OS
by John Hearns 17 Jul '25

17 Jul '25

Someone remind me please what is the SLurm node parameter to reserve an amount of memory for the OS? Thanks

3 2

Integrating Fabric Modules with SLURM – Best Practices for HPC Networking?
by xoyeyor178＠coderdir.com 15 Jul '25

15 Jul '25

Our team is exploring ways to optimize our HPC cluster’s network performance, particularly for multi-node SLURM workloads. We’re considering Network Devices Expansion Modules Fabric Modules https://serverorbit.com/network-devices/network-expansion-modules/fabric-mo… to enhance scalability and reduce latency between compute nodes. Has anyone successfully deployed Fabric Modules (e.g., Cisco Nexus, Arista, or Mellanox solutions) in a SLURM environment? Specifically: Interconnect Strategies – … [View More]

1 0

Slurm-wlm depends on X11?
by Jesse Hayward 10 Jul '25

10 Jul '25

Hello group, hopefully a quick one: I am trying to install the suite of standard slurm packages (slurm, munge, slurmdbd, slurm-wlm-basic-plugins) on a compute node. Ubuntu 22.04.05 LTS just upgraded. *However, these packages now bring down x11 (libx11-6 and all of that other garbage in Ubuntu land). * Does slurm actually depend on x11 now? Or do I need to take a look at my apt config and see what's going on over there, instead. Thanks! Jess -- Jesse Hayward Systems Administrator for High … [View More]

3 6

Is DefCpuPerGPU affected by --ntasks-per-gpu?
by Milad Alizadeh 10 Jul '25

10 Jul '25

I have defined DefCpuPerGPU for a partition but when a job specifies --ntasks-per-gpu then DefCpuPerGPU is ignored and I only get 1 cpu: ``` #SBATCH --gpus=1 #SBATCH --ntasks-per-gpu=1 ``` Is this the expected behaviour? The man page for DefCpuPerGPU says: "This value is used only if the job didn't specify --cpus-per-task and --cpus-per-gpu" which I do not specify, at least not directly. If yes is there a way to set something like DefCpuPerTask?

1 0

Setting memory by assigned node with a plugin
by laddaoui＠telecom-paris.fr 09 Jul '25

09 Jul '25

Hello everyone, I'm writing a job_submit/lua plugin to set the memory allocated to a job depending on which node is assigned by Slurm. However, it appears that the node where the job will run is not available at this step of the submission process. Would a SPANK plugin be able to access this information and modify the memory allocation accordingly? Or is there another approach you would recommend for dynamically setting memory based on the assigned node? Any insights would be greatly … [View More]

1 0

Slurm versions 25.05.1 and 24.11.6 are now available
by Marshall Garey 08 Jul '25

08 Jul '25

We are pleased to announce the availability of Slurm versions 25.05.1 and 24.11.6. Changes in 25.05 include the following: * Fix many issues with the TLS Certificate Manager introduced in 25.05 * Optimize account deletion * Fix a bug when reordering the association hierarchy * Fix some issues that cause daemon crashes * Fix a variety of memory leaks Changes in 24.11 include the following: * Fix some issues that cause daemons to crash * Fix some race conditions on shutdown that cause daemons … [View More]

1 0

Energy profile data not recorded in InfluxDB or HDF5
by Xand Meaden 02 Jul '25

02 Jul '25

Hi, Environment: Slurm 23.11.5, Ubuntu Linux 22.04 I'm attempting to setup accounting of job energy data, collected from IPMI (with acct_gather_energy/ipmi) and then written to InfluxDB (with acct_gather_profile/influxdb), using this documentation: https://slurm.schedmd.com/acct_gather.conf.html This is successfully working for non-energy data. However, no Energy data appears in the InfluxDB. If I switch to using HDF5 instead, there is also no Energy data recorded there. Here's the config … [View More]

1 0

2025

2024

slurm-users ----- 2025 ----- July 2025 June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users