- slurm-users - lists.schedmd.com

SLUG 25?
by Dave Lloyd 11 Aug '25

11 Aug '25

Are there plans for a SLUG this year? I didn’t see any announcements on SchedMD’s website nor anything in the list. Thanks, -- Dave Lloyd This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

1 0

sinfo history
by Steve Kirk 11 Aug '25

11 Aug '25

Hi, Am I correct in thinking that the history of a *node* as shown by sinfo isn't stored anywhere by Slurm? Interested to know if slurm can tell me historically when a node was draining,drained etc. Regards, Steve

5 6

Slurm version 25.05.2 is now available
by Marshall Garey 07 Aug '25

07 Aug '25

We are pleased to announce the availability of Slurm version 25.05.2. This version fixes a few regressions with x11 forwarding in 25.05 that may prevent applications from launching, adds support for PMIx v6.x, fixes a variety of stability issues, fixes a regression where --tres-per-task was ignored, and fixes additional minor to moderate severity issues. The full list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Building Debian packages - problem
by John Hearns 05 Aug '25

05 Aug '25

I am trying to build Slurm version 24.11.6 on an Ubuntu 22.04 system. I download and unpack the source mk-build-deps -i debian/control (not run as root) debuild -b -uc -us I get this error repeated many times: dpkg-shlibdeps: warning: can't extract name and version from library name 'libslurmfull.so' Has anyone seen similar issues? For information, there is an existing SLurm installation on this syste, with version 21 dpkgs from the Ubuntu repository.

4 4

slurmctld failed to start
by Dhumal, Dr. Nilesh 01 Aug '25

01 Aug '25

Hello, We recently installed slurm-25 on Redhat linux. We failed to start the slurmctld service. sudo systemctl start slurmctld Job for slurmctld.service failed because the control process exited with error code. See "systemctl status slurmctld.service" and "journalctl -xeu slurmctld.service" for details. sudo systemctl status slurmctld × slurmctld.service - Slurm controller daemon Loaded: loaded (/usr/local/lib/systemd/system/slurmctld.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Thu 2025-07-31 22:23:18 EDT; 1min 3s ago Process: 44317 ExecStart=/usr/local/sbin/slurmctld --systemd $SLURMCTLD_OPTIONS (code=exited, status=1/FAILURE) Main PID: 44317 (code=exited, status=1/FAILURE) CPU: 35ms Jul 31 22:22:29 fgcu-compute01 systemd[1]: Starting Slurm controller daemon... Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: If munged is up, restart with --num-threads=10 Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: Munge encode failed: Failed to connect to "/run/munge/mung> Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: Failed to create MUNGE Credential Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: Couldn't load specified plugin name for auth/munge: Plugin> Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: cannot create auth context for auth/munge Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] fatal: failed to initialize auth plugin Jul 31 22:23:18 fgcu-compute01 systemd[1]: slurmctld.service: Main process exited, code=exited, status=1/FAILURE Jul 31 22:23:18 fgcu-compute01 systemd[1]: slurmctld.service: Failed with result 'exit-code'. Jul 31 22:23:18 fgcu-compute01 systemd[1]: Failed to start Slurm controller daemon. Here is munge service status. munge.service - MUNGE authentication service Loaded: loaded (/usr/local/lib/systemd/system/munge.service; enabled; preset: disabled) Active: active (running) since Thu 2025-07-31 22:06:14 EDT; 19min ago Docs: man:munged(8) Main PID: 44039 (munged) Tasks: 4 (limit: 606218) Memory: 1.4M CPU: 18ms CGroup: /system.slice/munge.service └─44039 /usr/local/sbin/munged Jul 31 22:06:14 fgcu-compute01 systemd[1]: Starting MUNGE authentication service... Jul 31 22:06:14 fgcu-compute01 systemd[1]: Started MUNGE authentication service. Any suggestion is apprecieted to resolve this issue. Thanks, Nilesh Dhumal Associate Professor of Chemistry, http://faculty.fgcu.edu/ndhumal/ Coordinator, FGCU Computational Facility, https://www.fgcu.edu/cas/facultyresources/computationalfacility/ SH-430; Department of Chemistry and Physics Florida Gulf Coast University 10501 FGCU Boulevard South Fort Myers, FL 33965-6565 Phone: (239) 745-4394 Email: ndhumal(a)fgcu.edu

2 3

X11 forwarding broken in 25.05.1
by Patryk Bełzak 24 Jul '25

24 Jul '25

Hi, we've recentry upgraded our slurm from 24.11.3 to 25.05.1 and it seems that since the upgrade the ssh X11 forwaring is broken. Quick recap - * on Monday 14'th I performed slurdbd and slurmctld upgrades - X forwarding was still working * on Tuesday 15'th I performed slurmd upgrades - X forwarding stopped working The issue is very hard to determine and it looks like it sits somhere in slurm code. You can submit a job with --x11 and it starts corretly. Xauthority is created, you have all the magic cookies needed, but when you try to start any application, you get error related to permissions I guess, see for yourself: ``` me@sand ~ ssh -X -Y ui [wcss] me@ui.wcss.pl:~ > srun -p lem-cpu-short -A kdm-staff --gres=storage:local:50G -c 12 --mem 12G -t 1:0:0 --x11 --pty /bin/bash [wcss] me@r17ch05b01 ~ > xauth list r17ch05b01.lem.kdm.wcss.pl/unix:91 MIT-MAGIC-COOKIE-1 d82a2efd [wcss] me@r17ch05b01 ~ > xterm xterm: Xt error: Can't open display: localhost:91.0 [wcss] me@r17ch05b01 ~ > date && telnet -4 localhost 6091 || date Wed Jul 23 12:02:39 CEST 2025 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Connection closed by foreign host. Wed Jul 23 12:02:41 CEST 2025 ``` As you can see the connection to port is being dropped/killed after a second or two. Now, it doesn't really matter which flags for ssh you pick (-X or -Y or both). X forwarding is working when you log in as a regular user outside of slurm job. Also if I do ssh localhost inside a job, then I can perform connection to port assigned to $DISPLAY and it isn't dropped - but it doesn't work since $DISPLAY and cookies are being messed up when you perform triple jump and one within same host. Our worker nodes are mostly on el9.5 AlmaLinux. Some are on el8.10 - and there acutally you can do some X forwarding but you must use both -X and -Y (which wasn't the case before slurm upgrade). TLS is disabled in slurm.conf. I am 100% sure that both SSHD and Xorg are properly configured. Has anyone encountered similiar issue? Or any comment from slurm dev team? Best regards Patryk -- Wroclaw Centre for Networking and Supercomputing

3 2

New SchedMD Bike Kits Are Here - Preorder Now Open!
by Victoria Hobson 23 Jul '25

23 Jul '25

We’re excited to announce that SchedMD-branded bike kits are officially available for preorder! Each kit features both the SchedMD and Slurm logos—designed for comfort, performance, and pride.Pick between thoughtfully styled designs for mountain and road biking in each of the following categories: • Jerseys (zip-up or v-neck short sleeve) • Bibs (gravel bib with side and back pockets or switch bib) • Gloves (tech or summer) • Cycling socks with the SchedMD or Slurm logo (the right answer here is to just order both!) Preorders are open now through July 30, 2025. After the store closes, kits will be handcrafted and shipped directly to your door. Please note: delivery will take 8–10 weeks after preorder ends. Want 20% off your order? Reach out to Jess directly (jess(a)schedmd.com) to get an exclusive promo code! Ready to ride in style? Check out the SchedMD bike kits here: https://store.dnacycling.com/schedmd — Thanks for being part of the SchedMD community!

1 0

strange GPU "no resources selected" error
by Paul Raines 23 Jul '25

23 Jul '25

Recently (past 3 days) on our NVIDIA DGX A100 systems running Ubuntu 22.04.5 and slurm 24.11.5 we have had jobs that ask for a gpu, get started by Slurm, but fail to be given a GPU and then fail. In the slurmctld log we see a line like: [2025-07-22T02:46:29.697] error: gres/gpu: job 6919154 node A100-04 no resources selected on the slurmd log I see no errors for the job but there is a line like [2025-07-22T02:46:29.757] [6919154.extern] task/cgroup: _handle_device_access: GRES: job devices.deny: adding c 195:0 rwm(/dev/nvidia0) for all 8 of the GPUs on the node. Other jobs still seem to start up and get a GPU fine. If you look at the job stats one sees: ReqTRES : billing=7,cpu=1,gres/gpu=1,mem=96G,node=1 AllocTRES : billing=3,cpu=1,mem=96G,node=1 showing that even though the gpu was requested, it was not allocated. Occasionly on these boxes (and only these -- my Dell Rocky 8 boxes with GPUS have no problem) we see the nodes go into drain mode with the "res/gpu GRES core specification ... doesn't match socket boundaries." message as per https://support.schedmd.com/show_bug.cgi?id=22498 It does seem to happen after slurmctld restart. I then restart slurmd on the nodes and can resume SLURM on the nodes whenever that happens. Otherwise nothing has changed on these nodes with SLURM config or the OS in over a month. Definition of the nodes are NodeName=A100-[01-04] \ CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 \ ThreadsPerCore=1 RealMemory=1031000 MemSpecLimit=2048 \ TmpDisk=1400000 Feature=amd,epyc,a100 \ Gres=gpu:a100-sxm4-40gb:8 and gres.conf on the nodes is simply AutoDetect=nvml --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

slurm 25.05.0 : allocates less GPUs than requested by --gres=gpu:count
by Jan Gmys 23 Jul '25

23 Jul '25

Since upgrading slurm to 25.05.0 (22.05.9 -> 23.11.11 -> 25.05.0) some jobs requesting --gres=gpu:reqcount GPUs are allocated less than reqcount GPUs if some of the node's GPUs are already in use by other jobs. We have a node - let's call it ares-c02-06 - with 2 GPUs. Consider the following test script: > #!/bin/bash > #SBATCH --nodes=1 > #SBATCH --ntasks-per-node=1 > #SBATCH --time=1-00:00:00 > > echo "CUDA_VISIBLE_DEVICES: " $CUDA_VISIBLE_DEVICES > echo "SLURM_JOB_GPUS: " $SLURM_JOB_GPUS > echo "SLURM_GPUS_ON_NODE: " $SLURM_GPUS_ON_NODE > sleep 10d Submit a job to the node: > > sbatch *--gres=gpu:1* --nodelist=ares-c02-06 job.sh > > Submitted batch job 1950559 > The job starts. Now submit the script again, asking for 2 GPUs > sbatch*--gres=gpu:2* --nodelist=ares-c02-06 job.sh > > Submitted batch job 1950567 This second job should not start as the ressources are not available. Surprisingly, _both jobs are running_ > $ squeue -w ares-c02-06 > JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) > 1950567 gpuai job.sh jan.gmys R 1:55 1 ares-c02-06 > 1950559 gpuai job.sh jan.gmys R 2:58 1 ares-c02-06 In the second job - 1950567 - AllocTRES shows gres/gpu=1, instead of the requested gres/gpu=2. > # sacct -j 1950567,1950559 -X -o jobid%10,reqtres%45,alloctres%60 > JobID ReqTRES AllocTRES > ---------- --------------------------------------------- ------------------------------------------------------------ > 1950559 billing=1,cpu=1,gres/gpu=1,mem=4G,node=1 billing=1,cpu=1,gres/gpu:l40s=1,gres/gpu=1,mem=4G,node=1 > 1950567 billing=1,cpu=1,*gres/gpu=2*,mem=4G,node=1 billing=1,cpu=1,gres/gpu:l40s=1,*gres/gpu=1*,mem=4G,node=1 The output of both jobs > $ cat slurm-1950559.out > CUDA_VISIBLE_DEVICES: 0 > SLURM_JOB_GPUS: 0 > SLURM_GPUS_ON_NODE: 1 > > $ cat slurm-1950567.out > CUDA_VISIBLE_DEVICES: 0 > SLURM_JOB_GPUS: 1 > SLURM_GPUS_ON_NODE: 1 CUDA_VISIBLE_DEVICES is set to 0 for both jobs. SLURM_JOB_GPUS is 0 resp. 1. *Environment: * - RHEL 9.1 - slurm 25.05.0 - The GRES configuration seems fine, AutoDetect is off : > # /usr/sbin/slurmd -G --conf-server hpc-slurm.cluster.hpc -v > [2025-07-22T16:44:05.548] GRES: Global*AutoDetect=off*(4) > [2025-07-22T16:44:05.548] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, > minor 0 > [2025-07-22T16:44:05.548] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, > minor 1 > [2025-07-22T16:44:05.548] GRES: gpu device number 0(/dev/nvidia0):c 195:0 rwm > [2025-07-22T16:44:05.548] GRES: gpu device number 1(/dev/nvidia1):c 195:1 rwm > [2025-07-22T16:44:05.548] Gres Name=gpu Type=L40S Count=2 Index=0 ID=7696487 > File=/dev/nvidia[0-1] Links=(null) > Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT and 'scontrol show node' > NodeName=ares-c02-06 Arch=x86_64 CoresPerSocket=24 > CPUAlloc=0 CPUEfctv=48 CPUTot=48 CPULoad=0.00 > AvailableFeatures=(null) > ActiveFeatures=(null) > Gres=gpu:L40S:2 > NodeAddr=ares-c02-06 NodeHostName=ares-c02-06 Version=25.05.0 > OS=Linux 5.14.0-162.6.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 30 > 07:36:03 EDT 2022 > RealMemory=386000 AllocMem=0 FreeMem=363069 Sockets=2 Boards=1 > State=IDLE+RESERVED+PLANNED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A > MCS_label=N/A > Partitions=gpuai > BootTime=2025-07-04T10:26:27 SlurmdStartTime=2025-07-22T14:48:54 > LastBusyTime=2025-07-22T16:26:36 ResumeAfterTime=None > CfgTRES=cpu=48,mem=386000M,billing=48,gres/gpu=2 > AllocTRES= *Debug notes :* - When using the --gpus option instead of --gres everything works as expected : the second job is PENDING (Ressources) - Tried both, ConstrainDevices=on/off in cgroup.conf, same result - The same is happening on other multi-GPU nodes of the cluster - When the --gres=gpu:2 job is submitted first, i.e. when all GPUs are taken, the second (--gres=gpu:1) job waits correctly. - When both GPUs are free, the --gres=gpu:2 job, correctly gets both GPUs : CUDA_VISIBLE_DEVICES: 0,1 - It worked in slurm 22.05.9 (we recently upgraded in two steps -> 23.11.11 -> 25.05.0) - The only viable workaround I see for the moment is to intercept --gres and --gpus-per-node (don't even have it in job_desc I think!! :-/) options in job_submit.lua and force users to use the --gpus option, which seems to works fine. Anyone experienced similar issues? Any idea how to solve this would be highly appreciated. Jan == -- Jan Gmys Ingénieur de recherche Support HPC/IA pour la plateforme MesoNET Mésocentre de Calcul Scientifique Intensif de l'Université de Lille

1 0

Reserve memory for OS
by John Hearns 17 Jul '25

17 Jul '25

Someone remind me please what is the SLurm node parameter to reserve an amount of memory for the OS? Thanks

3 2

Integrating Fabric Modules with SLURM – Best Practices for HPC Networking?
by xoyeyor178＠coderdir.com 15 Jul '25

15 Jul '25

Our team is exploring ways to optimize our HPC cluster’s network performance, particularly for multi-node SLURM workloads. We’re considering Network Devices Expansion Modules Fabric Modules https://serverorbit.com/network-devices/network-expansion-modules/fabric-mo… to enhance scalability and reduce latency between compute nodes. Has anyone successfully deployed Fabric Modules (e.g., Cisco Nexus, Arista, or Mellanox solutions) in a SLURM environment? Specifically: Interconnect Strategies – Any tips for configuring Fabric Modules to handle SLURM’s bursty traffic patterns? Performance Gains – Measurable improvements in job throughput or MPI communication? Troubleshooting – Known conflicts with SLURM’s network topology detection?

1 0

Slurm-wlm depends on X11?
by Jesse Hayward 10 Jul '25

10 Jul '25

Hello group, hopefully a quick one: I am trying to install the suite of standard slurm packages (slurm, munge, slurmdbd, slurm-wlm-basic-plugins) on a compute node. Ubuntu 22.04.05 LTS just upgraded. *However, these packages now bring down x11 (libx11-6 and all of that other garbage in Ubuntu land). * Does slurm actually depend on x11 now? Or do I need to take a look at my apt config and see what's going on over there, instead. Thanks! Jess -- Jesse Hayward Systems Administrator for High Performance Computing Vassar College 845.437.7521 CIS 207

3 6

Is DefCpuPerGPU affected by --ntasks-per-gpu?
by Milad Alizadeh 10 Jul '25

10 Jul '25

I have defined DefCpuPerGPU for a partition but when a job specifies --ntasks-per-gpu then DefCpuPerGPU is ignored and I only get 1 cpu: ``` #SBATCH --gpus=1 #SBATCH --ntasks-per-gpu=1 ``` Is this the expected behaviour? The man page for DefCpuPerGPU says: "This value is used only if the job didn't specify --cpus-per-task and --cpus-per-gpu" which I do not specify, at least not directly. If yes is there a way to set something like DefCpuPerTask?

1 0

Setting memory by assigned node with a plugin
by laddaoui＠telecom-paris.fr 09 Jul '25

09 Jul '25

Hello everyone, I'm writing a job_submit/lua plugin to set the memory allocated to a job depending on which node is assigned by Slurm. However, it appears that the node where the job will run is not available at this step of the submission process. Would a SPANK plugin be able to access this information and modify the memory allocation accordingly? Or is there another approach you would recommend for dynamically setting memory based on the assigned node? Any insights would be greatly appreciated. Best regards, Nacereddine

1 0

Slurm versions 25.05.1 and 24.11.6 are now available
by Marshall Garey 08 Jul '25

08 Jul '25

We are pleased to announce the availability of Slurm versions 25.05.1 and 24.11.6. Changes in 25.05 include the following: * Fix many issues with the TLS Certificate Manager introduced in 25.05 * Optimize account deletion * Fix a bug when reordering the association hierarchy * Fix some issues that cause daemon crashes * Fix a variety of memory leaks Changes in 24.11 include the following: * Fix some issues that cause daemons to crash * Fix some race conditions on shutdown that cause daemons to crash or hang The full list of changes are available in the CHANGELOG for each version: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md https://github.com/SchedMD/slurm/blob/slurm-24.11/CHANGELOG/slurm-24.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Energy profile data not recorded in InfluxDB or HDF5
by Xand Meaden 02 Jul '25

02 Jul '25

Hi, Environment: Slurm 23.11.5, Ubuntu Linux 22.04 I'm attempting to setup accounting of job energy data, collected from IPMI (with acct_gather_energy/ipmi) and then written to InfluxDB (with acct_gather_profile/influxdb), using this documentation: https://slurm.schedmd.com/acct_gather.conf.html This is successfully working for non-energy data. However, no Energy data appears in the InfluxDB. If I switch to using HDF5 instead, there is also no Energy data recorded there. Here's the config being used: root@erc-hpc-comp06t:/etc/slurm# grep Acct /etc/slurm/slurm.conf JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup AcctGatherEnergyType=acct_gather_energy/ipmi AcctGatherProfileType=acct_gather_profile/influxdb root@erc-hpc-comp06t:/etc/slurm# cat acct_gather.conf # Energy accounting gatherer configuration EnergyIPMIFrequency=60 EnergyIPMICalcAdjustment=yes EnergyIPMIUsername=xx EnergyIPMIPassword=xxxxx # InfluxDB configuration - record Energy and Task data by default ProfileInfluxDBDefault=Energy,Task # Using stunnel to reach InfluxDB over TLS ProfileInfluxDBHost=localhost:8086 ProfileInfluxDBDatabase=slurmpr_hpc_test ProfileInfluxDBUser=slurmpr_hpc_test_rw ProfileInfluxDBPass=xx ProfileInfluxDBRTPolicy=autogen The Task data is recorded just fine. And I can see that the IPMI plugin is working: root@erc-hpc-comp06t:/etc/slurm# scontrol show node erc-hpc-comp06t | grep CurrentWatts CurrentWatts=85 AveWatts=84 Does anyone have this working? I really hope I'm missing something obvious here! Thanks, Xand -- Xand Meaden | Principal Research Infrastructure Engineer e-Research | King's College London

1 0

sbatch strange behavior with different --nodelist (-w) options
by Xinghong He 30 Jun '25

30 Jun '25

Dear Community, I'm seeing strange behavior from sbatch with different --nodelist (-w) options on my two node cluster. Here are my test scripts: *~/slurm$ cat mpirun.slm#!/bin/bash#SBATCH --job-name=mpirun_2x1#SBATCH --nodes=2#SBATCH --ntasks-per-node=1#SBATCH --exclusivesource /usr/mpi/gcc/openmpi-4.1.7a1/bin/mpivars.shmpirun ./a.sh* *~/slurm$ cat a.sh#!/bin/bashecho "`uname -n` OMPI_COMM_WORLD_RANK = $OMPI_COMM_WORLD_RANK SLURM_NODEID = $SLURM_NODEID SLURM_JOB_NODELIST = $SLURM_JOB_NODELIST"* If I do not specify any -w option or if I include both nodes in -w option, I get expected results. *~/slurm$ sbatch mpirun.slmSubmitted batch job 71~/slurm$ cat slurm-71.outstd-199 OMPI_COMM_WORLD_RANK = 0 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]std-271 OMPI_COMM_WORLD_RANK = 1 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]* However, if I specify only one node in -w option, but still want two nodes, I always get one expected result and one unexpected result. The unexpected one will dispatch both MPI tasks to the same node. This one is expected - running two MPI tasks across nodes. *~/slurm$ sbatch -w std-199 mpirun.slmSubmitted batch job 72~/slurm$ cat slurm-72.outstd-199 OMPI_COMM_WORLD_RANK = 0 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]std-271 OMPI_COMM_WORLD_RANK = 1 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]* This one if unexpected - ends up running two MPI tasks on the same node, though SLURM_JOB_NODELIST also gives the correct two nodes. *~/slurm$ sbatch -w std-271 mpirun.slmSubmitted batch job 73ubuntu@bright-anchovy-controller:~/slurm$ cat slurm-73.outstd-199 OMPI_COMM_WORLD_RANK = 0 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]std-199 OMPI_COMM_WORLD_RANK = 1 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]* I first saw the problem on a larger cluster where I needed to specify both -w and -x options to include and exclude nodes. I then narrowed it down to a two node cluster. I tried adding options like --hostfile or --rankfile, or -npernode, all does not change how the tasks are dispatched to nodes. The problem is repeatable. Here are the tested systems: Slurm 23.02.5 on Ubuntu 22.04.5 LTS Slurm 24.05.1 on Ubuntu 22.04.4 LTS How to make the last one work. I.e., requesting *-w std-271* and make it run on two nodes. I'd appreciate any help! Regards, Xinghong

1 0

extern step does not exit
by william＠signalbox.org.uk 27 Jun '25

27 Jun '25

I am building a new small cluster on Rocky Linux 9 with Slurm 24.11.5. Slurm was compiled with the default options except that we added --with pmix We do use some additional complications - we use NFSv4 with AD authentication (so we use auks), and we use SElinux in enforcing mode. However, I do not currently see any evidence that those relate to the problem that we have; though note that the slurm account is an AD account. All relevant AD accounts have the RFC 2307 attributes (uid, gid etc.) configured. I had also configured pam_slurm_adopt but as that was causing issues I backed out the changes that enable it. It was too hard to troubleshoot without being able to login to the test compute node easily. The issue I am having in testing is that whatever job I launch, when it terminates the extern job step does not terminate. These are the more relevant slurm.conf settings: Epilog=/etc/slurm/epilog.sh JobAcctGatherType=jobacct_gather/cgroup JobCompType=jobcomp/none KillOnBadExit=1 KillWait=30 ProctrackType=proctrack/cgroup Prolog=/etc/slurm/prolog.sh PrologFlags=Alloc,Contain SelectTypeParameters=CR_Core_Memory SelectType=select/cons_tres SlurmUser=slurm TaskPlugin=task/affinity,task/cgroup TaskProlog=/etc/slurm/taskprolog.sh The slurmd.log on the compute node for the job shows a sequence like this (with -vv debug enabled): [2025-06-26T14:57:23.272] select/cons_tres: init: select/cons_tres loaded [2025-06-26T14:57:23.272] select/linear: init: Linear node selection plugin loaded with argument 20 [2025-06-26T14:57:23.272] cred/munge: init: Munge credential signature plugin loaded [2025-06-26T14:57:23.272] [34.extern] debug: auth/munge: init: loaded [2025-06-26T14:57:23.273] [34.extern] debug: Reading cgroup.conf file /etc/slurm/cgroup.conf [2025-06-26T14:57:23.279] [34.extern] debug: cgroup/v2: init: Cgroup v2 plugin loaded [2025-06-26T14:57:23.290] [34.extern] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2025-06-26T14:57:23.291] [34.extern] debug: CPUs:40 Boards:1 Sockets:2 CoresPerSocket:20 ThreadsPerCore:1 [2025-06-26T14:57:23.291] [34.extern] debug: task/cgroup: init: core enforcement enabled [2025-06-26T14:57:23.291] [34.extern] debug: task/cgroup: task_cgroup_memory_init: task/cgroup/memory: TotCfgRealMem:191587M allowed:100%(enforced), swap:0%(enforced), max:100%(191587M) max+swap:100%(383174M) min:30M [2025-06-26T14:57:23.291] [34.extern] debug: task/cgroup: init: memory enforcement enabled [2025-06-26T14:57:23.291] [34.extern] debug: task/cgroup: init: device enforcement enabled [2025-06-26T14:57:23.291] [34.extern] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2025-06-26T14:57:23.291] [34.extern] task/affinity: init: task affinity plugin loaded with CPU mask 0xffffffffff [2025-06-26T14:57:23.291] [34.extern] debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded [2025-06-26T14:57:23.291] [34.extern] topology/default: init: topology Default plugin loaded [2025-06-26T14:57:23.292] [34.extern] debug: gpu/generic: init: init: GPU Generic plugin loaded [2025-06-26T14:57:23.293] [34.extern] debug: Setting slurmstepd(2666071) oom_score_adj to -1000 [2025-06-26T14:57:23.293] [34.extern] debug: Message thread started pid = 2666071 [2025-06-26T14:57:23.294] [34.extern] debug: spank: opening plugin stack /etc/slurm/plugstack.conf [2025-06-26T14:57:23.294] [34.extern] debug: /etc/slurm/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf" [2025-06-26T14:57:23.294] [34.extern] debug: spank: opening plugin stack /etc/slurm/plugstack.conf.d/auks.conf [2025-06-26T14:57:23.298] [34.extern] debug: spank: /etc/slurm/plugstack.conf.d/auks.conf:57: Loaded plugin auks.so [2025-06-26T14:57:23.298] [34.extern] debug: SPANK: appending plugin option "auks" [2025-06-26T14:57:23.352] [34.extern] spank-auks: new unique ccache is KCM:71433:49222 <<<< This UID 71433 is the user running the job [2025-06-26T14:57:23.364] [34.extern] spank-auks: user '71433' cred stored in ccache KCM:71433:49222 [2025-06-26T14:57:23.380] [34.extern] debug: task/cgroup: task_cgroup_cpuset_create: job abstract cores are '0' [2025-06-26T14:57:23.381] [34.extern] debug: task/cgroup: task_cgroup_cpuset_create: step abstract cores are '0' [2025-06-26T14:57:23.381] [34.extern] debug: task/cgroup: task_cgroup_cpuset_create: job physical CPUs are '0' [2025-06-26T14:57:23.381] [34.extern] debug: task/cgroup: task_cgroup_cpuset_create: step physical CPUs are '0' [2025-06-26T14:57:23.381] [34.extern] task/cgroup: _memcg_initialize: job: alloc=1024MB mem.limit=1024MB memsw.limit=1024MB job_swappiness=18446744073709551614 [2025-06-26T14:57:23.381] [34.extern] task/cgroup: _memcg_initialize: step: alloc=1024MB mem.limit=1024MB memsw.limit=1024MB job_swappiness=18446744073709551614 [2025-06-26T14:57:23.385] [34.extern] debug: close_slurmd_conn: sending 0: No error [2025-06-26T14:57:24.371] launch task StepId=34.interactive request from UID:71433 GID:70668 HOST:172.17.11.22 PORT:46948 [2025-06-26T14:57:24.372] task/affinity: lllp_distribution: JobId=34 manual binding: mask_cpu,one_thread [2025-06-26T14:57:24.372] debug: Waiting for job 34's prolog to complete [2025-06-26T14:57:24.372] debug: Finished wait for job 34's prolog to complete [2025-06-26T14:57:24.377] select/cons_tres: init: select/cons_tres loaded [2025-06-26T14:57:24.377] select/linear: init: Linear node selection plugin loaded with argument 20 [2025-06-26T14:57:24.377] cred/munge: init: Munge credential signature plugin loaded [2025-06-26T14:57:24.377] [34.interactive] debug: auth/munge: init: loaded [2025-06-26T14:57:24.379] [34.interactive] debug: Reading cgroup.conf file /etc/slurm/cgroup.conf [2025-06-26T14:57:24.384] [34.interactive] debug: cgroup/v2: init: Cgroup v2 plugin loaded [2025-06-26T14:57:24.403] [34.interactive] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2025-06-26T14:57:24.404] [34.interactive] debug: CPUs:40 Boards:1 Sockets:2 CoresPerSocket:20 ThreadsPerCore:1 [2025-06-26T14:57:24.404] [34.interactive] debug: task/cgroup: init: core enforcement enabled [2025-06-26T14:57:24.404] [34.interactive] debug: task/cgroup: task_cgroup_memory_init: task/cgroup/memory: TotCfgRealMem:191587M allowed:100%(enforced), swap:0%(enforced), max:100%(191587M) max+swap:100%(383174M) min:30M [2025-06-26T14:57:24.404] [34.interactive] debug: task/cgroup: init: memory enforcement enabled [2025-06-26T14:57:24.404] [34.interactive] debug: task/cgroup: init: device enforcement enabled [2025-06-26T14:57:24.404] [34.interactive] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2025-06-26T14:57:24.404] [34.interactive] task/affinity: init: task affinity plugin loaded with CPU mask 0xffffffffff [2025-06-26T14:57:24.404] [34.interactive] debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded [2025-06-26T14:57:24.404] [34.interactive] topology/default: init: topology Default plugin loaded [2025-06-26T14:57:24.404] [34.interactive] debug: gpu/generic: init: init: GPU Generic plugin loaded [2025-06-26T14:57:24.406] [34.interactive] debug: close_slurmd_conn: sending 0: No error [2025-06-26T14:57:24.406] [34.interactive] debug: Message thread started pid = 2666089 [2025-06-26T14:57:24.406] [34.interactive] debug: Setting slurmstepd(2666089) oom_score_adj to -1000 [2025-06-26T14:57:24.407] [34.interactive] debug: spank: opening plugin stack /etc/slurm/plugstack.conf [2025-06-26T14:57:24.407] [34.interactive] debug: /etc/slurm/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf" [2025-06-26T14:57:24.407] [34.interactive] debug: spank: opening plugin stack /etc/slurm/plugstack.conf.d/auks.conf [2025-06-26T14:57:24.411] [34.interactive] debug: spank: /etc/slurm/plugstack.conf.d/auks.conf:57: Loaded plugin auks.so [2025-06-26T14:57:24.411] [34.interactive] debug: SPANK: appending plugin option "auks" [2025-06-26T14:57:24.461] [34.interactive] spank-auks: new unique ccache is KCM:71433:44156 [2025-06-26T14:57:24.467] [34.interactive] spank-auks: user '71433' cred stored in ccache KCM:71433:44156 [2025-06-26T14:57:24.480] [34.interactive] debug: task/cgroup: task_cgroup_cpuset_create: job abstract cores are '0' [2025-06-26T14:57:24.480] [34.interactive] debug: task/cgroup: task_cgroup_cpuset_create: step abstract cores are '0' [2025-06-26T14:57:24.480] [34.interactive] debug: task/cgroup: task_cgroup_cpuset_create: job physical CPUs are '0' [2025-06-26T14:57:24.480] [34.interactive] debug: task/cgroup: task_cgroup_cpuset_create: step physical CPUs are '0' [2025-06-26T14:57:24.481] [34.interactive] task/cgroup: _memcg_initialize: job: alloc=1024MB mem.limit=1024MB memsw.limit=1024MB job_swappiness=18446744073709551614 [2025-06-26T14:57:24.481] [34.interactive] task/cgroup: _memcg_initialize: step: alloc=1024MB mem.limit=1024MB memsw.limit=1024MB job_swappiness=18446744073709551614 [2025-06-26T14:57:24.501] [34.interactive] warning: restricted to a subset of cpus [2025-06-26T14:57:24.503] [34.interactive] debug: stdin uses a pty object [2025-06-26T14:57:24.504] [34.interactive] debug: init pty size 48:324 [2025-06-26T14:57:24.506] [34.interactive] debug levels are stderr='error', logfile='debug', syslog='quiet' [2025-06-26T14:57:24.507] [34.interactive] debug: IO handler started pid=2666089 [2025-06-26T14:57:24.517] [34.interactive] spank-auks: credential renewer launched (pid=2666107) [2025-06-26T14:57:24.517] [34.interactive] starting 1 tasks [2025-06-26T14:57:24.517] [34.interactive] task 0 (2666108) started 2025-06-26T14:57:24 [2025-06-26T14:57:24.537] [34.interactive] debug: task/affinity: task_p_pre_launch: affinity StepId=34.interactive, task:0 bind:mask_cpu,one_thread [2025-06-26T14:57:24.537] [34.interactive] debug: [job 34] attempting to run slurm task_prolog [/etc/slurm/taskprolog.sh] [2025-06-26T14:57:24.537] [34.interactive] debug: Sending launch resp rc=0 [2025-06-26T14:57:24.563] [34.interactive] debug: export name:TMPDIR:val:/local/scratch/34: [2025-06-26T14:59:42.726] [34.interactive] task 0 (2666108) exited with exit code 0. <<<< Here the interactive job was closed [2025-06-26T14:59:42.727] [34.interactive] spank-auks: all tasks exited, killing credential renewer (pid=2666107) [2025-06-26T14:59:42.729] [34.interactive] debug: task/affinity: task_p_post_term: affinity StepId=34.interactive, task 0 [2025-06-26T14:59:42.729] [34.interactive] debug: signaling condition [2025-06-26T14:59:42.729] [34.interactive] debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded [2025-06-26T14:59:42.729] [34.interactive] debug: Waiting for IO [2025-06-26T14:59:42.729] [34.interactive] debug: Closing debug channel [2025-06-26T14:59:42.729] [34.interactive] debug: IO handler exited, rc=0 [2025-06-26T14:59:42.729] [34.interactive] debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded [2025-06-26T14:59:42.731] [34.interactive] debug: slurm_recv_timeout at 0 of 4, recv zero bytes [2025-06-26T14:59:42.738] [34.interactive] spank-auks: Destroyed ccache KCM:71433:44156 [2025-06-26T14:59:42.740] debug: _rpc_terminate_job: uid = 71755 JobId=34 <<<< That uid 71755 is for the slurm account which does not run any daemons on a compute node [2025-06-26T14:59:42.740] debug: credential for job 34 revoked [2025-06-26T14:59:42.742] [34.extern] debug: Handling REQUEST_SIGNAL_CONTAINER [2025-06-26T14:59:42.742] [34.extern] debug: _handle_signal_container for StepId=34.extern uid=71755 signal=18 flag=0x0 [2025-06-26T14:59:42.742] [34.extern] Sent signal 18 to StepId=34.extern [2025-06-26T14:59:42.742] [34.interactive] debug: Handling REQUEST_SIGNAL_CONTAINER [2025-06-26T14:59:42.742] [34.interactive] debug: _handle_signal_container for StepId=34.interactive uid=71755 signal=18 flag=0x0 [2025-06-26T14:59:42.742] [34.interactive] Sent signal 18 to StepId=34.interactive [2025-06-26T14:59:42.743] [34.extern] debug: Handling REQUEST_SIGNAL_CONTAINER [2025-06-26T14:59:42.743] [34.extern] debug: _handle_signal_container for StepId=34.extern uid=71755 signal=15 flag=0x0 [2025-06-26T14:59:42.743] [34.extern] Sent signal 15 to StepId=34.extern [2025-06-26T14:59:42.743] [34.interactive] debug: Handling REQUEST_SIGNAL_CONTAINER [2025-06-26T14:59:42.743] [34.interactive] debug: _handle_signal_container for StepId=34.interactive uid=71755 signal=15 flag=0x0 [2025-06-26T14:59:42.743] [34.interactive] Sent signal 15 to StepId=34.interactive [2025-06-26T14:59:42.744] [34.interactive] debug: Handling REQUEST_STATE [2025-06-26T14:59:42.744] [34.interactive] debug: Message thread exited [2025-06-26T14:59:42.744] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:42.763] [34.interactive] done with step [2025-06-26T14:59:42.766] [34.extern] debug: signaling condition [2025-06-26T14:59:42.766] [34.extern] debug: task/affinity: task_p_post_term: affinity StepId=34.extern, task 0 [2025-06-26T14:59:42.766] [34.extern] debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded [2025-06-26T14:59:42.766] [34.extern] debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded [2025-06-26T14:59:42.766] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:42.766] [34.extern] debug: Terminate signal (SIGTERM) received [2025-06-26T14:59:42.768] [34.extern] error: setgroups: Operation not permitted [2025-06-26T14:59:42.768] [34.extern] error: _shutdown_x11_forward: Unable to drop privileges [2025-06-26T14:59:42.771] [34.extern] spank-auks: Destroyed ccache KCM:71433:49222 [2025-06-26T14:59:42.817] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:42.918] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:43.419] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:44.420] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:45.420] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:46.421] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:47.422] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:48.423] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:49.424] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:50.425] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:51.426] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:52.426] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:53.427] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:03.428] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:12.429] [34.extern] debug: Handling REQUEST_STEP_TERMINATE [2025-06-26T15:00:12.429] [34.extern] debug: _handle_terminate for StepId=34.extern uid=0 [2025-06-26T15:00:12.429] [34.extern] Sent SIGKILL signal to StepId=34.extern [2025-06-26T15:00:12.429] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:12.450] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:12.501] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:12.602] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:13.102] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:14.103] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:14.104] [34.extern] debug: Handling REQUEST_STEP_TERMINATE [2025-06-26T15:00:14.104] [34.extern] debug: _handle_terminate for StepId=34.extern uid=0 [2025-06-26T15:00:14.104] [34.extern] Sent SIGKILL signal to StepId=34.extern [2025-06-26T15:00:15.105] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:15.105] [34.extern] debug: Handling REQUEST_STEP_TERMINATE . . We are left with a process "slurmstepd: [34.extern]" running, and a matching socket file for job 34 in /var/spool/slurmd; the job shows state 'CG' (Completing). I can clean up by issuing a kill -9 to the slurmstepd 'extern' job on the compute node. I do not know a way to get any logging output from that job step, but it has a file descriptor open to /var/log/slurm/slurmd.log (which is what we configured) so I assume that it is writing some of the above log. In some runs I also see this just as the main task exits (but from the .0 job step, not from .extern) : [2025-06-27T21:00:47.265] [39.0] error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/slurmstepd.scope/job_39/step_0/user/cgroup.free ze' for writing: Permission denied I can see that between reboots, the directories '/sys/fs/cgroup/system.slice/slurmstepd.scope/job_NN' remain, with the /sys/fs/cgroup/system.slice/slurmstepd.scope/job_NN/step_extern folder. The permissions of the files referred to as 'permission denied' seem fine to me: # ls -lhZ /sys/fs/cgroup/system.slice/slurmstepd.scope/job_42/step_extern/user/cgroup. freeze -rw-r--r--. 1 root root system_u:object_r:cgroup_t:s0 0 Jun 27 21:46 /sys/fs/cgroup/system.slice/slurmstepd.scope/job_42/step_extern/user/cgroup. freeze There are no SElinux alerts on the system. I am not sure whether the error messages about setgroups and _shutdown_x11_forward are actually the problem, or just something else being reported. The only system that I have to compare with is running Slurm 19.05 on CentOS 7 and is rather different. I would be interested to know if anyone else has had problems with extern job steps not shutting down. William

1 0

Job information if job is completed
by Gestió Servidors 24 Jun '25

24 Jun '25

Hello, Is there any way to get all information (like submit script or submit node) from a job that is completed? Something like "scontrol show jobid=XXX" when job is "running" or "pending". I need to inspect the submit script of a job but I only know job_id. Thanks.

6 5

Implementing a "soft" wall clock limit
by Davide DelVento 24 Jun '25

24 Jun '25

In the institution where I work, so far we have managed to live without mandatory wallclock limits (a policy decided well before I joined the organization), and that has been possible because the cluster was not very much utilized. Now that is changing, with more jobs being submitted and those being larger ones. As such I would like to introduce wallclock limits to allow slurm to be more efficient in scheduling jobs, including with backfill. My concern is that this user base is not used to it and therefore I want to make it easier for them, and avoid common complaints. I anticipate one of them would be "my job was cancelled even though there were enough nodes idle and no other job in line after mine" (since the cluster utilization is increasing, but not yet always full like it has been at most other places I know). So my question is: is it possible to implement "soft" wallclock limits in slurm, namely ones which would not be enforced unless necessary to run more jobs? In other words, is it possible to change the pre-emptability of a job only after some time has passed? I can think of some ways to hack this functionality myself with some cron or at jobs, and that might be easy enough to do, but I am not sure I can make it robust enough to cover all situations, so I'm looking for something either slurm-native or (if external solution) field-tested by someone else already, so that at least the worst kinks have been already ironed out. Thanks in advance for any suggestions you may provide!

7 14

enforce Qos to users
by laddaoui＠telecom-paris.fr 24 Jun '25

24 Jun '25

Hello everyone, I'm trying to use QoS to enforce resource limits on an association, but I'm having trouble with proper enforcement. I created a QoS with resource limits: ``` sacctmgr add qos qos_gpus flags=denyonlimit,overpartqos maxjobsperuser=4 maxtresperjob=gres/gpu=1 ``` Then I assigned it to an account: ``` sacctmgr modify account name=account-a set qos=qos_gpus defaultqos=qos_gpus systemctl restart slurmctld ``` Users in this account can bypass the QoS limits by explicitly specifying a different QoS when submitting jobs: ``` srun --qos=(normal|qos_gpus) ... ``` Even though I set `defaultqos=qos_gpus`, users can still choose any available QoS and bypass the intended resource limits. My question is: How can I restrict users to only using their assigned QoS and prevent them from specifying other QoS options? Is there a configuration I'm missing to enforce QoS restrictions properly? Best, --- info about my setup slurm version : tested on 23.11.4 and 23.02.7 AccountingStorageEnforce = associations,limits EnforcePartLimits = ALL

2 2

pam_slurm_adopt - ssh to compute nodes not working in slurm 24.11
by Marx, Wolfgang 23 Jun '25

23 Jun '25

Hi, We have defined in our cluster, that users can logon to a compute node from a login node,when they have actual a job running on the compute node. To get this functionallity working, we are usinge the pam_slurm_adopt. As long as we were using slurm 23.05 it all was working well. Now we have upgraded to slurm 24.11.5 and the ssh login to the compute nodes is not longer working. When a job of a user is running on a compute node the ssh of the user to this compute node is refused form the compute node. We have not changed anything in our configuration for the pam_slurm_adopt. Also there is no indication in the release notes that anything has changed regarding pam_slurm_adopt. Is this a known bug in Slurm 24.11 and has anyone facing the same problem. This is a very important feature, especially for our ANSYS users. Thanks Wolfgang Marx Wolfgang Marx, Basisdienste, Gruppe Hochleistungrechnen Technische Universität Darmstadt, Hochschulrechenzentrum Alexanderstraße 2, 64283 Darmstadt Tel.: +496151/16-71158 E-Mail: wolfgang.marx(a)tu-darmstadt.de Web: www.hrz.tu-darmstadt.de

2 1

read-only slurm user
by Hagdorn, Magnus Karl Moritz 23 Jun '25

23 Jun '25

Hi there, we use the slurm prometheus exporter to collect slurm metrics. This works pretty well. However, we have noticed that metrics for some of the restricted partitions are not collected. It occurred to me that this is because we are using an unprivileged user to run the exporter. I am trying to figure out the best way to allow an unprivileged user to collect all metrics. I could add the user to all the relevant groups. However, I am also thinking of using the new slurm exporter that uses the API in which case I need to somehow handle a token. It would be nice to have a readonly user, ie a user that cannot submit any jobs but only read the current state of the cluster. I guess setting MaxJobs and MaxSubmitJobs to 0 would do this. Any other suggestions? Regards magnus -- Dr. Magnus Hagdorn Charité – Universitätsmedizin Berlin Geschäftsbereich IT | Scientific Computing Campus Charité Mitte BALTIC - Invalidenstraße 120/121 10115 Berlin https://www.charite.de HPC Helpdesk: sc-hpc-helpdesk(a)charite.de

1 0

Doc Clarification: Heterogeneous Steps in Heterogeneous Job
by Steffen Christgau 19 Jun '25

19 Jun '25

Hi everybody, I am (along with others) a little bit puzzled by the meaning of a statement in the documentation concerning heterogeneous job steps inside het. jobs. The docs state (https://slurm.schedmd.com/archive/slurm-24.11.5/heterogeneous_jobs.html#het…): > You also cannot request heterogeneous steps from within a heterogeneous job. (A) On a very small Slurm test installation with just two nodes, the following het job that requests het steps (does it, right?!) runs fine: $ cat hetjob-steps.sh #!/bin/bash #SBATCH --mem-per-cpu=2g --nodes=1 --cpus-per-task=8 #SBATCH hetjob #SBATCH --mem-per-cpu=1g --nodes=1 --cpus-per-task=4 srun -l --cpus-per-task=4 nproc : -l --cpus-per-task=2 nproc $ cat slurm-125.out 1: 4 2: 2 3: 2 0: 4 The output looks reasonable and it looks like the above quote does not apply since one can apparently request het steps in a het job. Or am I wrong? The intro in the respective section also gives the impression that het jobsteps are a convenience feature that does not require het jobs, but it does not explicitly exclude the usage of het steps in het jobs: > Slurm version 20.11 introduces the ability to request heterogeneous job steps from within a non-homogeneous job allocation. This allows you the flexibility to have different layouts for job steps without requiring the use of heterogeneous jobs, where having separate jobs for the components may be undesirable. So what does the initial statement (A) actually mean then? Am I just using a lucky example which is actually not supported? A short clarification would be helpful. Thanks in advance Steffen

1 0

Wrong MaxRSS Behavior with cgroup v2 in Slurm
by Guillaume COCHARD 19 Jun '25

19 Jun '25

Hello, We've noticed a recent change in how MaxRSS is reported on our cluster. Specifically, the MaxRSS value for many jobs now often matches the allocated memory, which was not the case previously. It appears this change is due to how Slurm accounts for memory when copying large files, likely as a result of moving from cgroup v1 to cgroup v2. Here’s a simple example: copy_file.sh #!/bin/bash cp /distributed/filesystem/file5G /tmp cp /tmp/file5G ~ Two jobs with different memory allocations: Job 1 sbatch -c 1 --mem=1G copy_file.sh seff <jobid> Memory Utilized: 1021.87 MB Memory Efficiency: 99.79% of 1.00 GB Job 2 sbatch -c 1 --mem=10G copy_file.sh seff <jobid> Memory Utilized: 4.02 GB Memory Efficiency: 40.21% of 10.00 GB With cgroup v1, this script typically showed minimal memory usage. Now, under cgroup v2, memory usage appears inflated and depends on the allocated memory, which seems wrong. I believe this behavior aligns with similar issues raised by the Kubernetes community [1], and is consistent with how memory.current behaves in cgroup v2 [3]. According to Slurm’s documentation about cgroup v2, "this plugin provides cgroup's memory.current value from the memory interface, which is not equal to the RSS value provided by procfs. Nevertheless it is the same value that the kernel uses in its OOM killer logic." [2] While technically correct, this seems to mark a significant change in what MaxRSS and "Memory Efficiency" actually measure and renders those metrics almost useless. Our Configuration: ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup,task/affinity Question: Is there a way to restore more realistic MaxRSS values — specifically, ones that exclude file-backed page cache — while still using cgroup v2? Thanks, Guillaume References: [1] https://github.com/kubernetes/kubernetes/issues/118916 [2] https://slurm.schedmd.com/cgroup_v2.html#limitations [3] https://facebookmicrosites.github.io/cgroup2/docs/memory-controller.html

3 4

Re: Job information if job is completed
by Gestió Servidors 18 Jun '25

18 Jun '25

Hi, After reconfiguring slurm.conf to add script into the database, I have tried with a "normal" user to get that information (from a owned finished job). However, when I run "sacct -vvvv -B -j 92656", I get: sacct: Jobs Eligible in the time window from Epoch 0 to Wed Jun 18 15:33:03 2025 sacct: debug: Options selected: opt_completion=no opt_dup=no opt_field_list=User,JobID,Jobname%18,partition,state,time,submit,start,end,elapsed,nnodes,ncpus,nodelist, opt_help=0 opt_no_steps=yes opt_whole_hetjob=(null) sacct: debug: accounting_storage/slurmdbd: _connect_dbd_conn: Sent PersistInit msg sacct: debug2: Clusters requested: mycluster sacct: debug2: Userids requested: all sacct: debug2: Jobs requested: sacct: debug2: : 92656 sacct: error: Unknown error 1064 I have read that "sacct: error: Unknown error 1064" could be a error in the MySQL query syntax... but if I run same command "sacct -vvvv -B -j 92656" as root, I get submit script. I suppose problem is in database permissions, so I have added new permissions with "grant all on slurmdb.* TO 'user-sacct'@'localhost' identified by '..user-sacct..' with grant option;" (where user "user-sacct" is a user that acts as coordinator in SLURM). However, if I open a terminal with "user-sacct" and run "sacct -vvvv -B -j 92656", result is the same: "sacct: error: Unknown error 1064" How could give permissions to user-sacct to allow "sacct -B" command? Thanks.

1 0

Re: Job information if job is completed
by Gestió Servidors 18 Jun '25

18 Jun '25

Hello, So if there is no way to get all information from a finished job (specially submit script, not command line, but all content from submit script, like a copy of it), maybe a solution would be run a "prolog" script in each job to run a "cp" from the submit script. However, how could I copy the submit scrip from the prolog script? Because from the prolog script, I can access to some SLURM variables (https://slurm.schedmd.com/prolog_epilog.html) but I don't know how to know what the script is and run a simple "cp" to a destination folder. Thanks.

4 4

Re: [EXT] Re: slurm_pam_adopt module not working
by Ratnasamy, Fritz 17 Jun '25

17 Jun '25

Yes the file exists in /usr/lib64/security/. Best, *Fritz Ratnasamy*Data Scientist Information Technology On Tue, Jun 17, 2025 at 12:17 AM Sean Crosby <scrosby(a)unimelb.edu.au> wrote: > Hi Fritz, > > Does pam_slurm_adopt.so exist in the right location on the node? Normally > on EL hosts it would be /usr/lib64/security/pam_slurm_adopt.so > > # ls /usr/lib64/security/pam_slurm_adopt.so -la > -rwxr-xr-x 1 root root 291936 Mar 4 12:44 > /usr/lib64/security/pam_slurm_adopt.so > > If the file doesn't exist, pam would abnormally exit and not allow anyone > to log in. > > Sean > ------------------------------ > *From:* Ratnasamy, Fritz via slurm-users <slurm-users(a)lists.schedmd.com> > *Sent:* Tuesday, 17 June 2025 14:55 > *To:* Kevin Buckley <kevin.buckley.pawsey.org.au(a)gmail.com> > *Cc:* slurm-users(a)lists.schedmd.com <slurm-users(a)lists.schedmd.com> > *Subject:* [EXT] [slurm-users] Re: slurm_pam_adopt module not working > > * External email: Please exercise caution * > ------------------------------ > Thanks, for some reason I edited the /etc/pam.d/sshd via ansible but that > locked all users to the cluster. That same file works on a different > cluster where the files are pushed via puppet but with ansible it looks > like it is locking all users to the cluster. See below config file sshd: > > auth required pam_sepermit.so > auth substack password-auth > auth include postlogin > # Used with polkit to reauthorize users in remote sessions > -auth optional pam_reauthorize.so prepare > account required pam_nologin.so > ##SLURM > account sufficient pam_slurm_adopt.so action_no_jobs=deny > action_unknown=newest action_adopt_failure=deny action_generic_failure=deny > account sufficient pam_access.so > ##END SLURM > password include password-auth > # pam_selinux.so close should be the first session rule > session required pam_selinux.so close > session required pam_loginuid.so > # pam_selinux.so open should only be followed by sessions to be executed > in the user context > session required pam_selinux.so open env_params > session required pam_namespace.so > session optional pam_keyinit.so force revoke > session include password-auth > session include postlogin > # Used with polkit to reauthorize users in remote sessions > -session optional pam_reauthorize.so prepare > > > > *Fritz Ratnasamy *Data Scientist > Information Technology > > > > > On Wed, Jun 11, 2025 at 8:29 PM Kevin Buckley via slurm-users < > slurm-users(a)lists.schedmd.com> wrote: > > On 2025/06/11 12:46, Ratnasamy, Fritz via slurm-users wrote: > > > > We wanted to block users from ssh to a node where there are no jobs > > running however it looks like users are able to do so. We have installed > > the slurm_pam_adopt_module and set up the slurm.conf accordingly (the > same > > way we did on our first cluster where the pam module denies ssh access > > correctly). > > We saw a similar issue whereby the way that we had PAM setup, meant > that, and here I quote from SchedMD's Daniel Armengod: > > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---- > This is almost certainly caused by the fact that SSH's > `keyboard-interactive` > (not to be confused with `password`) AuthMethod forks a short-lived child > process that is involved in the authentication logic. Slurm's > pam_slurm_adopt > module latches on to that process (which is the wrong one, of course) and > things break in interesting ways from there. > > SSH authmethods `publickey` and `password` do not exhibit this behaviour > as SSH > does not fork a child process to offload the authentication > challenge-response > dialogue to. > > ... > > The key bit here is that in your last test you're forcing > `PreferredAuthentications=password`, which isn't actually the > `keyboard-interactive` AuthMethod that got picked before. > They work differently under the hood, even if as far as the > user is concerned, both methods just ask for a password. > > ... > > In summary: try disabling the `keyboard-interactive` authentication method > in > your compute nodes. pam_slurm_adopt should work correctly now. > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---- > > Maybe that's also your issue. > > > Daniel did say that SchedMD were going to update their documentation > to make that distinction, and it's effect, more explciit, so I would > expect it to be in the mainstream docs by now. > > HTH > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > CAUTION: This email has originated outside of University email systems. > Please do not click links or open attachments unless you recognize the > sender and trust the contents as safe. > > CAUTION: This email has originated outside of University email systems. > Please do not click links or open attachments unless you recognize the > sender and trust the contents as safe. > >

2 1

Re: slurm_pam_adopt module not working
by Ratnasamy, Fritz 17 Jun '25

17 Jun '25

Thanks, for some reason I edited the /etc/pam.d/sshd via ansible but that locked all users to the cluster. That same file works on a different cluster where the files are pushed via puppet but with ansible it looks like it is locking all users to the cluster. See below config file sshd: auth required pam_sepermit.so auth substack password-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so ##SLURM account sufficient pam_slurm_adopt.so action_no_jobs=deny action_unknown=newest action_adopt_failure=deny action_generic_failure=deny account sufficient pam_access.so ##END SLURM password include password-auth # pam_selinux.so close should be the first session rule session required pam_selinux.so close session required pam_loginuid.so # pam_selinux.so open should only be followed by sessions to be executed in the user context session required pam_selinux.so open env_params session required pam_namespace.so session optional pam_keyinit.so force revoke session include password-auth session include postlogin # Used with polkit to reauthorize users in remote sessions -session optional pam_reauthorize.so prepare *Fritz Ratnasamy*Data Scientist Information Technology On Wed, Jun 11, 2025 at 8:29 PM Kevin Buckley via slurm-users < slurm-users(a)lists.schedmd.com> wrote: > On 2025/06/11 12:46, Ratnasamy, Fritz via slurm-users wrote: > > > > We wanted to block users from ssh to a node where there are no jobs > > running however it looks like users are able to do so. We have installed > > the slurm_pam_adopt_module and set up the slurm.conf accordingly (the > same > > way we did on our first cluster where the pam module denies ssh access > > correctly). > > We saw a similar issue whereby the way that we had PAM setup, meant > that, and here I quote from SchedMD's Daniel Armengod: > > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---- > This is almost certainly caused by the fact that SSH's > `keyboard-interactive` > (not to be confused with `password`) AuthMethod forks a short-lived child > process that is involved in the authentication logic. Slurm's > pam_slurm_adopt > module latches on to that process (which is the wrong one, of course) and > things break in interesting ways from there. > > SSH authmethods `publickey` and `password` do not exhibit this behaviour > as SSH > does not fork a child process to offload the authentication > challenge-response > dialogue to. > > ... > > The key bit here is that in your last test you're forcing > `PreferredAuthentications=password`, which isn't actually the > `keyboard-interactive` AuthMethod that got picked before. > They work differently under the hood, even if as far as the > user is concerned, both methods just ask for a password. > > ... > > In summary: try disabling the `keyboard-interactive` authentication method > in > your compute nodes. pam_slurm_adopt should work correctly now. > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---- > > Maybe that's also your issue. > > > Daniel did say that SchedMD were going to update their documentation > to make that distinction, and it's effect, more explciit, so I would > expect it to be in the mainstream docs by now. > > HTH > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > CAUTION: This email has originated outside of University email systems. > Please do not click links or open attachments unless you recognize the > sender and trust the contents as safe. > >

2 1

MIG H100 with xeon Intel
by Richard Lefebvre 17 Jun '25

17 Jun '25

I'm having problems with Autodetect=nvml in gres.conf. I get on the controller log the following: error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 16-31 on node node001 doesn't match socket boundaries. (Socket 0 is cores 0-31). Consider setting SlurmdParameters=l3cache_as_socket (recommended) or override this by manually specifying core affinity in gres.conf. I did set l3cache_as_socket in the slurm.conf of the node, but I still get the error on the slurm controler I'm running 24.11.5 on AlmaLinux 9.5 Richard

2 1

How are the results produced by 'seff'?
by Loris Bennett 12 Jun '25

12 Jun '25

Hi, With Slurm 24.11.5 for some jobs I am seeing differences between the memory usage reported by 'seff' and that shown by Prometheus as 'cgroup_memory_rss_bytes' (and ultimately reported by 'jobstats' [1]). Certainly at the University of Delft they seem to feel that the memory usage reported by 'seff' is unreliable [2]. Is that indeed the case? Cheers, Loris Footnotes: [1] https://github.com/PrincetonUniversity/jobstats [2] https://doc.dhpc.tudelft.nl/delftblue/Slurm-trouble-shooting/ -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

1 0

slurm_pam_adopt module not working
by Ratnasamy, Fritz 11 Jun '25

11 Jun '25

Hi, We wanted to block users from ssh to a node where there are no jobs running however it looks like users are able to do so. We have installed the slurm_pam_adopt_module and set up the slurm.conf accordingly (the same way we did on our first cluster where the pam module denies ssh access correctly). We have PrologFlags=contain in slurm.conf and inside /etc/pam.d/sshd, we set the last row to : account sufficient pam_slurm_adopt.so action_no_jobs=deny action_unknown=newest action_adopt_failure=deny action_generic_failure=deny Are we missing anything? Thanks *Fritz Ratnasamy*Data Scientist Information Technology

3 2

CR_CPU used but only cores used
by Adrian Sevcenco 11 Jun '25

11 Jun '25

Hi! i have a weird situation in which only cores are used instead of CPUs this is Alma9/slurm 22.05.9 (the last one from epel) I have: conf.d/resources.conf 9:SelectType=select/cons_res conf.d/resources.conf 11:SelectTypeParameters=CR_CPU conf.d/resources.conf 5:TaskPluginParam=autobind=threads conf.d/resources.conf 2:TaskPlugin=task/affinity,task/cgroup cgroup.conf 2:ConstrainCores=no conf.d/parts_issaf.conf 8:PartitionName=CLUSTER DEFAULT=YES State=UP nodes=DEFAULT DefaultTime=240:00:00 MaxTime=480:00:00 LLN=yes MaxNodes=1 What am i missing? Why are only cores used? Thanks a lot!! Adrian

2 2

Restrict and prioritize usage of certain nodes according to accounts
by thomas.hartmann＠th-ht.de 09 Jun '25

09 Jun '25

Hi, I'm going to have the following situation on my hands and I would like to ask for some suggestions how to solve this in slurm. We're talking about a cluster that is not yet operational so there are no real legacy configs that one needs to take into account. I have two sets of physical nodes: 1. node01 - nodeXX: General Purpose Nodes. Everybody can use them 2. project_A_01 - project_A_XX: These nodes come with some restrictions regarding who can and who must not run jobs on them. Users are organized into accounts, pseudohierarchically representing workgroups / projects etc within the university. For instance, there is going to be a "physics" account with "project_blackhole", "workgroup_miller" as children. Users are associated with one or more accounts. I have three sets of accounts (each can have child accounts): 1. "General" accounts: These are allowed to use all physical nodes. 2. "ForProfit" accounts: These absolutely must not use the project_A_* nodes 3. "Project_A" accounts: Their jobs should run first and foremost on the project_A_* nodes but they are also allowed to run on the node* nodes. My first idea would be to create two partitions, one with all nodes in there and a second one with only those nodes that the "ForProfit" are allowed to use. Using "AllowAccounts" or "DenyAccounts" would implement the restriction I need. In the future, there might be more project specific nodes for other projects. I would be grateful for any alternative solution or hints, how this could be solved optimally. Thanks a lot! Thomas

5 4

Problem querying slurm batch script archive
by Davide DelVento 09 Jun '25

09 Jun '25

I configured slurm.conf on our machine with AccountingStoreFlags=job_script,job_comment So that people can run sacct -Bj <job_ID> and check what the batch script was for the specified job_ID. This works fine for root and works great for many users (who can see their own job scripts, but not those of other users, as it ought to be). Unfortunately, for several users (but not even close to half of the users of the machine), the sacct -Bj command returns nothing, like if they queried someone else's job, even when querying their own. Does anybody have any clue on what might be wrong and triggering this odd behavior? Thanks!

1 0

seff command not found
by Ratnasamy, Fritz 09 Jun '25

09 Jun '25

Hi, We installed a new slurm version and it returns "command not found" for seff. I do not remember doing any manual installation for the previous versions, I thought it was coming with sacct, sbatch, ect... Any idea how I would need to set it up? I read online seff is actually a perl script. Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Re: Integration testing of Slinky images
by Nicolas Bock 06 Jun '25

06 Jun '25

Hi Brian, sorry for breaking the email thread, my subscription settings were not set correctly and I didn't receive your response as email. Thanks for the tips! I'll give that a try. Best, Nick > Nick, > Presuming you have followed the SchedMD instructions, you should be able > to get a session in your login container: > kubectl --namespace=slurm exec -it statefulsets/slurm-controller -- bash > --login > From there, you can do any standard testing you like. Simple 'srun > hostname' should work to let you know slurm itself is doing it's part. > You can also do commands such as 'scontrol show nodes' to see what and > how many resources you have configured. > Outside that, you need to ensure you have configured your slurm > containers to request the resources you plan on using (eg: gpus) and > that you have enough of them for the script(s) you wish to run. > Brian Andrus

1 0

X11 Forwarding Crashes X Session for too large windows
by Felix Winterhalter 06 Jun '25

06 Jun '25

Hello everyone, we have run into a peculiar issue with our setup. First and foremost let me say that X forwarding using ssh -X or ssh -Y works fine and doesn't show any irregularities. Trying to use the X forwarding of slurm however results in a weird situation. We are using it like: srun --pty --x11 bash And then run something like glxgears. The program starts up normal however whenever you increase the window size beyond a certain size the connection to the X server gets broken and glxgears crashes. The same is true for any other program using X, some don't even start as their inital window size is too large, others crash as soon as you resize them beyond a certain point. We have no clue what might be the cause only that it seems to be related to how slurm does the XForwarding. We've tried X11Parameters both with and without host_authority. We're running 25.05.0 Any help or idea would be very appreciated. Cheers, Felix This message and any attachments are for the sole use of the intended recipient(s). It may contain information that is confidential and privileged. If you are not the intended recipient of this message, you are prohibited from printing, copying, forwarding or saving it. Please delete the message and attachments and notify the sender immediately.

1 1

Integration testing of Slinky images
by Nicolas Bock 06 Jun '25

06 Jun '25

Hi, I was wondering whether there is any integration testing done on the containers and the operator published under the slinky project. Or asked differently, once I have deployed slinky, what tests should I run to confirm that the cluster is working as intended? Thanks! Nick

2 1

X11 performance terrible using plugin
by Simon Andrews 06 Jun '25

06 Jun '25

On our cluster we've noticed that if we use the native x11 slurm plugin (PrologFlags=x11) then X applications work, but are really slow and unresponsive. Even opening menus on graphical application is painfully slow. On the same system if I do a direct ssh connection with ssh -YC from the head node the same applications are quick and responsive. Any suggestions for what might be causing this, and how I can get the native x11 to have the same responsiveness as a direct ssh connection? Many thanks Simon.

6 8

Transfer from GPFS via slurm
by Ratnasamy, Fritz 06 Jun '25

06 Jun '25

Hi, We were told by our hardware provider that large datasets copied from NFS location to GPFS could be conducted via slurm to monitor the transfer. I am not sure of this works as I could not find much online. Otherwise apart from globus and rsync, what would you suggest as a tool/command to copy files from on a cluster between NFS and GPFS? Best, *Fritz Ratnasamy*Data Scientist Information Technology

2 1

Changing account names in sacctmgr
by Burian, John 05 Jun '25

05 Jun '25

All, For reasons I won’t go into, we find we need to change the POSIX group names for all of our users. My understanding is that in the absence of an --account option, sbatch/salloc assumes the account is the user’s primary POSIX group. So, to change group names, we’ll need to change the account names on all the associations in sacctmgr. And ideally, we would change account names in accounting records so we don’t lose accounting history. One is tempted to get into the Slurm DB and start running SQL updates. Is that path too fraught with peril to consider? Or should I just recreate the associations with the new names and take my lumps on the accounting history? John

4 4

error building debug packages on ubuntu 24.04
by Charles Hedrick 03 Jun '25

03 Jun '25

It was unable to find lib/security/pam_slurm_adopt.so It was built as lib64/security, not lib/security. ./debian/slurm-smd-libpam-slurm-adopt.install needs to be lib64/security/pam_slurm_adopt.so More likely, it needs to be built dynamically, so it will work everywhere.

1 0

Slurm version 25.05 is now available
by Tim Wickberg 29 May '25

29 May '25

We are pleased to announce the availability of Slurm 25.05. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md The Slurm documentation has also been updated to the 25.05 release: https://slurm.schedmd.com Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

Slurm accounts managed via ansible
by Ratnasamy, Fritz 28 May '25

28 May '25

Hi, I was wondering whether there might be built in support for managing slurm accounts,users,associations in ansible. it would be nice to be able to organize accounts in a yaml style file and modify accounts settings via gitlab CI/CD. For example, in a format as such: accounts: - name: "Boss1" parent: "root" description: "Big Boos" fairshare: 20 - name: "UnderBoss1" parent: "root" description: "working for Boss1" fairshare: 1 users: -username: "user1" -username: "user2" *Fritz Ratnasamy* Data Scientist Information Technology

3 2

Account getting duplicated
by Ratnasamy, Fritz 27 May '25

27 May '25

Hi, the slurm db duplicates all our account associations. one with cluster=cluster and another where cluster=venus (which is our actual cluster). is that intended? Or should I make any changes? *Fritz Ratnasamy*Data Scientist Information Technology

2 1

Slurm Reporting Difference between sreport and sacct
by Passant Hafez 23 May '25

23 May '25

Hi all, I was wondering if someone can help explaining this discrepancy. I have different values for project gpu consumption using sreport vs sacct (+ some calculations) This is an example that shows this: sreport -t hours -T gres/gpu cluster AccountUtilizationByuser start=2025-04-01 end=2025-04-05 | grep project1234 gives 178 while sacct -n -X --allusers --accounts=project1234 --start=2025-04-01 --end=2025-04-05 -o elapsedraw,AllocTRES%80,user,partition gives 213480 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus 249507 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus 13908 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus 9552 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus 4 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu 11 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu ... I will not bore you with the full output and its calculation, but the first job alone consumed 213480 seconds/60/60 * 8 gpus that's 474.4 gpu hours which is way more than the 178 hrs reported by sreport Any clue why these are inconsistent? or how sreport calculated the 178 value? All the best, Passant

3 3

Crash in "slurmd -C" when latest NVIDIA drivers are used
by Taras Shapovalov 22 May '25

22 May '25

Hello, Does someone have idea why "slurmd -C" crashes when it unloads gpu_nrt.so with latest NVIDIA drivers (570 and 575)? We checked, there is no crash in cuda at the moment and gpu_nvml.so works fine, all nvml calls finish successfully, dlclose on gpu_nvml.so works fine. The crash does not depend whether real GPUs present or not. Steps to reproduce: 1. Install Ubuntu 24.04 2. wget https://download.schedmd.com/slurm/slurm-24.11.4.tar.bz2<https://download.schedmd.com/slurm/slurm-24.11.4.tar.bz2;> 3. tar fx ./slurm-24.11.4.tar.bz2 4. cd slurm-24.11.4 5. apt-get install cuda-12-8 hwloc libmunge-dev -y 6. ./configure 7. make && make install 8. Run "slurmd -C", or sometimes "slurmd -vvv -C" to get the crash. Stack trace: #0 0x0000155555544b2a strlen (ld-linux-x86-64.so.2 + 0x28b2a) #1 0x000015555551fc08 __GI__dl_exception_create (ld-linux-x86-64.so.2 + 0x3c08) #2 0x000015555551d298 __GI__dl_signal_error (ld-linux-x86-64.so.2 + 0x1298) #3 0x000015555551e81d _dl_close (ld-linux-x86-64.so.2 + 0x281d) #4 0x000015555551d51c __GI__dl_catch_exception (ld-linux-x86-64.so.2 + 0x151c) #5 0x000015555551d669 _dl_catch_error (ld-linux-x86-64.so.2 + 0x1669) #6 0x0000155554e97c73 _dlerror_run (libc.so.6 + 0x97c73) #7 0x0000155554e979a6 __dlclose (libc.so.6 + 0x979a6) #8 0x0000155555388a25 gpu_plugin_fini (libslurmfull.so + 0x188a25) #9 0x000015555538f2ef gres_get_autodetected_gpus (libslurmfull.so + 0x18f2ef) #10 0x0000555555564828 _print_config (slurmd + 0x10828) #11 0x0000155554e2a1ca __libc_start_call_main (libc.so.6 + 0x2a1ca) #12 0x0000155554e2a28b __libc_start_main_impl (libc.so.6 + 0x2a28b) #13 0x000055555555fc75 _start (slurmd + 0xbc75) I don't really think the problem is in gpu_nrt itself, seems the problem is in memory corruption somewhere else, but I am not sure. The issue is reproduced constantly. A Best regards, Taras

2 3

Slurm 24.11.x + pyxis + TaskProlog -> fail
by Stefan Staeglich 21 May '25

21 May '25

Hi, we are currently evaluating Slurm 24.11. We are using pyxis in our cluster and have encountered the following problem in connection with TaskProlog: https://github.com/NVIDIA/pyxis/issues/163 Do you know more about the problem and the cause? Best, Stefan

1 0

sacctmgr: error: Sending PersistInit msg: Connection refused
by Ratnasamy, Fritz 18 May '25

18 May '25

Hi, We are working on a test cluster with slurm 24.11.3 and I am getting this error message from the login or compute nodes (note that the error does not show when run from the controller node): sacctmgr list associations tree format=cluster,account,user,maxnodes sacctmgr: error: _open_persist_conn: failed to open persistent connection to host:localhost:6819: Connection refused sacctmgr: error: Sending PersistInit msg: Connection refused Does someone know how fix that issue? Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

2 1

2026

2025

2024

slurm-users