- slurm-users - lists.schedmd.com

Run a healthcheck job on all nodes
by John Hearns 18 Aug '25

18 Aug '25

I may have asked this already. I want to run a healtcheck job on all nodes. I can select the nodes in a partition by hand, the write a bash cript to get a list of nodes using nodeset -e Then submit to each node in the list using sbatch -w Is there a cleaner way of doing this? John Hearns

4 5

SLUG 25?
by Dave Lloyd 11 Aug '25

11 Aug '25

Are there plans for a SLUG this year? I didn’t see any announcements on SchedMD’s website nor anything in the list. Thanks, -- Dave Lloyd This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

1 0

sinfo history
by Steve Kirk 11 Aug '25

11 Aug '25

Hi, Am I correct in thinking that the history of a *node* as shown by sinfo isn't stored anywhere by Slurm? Interested to know if slurm can tell me historically when a node was draining,drained etc. Regards, Steve

5 6

Slurm version 25.05.2 is now available
by Marshall Garey 07 Aug '25

07 Aug '25

We are pleased to announce the availability of Slurm version 25.05.2. This version fixes a few regressions with x11 forwarding in 25.05 that may prevent applications from launching, adds support for PMIx v6.x, fixes a variety of stability issues, fixes a regression where --tres-per-task was ignored, and fixes additional minor to moderate severity issues. The full list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Building Debian packages - problem
by John Hearns 05 Aug '25

05 Aug '25

I am trying to build Slurm version 24.11.6 on an Ubuntu 22.04 system. I download and unpack the source mk-build-deps -i debian/control (not run as root) debuild -b -uc -us I get this error repeated many times: dpkg-shlibdeps: warning: can't extract name and version from library name 'libslurmfull.so' Has anyone seen similar issues? For information, there is an existing SLurm installation on this syste, with version 21 dpkgs from the Ubuntu repository.

4 4

slurmctld failed to start
by Dhumal, Dr. Nilesh 01 Aug '25

01 Aug '25

Hello, We recently installed slurm-25 on Redhat linux. We failed to start the slurmctld service. sudo systemctl start slurmctld Job for slurmctld.service failed because the control process exited with error code. See "systemctl status slurmctld.service" and "journalctl -xeu slurmctld.service" for details. sudo systemctl status slurmctld × slurmctld.service - Slurm controller daemon Loaded: loaded (/usr/local/lib/systemd/system/slurmctld.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Thu 2025-07-31 22:23:18 EDT; 1min 3s ago Process: 44317 ExecStart=/usr/local/sbin/slurmctld --systemd $SLURMCTLD_OPTIONS (code=exited, status=1/FAILURE) Main PID: 44317 (code=exited, status=1/FAILURE) CPU: 35ms Jul 31 22:22:29 fgcu-compute01 systemd[1]: Starting Slurm controller daemon... Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: If munged is up, restart with --num-threads=10 Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: Munge encode failed: Failed to connect to "/run/munge/mung> Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: Failed to create MUNGE Credential Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: Couldn't load specified plugin name for auth/munge: Plugin> Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: cannot create auth context for auth/munge Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] fatal: failed to initialize auth plugin Jul 31 22:23:18 fgcu-compute01 systemd[1]: slurmctld.service: Main process exited, code=exited, status=1/FAILURE Jul 31 22:23:18 fgcu-compute01 systemd[1]: slurmctld.service: Failed with result 'exit-code'. Jul 31 22:23:18 fgcu-compute01 systemd[1]: Failed to start Slurm controller daemon. Here is munge service status. munge.service - MUNGE authentication service Loaded: loaded (/usr/local/lib/systemd/system/munge.service; enabled; preset: disabled) Active: active (running) since Thu 2025-07-31 22:06:14 EDT; 19min ago Docs: man:munged(8) Main PID: 44039 (munged) Tasks: 4 (limit: 606218) Memory: 1.4M CPU: 18ms CGroup: /system.slice/munge.service └─44039 /usr/local/sbin/munged Jul 31 22:06:14 fgcu-compute01 systemd[1]: Starting MUNGE authentication service... Jul 31 22:06:14 fgcu-compute01 systemd[1]: Started MUNGE authentication service. Any suggestion is apprecieted to resolve this issue. Thanks, Nilesh Dhumal Associate Professor of Chemistry, http://faculty.fgcu.edu/ndhumal/ Coordinator, FGCU Computational Facility, https://www.fgcu.edu/cas/facultyresources/computationalfacility/ SH-430; Department of Chemistry and Physics Florida Gulf Coast University 10501 FGCU Boulevard South Fort Myers, FL 33965-6565 Phone: (239) 745-4394 Email: ndhumal(a)fgcu.edu

2 3

X11 forwarding broken in 25.05.1
by Patryk Bełzak 24 Jul '25

24 Jul '25

Hi, we've recentry upgraded our slurm from 24.11.3 to 25.05.1 and it seems that since the upgrade the ssh X11 forwaring is broken. Quick recap - * on Monday 14'th I performed slurdbd and slurmctld upgrades - X forwarding was still working * on Tuesday 15'th I performed slurmd upgrades - X forwarding stopped working The issue is very hard to determine and it looks like it sits somhere in slurm code. You can submit a job with --x11 and it starts corretly. Xauthority is created, you have all the magic cookies needed, but when you try to start any application, you get error related to permissions I guess, see for yourself: ``` me@sand ~ ssh -X -Y ui [wcss] me@ui.wcss.pl:~ > srun -p lem-cpu-short -A kdm-staff --gres=storage:local:50G -c 12 --mem 12G -t 1:0:0 --x11 --pty /bin/bash [wcss] me@r17ch05b01 ~ > xauth list r17ch05b01.lem.kdm.wcss.pl/unix:91 MIT-MAGIC-COOKIE-1 d82a2efd [wcss] me@r17ch05b01 ~ > xterm xterm: Xt error: Can't open display: localhost:91.0 [wcss] me@r17ch05b01 ~ > date && telnet -4 localhost 6091 || date Wed Jul 23 12:02:39 CEST 2025 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Connection closed by foreign host. Wed Jul 23 12:02:41 CEST 2025 ``` As you can see the connection to port is being dropped/killed after a second or two. Now, it doesn't really matter which flags for ssh you pick (-X or -Y or both). X forwarding is working when you log in as a regular user outside of slurm job. Also if I do ssh localhost inside a job, then I can perform connection to port assigned to $DISPLAY and it isn't dropped - but it doesn't work since $DISPLAY and cookies are being messed up when you perform triple jump and one within same host. Our worker nodes are mostly on el9.5 AlmaLinux. Some are on el8.10 - and there acutally you can do some X forwarding but you must use both -X and -Y (which wasn't the case before slurm upgrade). TLS is disabled in slurm.conf. I am 100% sure that both SSHD and Xorg are properly configured. Has anyone encountered similiar issue? Or any comment from slurm dev team? Best regards Patryk -- Wroclaw Centre for Networking and Supercomputing

3 2

New SchedMD Bike Kits Are Here - Preorder Now Open!
by Victoria Hobson 24 Jul '25

24 Jul '25

We’re excited to announce that SchedMD-branded bike kits are officially available for preorder! Each kit features both the SchedMD and Slurm logos—designed for comfort, performance, and pride.Pick between thoughtfully styled designs for mountain and road biking in each of the following categories: • Jerseys (zip-up or v-neck short sleeve) • Bibs (gravel bib with side and back pockets or switch bib) • Gloves (tech or summer) • Cycling socks with the SchedMD or Slurm logo (the right answer here is to just order both!) Preorders are open now through July 30, 2025. After the store closes, kits will be handcrafted and shipped directly to your door. Please note: delivery will take 8–10 weeks after preorder ends. Want 20% off your order? Reach out to Jess directly (jess(a)schedmd.com) to get an exclusive promo code! Ready to ride in style? Check out the SchedMD bike kits here: https://store.dnacycling.com/schedmd — Thanks for being part of the SchedMD community!

1 0

strange GPU "no resources selected" error
by Paul Raines 23 Jul '25

23 Jul '25

Recently (past 3 days) on our NVIDIA DGX A100 systems running Ubuntu 22.04.5 and slurm 24.11.5 we have had jobs that ask for a gpu, get started by Slurm, but fail to be given a GPU and then fail. In the slurmctld log we see a line like: [2025-07-22T02:46:29.697] error: gres/gpu: job 6919154 node A100-04 no resources selected on the slurmd log I see no errors for the job but there is a line like [2025-07-22T02:46:29.757] [6919154.extern] task/cgroup: _handle_device_access: GRES: job devices.deny: adding c 195:0 rwm(/dev/nvidia0) for all 8 of the GPUs on the node. Other jobs still seem to start up and get a GPU fine. If you look at the job stats one sees: ReqTRES : billing=7,cpu=1,gres/gpu=1,mem=96G,node=1 AllocTRES : billing=3,cpu=1,mem=96G,node=1 showing that even though the gpu was requested, it was not allocated. Occasionly on these boxes (and only these -- my Dell Rocky 8 boxes with GPUS have no problem) we see the nodes go into drain mode with the "res/gpu GRES core specification ... doesn't match socket boundaries." message as per https://support.schedmd.com/show_bug.cgi?id=22498 It does seem to happen after slurmctld restart. I then restart slurmd on the nodes and can resume SLURM on the nodes whenever that happens. Otherwise nothing has changed on these nodes with SLURM config or the OS in over a month. Definition of the nodes are NodeName=A100-[01-04] \ CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 \ ThreadsPerCore=1 RealMemory=1031000 MemSpecLimit=2048 \ TmpDisk=1400000 Feature=amd,epyc,a100 \ Gres=gpu:a100-sxm4-40gb:8 and gres.conf on the nodes is simply AutoDetect=nvml --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

slurm 25.05.0 : allocates less GPUs than requested by --gres=gpu:count
by Jan Gmys 23 Jul '25

23 Jul '25

Since upgrading slurm to 25.05.0 (22.05.9 -> 23.11.11 -> 25.05.0) some jobs requesting --gres=gpu:reqcount GPUs are allocated less than reqcount GPUs if some of the node's GPUs are already in use by other jobs. We have a node - let's call it ares-c02-06 - with 2 GPUs. Consider the following test script: > #!/bin/bash > #SBATCH --nodes=1 > #SBATCH --ntasks-per-node=1 > #SBATCH --time=1-00:00:00 > > echo "CUDA_VISIBLE_DEVICES: " $CUDA_VISIBLE_DEVICES > echo "SLURM_JOB_GPUS: " $SLURM_JOB_GPUS > echo "SLURM_GPUS_ON_NODE: " $SLURM_GPUS_ON_NODE > sleep 10d Submit a job to the node: > > sbatch *--gres=gpu:1* --nodelist=ares-c02-06 job.sh > > Submitted batch job 1950559 > The job starts. Now submit the script again, asking for 2 GPUs > sbatch*--gres=gpu:2* --nodelist=ares-c02-06 job.sh > > Submitted batch job 1950567 This second job should not start as the ressources are not available. Surprisingly, _both jobs are running_ > $ squeue -w ares-c02-06 > JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) > 1950567 gpuai job.sh jan.gmys R 1:55 1 ares-c02-06 > 1950559 gpuai job.sh jan.gmys R 2:58 1 ares-c02-06 In the second job - 1950567 - AllocTRES shows gres/gpu=1, instead of the requested gres/gpu=2. > # sacct -j 1950567,1950559 -X -o jobid%10,reqtres%45,alloctres%60 > JobID ReqTRES AllocTRES > ---------- --------------------------------------------- ------------------------------------------------------------ > 1950559 billing=1,cpu=1,gres/gpu=1,mem=4G,node=1 billing=1,cpu=1,gres/gpu:l40s=1,gres/gpu=1,mem=4G,node=1 > 1950567 billing=1,cpu=1,*gres/gpu=2*,mem=4G,node=1 billing=1,cpu=1,gres/gpu:l40s=1,*gres/gpu=1*,mem=4G,node=1 The output of both jobs > $ cat slurm-1950559.out > CUDA_VISIBLE_DEVICES: 0 > SLURM_JOB_GPUS: 0 > SLURM_GPUS_ON_NODE: 1 > > $ cat slurm-1950567.out > CUDA_VISIBLE_DEVICES: 0 > SLURM_JOB_GPUS: 1 > SLURM_GPUS_ON_NODE: 1 CUDA_VISIBLE_DEVICES is set to 0 for both jobs. SLURM_JOB_GPUS is 0 resp. 1. *Environment: * - RHEL 9.1 - slurm 25.05.0 - The GRES configuration seems fine, AutoDetect is off : > # /usr/sbin/slurmd -G --conf-server hpc-slurm.cluster.hpc -v > [2025-07-22T16:44:05.548] GRES: Global*AutoDetect=off*(4) > [2025-07-22T16:44:05.548] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, > minor 0 > [2025-07-22T16:44:05.548] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, > minor 1 > [2025-07-22T16:44:05.548] GRES: gpu device number 0(/dev/nvidia0):c 195:0 rwm > [2025-07-22T16:44:05.548] GRES: gpu device number 1(/dev/nvidia1):c 195:1 rwm > [2025-07-22T16:44:05.548] Gres Name=gpu Type=L40S Count=2 Index=0 ID=7696487 > File=/dev/nvidia[0-1] Links=(null) > Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT and 'scontrol show node' > NodeName=ares-c02-06 Arch=x86_64 CoresPerSocket=24 > CPUAlloc=0 CPUEfctv=48 CPUTot=48 CPULoad=0.00 > AvailableFeatures=(null) > ActiveFeatures=(null) > Gres=gpu:L40S:2 > NodeAddr=ares-c02-06 NodeHostName=ares-c02-06 Version=25.05.0 > OS=Linux 5.14.0-162.6.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 30 > 07:36:03 EDT 2022 > RealMemory=386000 AllocMem=0 FreeMem=363069 Sockets=2 Boards=1 > State=IDLE+RESERVED+PLANNED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A > MCS_label=N/A > Partitions=gpuai > BootTime=2025-07-04T10:26:27 SlurmdStartTime=2025-07-22T14:48:54 > LastBusyTime=2025-07-22T16:26:36 ResumeAfterTime=None > CfgTRES=cpu=48,mem=386000M,billing=48,gres/gpu=2 > AllocTRES= *Debug notes :* - When using the --gpus option instead of --gres everything works as expected : the second job is PENDING (Ressources) - Tried both, ConstrainDevices=on/off in cgroup.conf, same result - The same is happening on other multi-GPU nodes of the cluster - When the --gres=gpu:2 job is submitted first, i.e. when all GPUs are taken, the second (--gres=gpu:1) job waits correctly. - When both GPUs are free, the --gres=gpu:2 job, correctly gets both GPUs : CUDA_VISIBLE_DEVICES: 0,1 - It worked in slurm 22.05.9 (we recently upgraded in two steps -> 23.11.11 -> 25.05.0) - The only viable workaround I see for the moment is to intercept --gres and --gpus-per-node (don't even have it in job_desc I think!! :-/) options in job_submit.lua and force users to use the --gpus option, which seems to works fine. Anyone experienced similar issues? Any idea how to solve this would be highly appreciated. Jan == -- Jan Gmys Ingénieur de recherche Support HPC/IA pour la plateforme MesoNET Mésocentre de Calcul Scientifique Intensif de l'Université de Lille

1 0

2025

2024

slurm-users ----- 2025 ----- September 2025 August 2025 July 2025 June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users