- slurm-users - lists.schedmd.com

pam_slurm_adopt malfunct after slurm upgrade to 25.11
by taleintervenor＠sjtu.edu.cn 16 Dec '25

16 Dec '25

Hello everyone: We recently upgrade slurm to 25.11 and find the pam_slurm_adopt got broken. This cause the users cannot ssh to the compute node where their jobs is running on. Of course we have make sure all the slurm related packages have been upgraded together. Simplest test process is like: ``` > cat test1.sh #!/bin/bash #SBATCH --job-name=test #SBATCH --partition=debug #SBATCH --nodes=1 #SBATCH --nodelist=cas639 sleep 6000 > sbatch test1.sh Submitted batch job 51047091 > squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 51047091 debug test tester R 0:28 1 cas639 > ssh cas639 (wait for a long time...) Connection closed by 172.16.3.129 port 22 (finally failed to ssh) ``` on the target compute node, we can see the following debug message from pam_slurm_adopt.so: ``` cas639 pam_slurm_adopt[1007301]: debug2: _establish_config_source: using config_file=/etc/slurm/slurm.conf (default) cas639 pam_slurm_adopt[1007301]: debug: slurm_conf_init: using config_file=/etc/slurm/slurm.conf cas639 pam_slurm_adopt[1007301]: debug: Reading slurm.conf file: /etc/slurm/slurm.conf cas639 pam_slurm_adopt[1007301]: PreemptMode=GANG is a cluster-wide option and cannot be set at partition level, option ignored. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/auth_munge.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge authentication plugin type:auth/munge version:0x190b00 cas639 pam_slurm_adopt[1007301]: debug: auth/munge: init: loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/certgen_script.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Certificate generation script plugin type:certgen/script version:0x190b00 cas639 pam_slurm_adopt[1007301]: debug: certgen/script: init: loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/hash_k12.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:KangarooTwelve hash plugin type:hash/k12 version:0x190b00 cas639 pam_slurm_adopt[1007301]: debug: hash/k12: init: init: KangarooTwelve hash plugin loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/tls_none.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Null tls plugin type:tls/none version:0x190b00 cas639 pam_slurm_adopt[1007301]: debug: tls/none: init: tls/none loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/accounting_storage_slurmdbd.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Accounting storage SLURMDBD plugin type:accounting_storage/slurmdbd version:0x190b00 cas639 pam_slurm_adopt[1007301]: accounting_storage/slurmdbd: init: Accounting storage SLURMDBD plugin loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug3: Trying to load plugin /usr/lib64/slurm/cred_munge.so cas639 pam_slurm_adopt[1007301]: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x190b00 cas639 pam_slurm_adopt[1007301]: cred/munge: init: Munge credential signature plugin loaded cas639 pam_slurm_adopt[1007301]: debug3: Success. cas639 pam_slurm_adopt[1007301]: debug: Reading cgroup.conf file /etc/slurm/cgroup.conf cas639 pam_slurm_adopt[1007301]: debug4: found StepId=51047091.extern cas639 pam_slurm_adopt[1007301]: debug4: found StepId=51047091.batch cas639 pam_slurm_adopt[1007301]: Connection by user tester: user has only one job 51047091 cas639 pam_slurm_adopt[1007301]: debug: _adopt_process: trying to get StepId=51047091.extern to adopt 1007301 cas639 pam_slurm_adopt[1007301]: debug: Leaving stepd_add_extern_pid cas639 pam_slurm_adopt[1007301]: debug: Leaving stepd_get_x11_display cas639 pam_slurm_adopt[1007301]: debug: entering stepd_get_namespace_fd ``` It looks like something block during stepd_get_namespace_fd? And we found nothing in slurmd log even with SlurmdDebug = debug5, so I guess the pam module had not run to the step to talk with slurmd (if it should). Would it be a compatibility problem between slurm25.11 and EL8 system or cgroup/v1? Or can anyone help to give some suggestion on how to further locate the fault point? Best regards, Hermes

1 0

opposite of --array
by Herbert Fruchtl 15 Dec '25

15 Dec '25

Folks, When I use sacct, it always shows all tasks of an array job individually. This is supposed to be switched on by --array, but it seems to be the default. How do I switch it off? Thanks in advance, Herbert -- Herbert Fruchtl (he/him) Senior Scientific Computing Officer / HPC Administrator School of Chemistry, IT Services University of St Andrews -- The University of St Andrews is a charity registered in Scotland: No SC013532

1 0

'sacct -a' missing running jobs 'held' by user
by Lee 15 Dec '25

15 Dec '25

Hello, I am using slurm 23.02.6. I have a strange issue. I periodically use sacct to dump job data. I then generate reports based on the resource allocation of our users. Recently, I noticed some 'missing' jobs from my query. The missing jobs came from a user who had a large array job, who then 'held' all of the array jobs. This included 'holding' the Running array jobs. Now, if I run `sacct -a -S YYYY-MM-DD --format="jobidraw,jobname"`, the job will be missing from that query. However, if I query specifically for that job, i.e. `sacct -j RAWJOBID -S YYYY-MM-DD --format="jobidraw,jobname", the job is present. *Question* : 1. How can I include the 'held' running job when I do my bulk query with `sacct -a`? Finding these outliers and adding them ad-hoc to my dumped file is too laborious and isn't feasible. *Minimum working example *: #. Submit a job : myuser@clusterb01:~$ srun --pty bash # landed on dgx29 #. Hold job myuser@clusterb01:~$ scontrol hold 120918 myuser@clusterb01:~$ scontrol show job=120918 JobId=120918 JobName=bash UserId=myuser(123456) GroupId=rdusers(7000) MCS_label=N/A Priority=0 Nice=0 Account=allusers QOS=normal JobState=*RUNNING* Reason=*JobHeldUser* Dependency=(null) Requeue=0 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:00:29 TimeLimit=7-00:00:00 TimeMin=N/A SubmitTime=2025-12-15T13:31:28 EligibleTime=Unknown AccrueTime=Unknown StartTime=2025-12-15T13:31:28 EndTime=2025-12-22T13:31:28 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-12-15T13:31:28 Scheduler=Main Partition=defq AllocNode:Sid=clusterb01:4145861 ReqNodeList=(null) ExcNodeList=(null) NodeList=dgx29 BatchHost=dgx29 NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=9070M,node=1,billing=1 AllocTRES=cpu=2,mem=18140M,node=1,billing=2 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=9070M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=bash WorkDir=/home/myuser Power= #. Release job myuser@clusterb01:~$ scontrol release 120918 #. Show job again myuser@clusterb01:~$ scontrol show job=120918 JobId=120918 JobName=bash UserId=myuser(123456) GroupId=rdusers(7000) MCS_label=N/A Priority=1741 Nice=0 Account=allusers QOS=normal JobState=*RUNNING* Reason=*None* Dependency=(null) Requeue=0 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:01:39 TimeLimit=7-00:00:00 TimeMin=N/A SubmitTime=2025-12-15T13:31:28 EligibleTime=Unknown AccrueTime=Unknown StartTime=2025-12-15T13:31:28 EndTime=2025-12-22T13:31:28 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-12-15T13:31:28 Scheduler=Main Partition=defq AllocNode:Sid=clusterb01:4145861 ReqNodeList=(null) ExcNodeList=(null) NodeList=dgx29 BatchHost=dgx29 NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=9070M,node=1,billing=1 AllocTRES=cpu=2,mem=18140M,node=1,billing=2 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=9070M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=bash WorkDir=/home/myuser/ Power= #. In slurmctld, I see : root@clusterb01:~# grep 120918 /var/log/slurmctld [2025-12-15T13:31:28.706] sched: _slurm_rpc_allocate_resources JobId=120918 NodeList=dgx29 usec=1269 [2025-12-15T13:31:47.751] sched: _hold_job_rec: hold on JobId=120918 by uid 123456 [2025-12-15T13:31:47.751] sched: _update_job: set priority to 0 for JobId=120918 [2025-12-15T13:31:47.751] _slurm_rpc_update_job: complete JobId=120918 uid=123456 usec=189 [2025-12-15T13:32:48.081] sched: _release_job_rec: release hold on JobId=120918 by uid 123456 [2025-12-15T13:32:48.081] _slurm_rpc_update_job: complete JobId=120918 uid=123456 usec=268 [2025-12-15T13:33:20.552] _job_complete: JobId=120918 WEXITSTATUS 0 [2025-12-15T13:33:20.552] _job_complete: JobId=120918 done #. Job is NOT missing, when identifying it by jobid myuser@clusterb01:~$ sacct -j 120918 --starttime=2025-12-12 -o "jobidraw,jobid,node,start,end,elapsed,state,submitline%30" JobIDRaw JobID NodeList Start End Elapsed State SubmitLine ------------ ------------ --------------- ------------------- ------------------- ---------- ---------- ------------------------------ 120918 120918 dgx29 2025-12-15T13:31:28 2025-12-15T13:33:20 00:01:52 COMPLETED srun --pty bash 120918.0 120918.0 dgx29 2025-12-15T13:31:28 2025-12-15T13:33:20 00:01:52 COMPLETED srun --pty bash #. Job IS *missing* when using -a myuser@clusterb01:~$ sacct -a --starttime=2025-12-12 -o "jobidraw,jobid,node,start,end,elapsed,state,submitline%30" | grep -i 120918 ## *MISSING* Best regards, Lee

1 0

Build 22.05.8 for Debian Trixie?
by Steffen Grunewald 15 Dec '25

15 Dec '25

Good afternoon, we're still running our HPC cluster on Debian 12 Bookworm, which comes with Slurm 22.05.8 - no issues so far. Yesterday, another machine was added as a submit node (slurm-client only), with Debian 13 Trixie / Slurm 24.11.5. Now on that the commands (sinfo etc.) don't work and the slurmctld logs "Incompatible versions of client and server code". Since upgrading the whole cluster is not an option yet: Has someone attempted (and possibly succeeded) to build 22.05.8 (or 23.11.4? which IIRC would be the last compatible one) for Trixie, and would be willing to share ideas? Thanks, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

3 3

SchedMD joins forces with NVIDIA
by Danny Auble 15 Dec '25

15 Dec '25

Today marks a new journey in the history of SchedMD - we are joining forces with NVIDIA. NVIDIA is committed to developing and supporting Slurm as the open-source vendor-neutral workload manager it has always been. For more details, please see NVIDIA's blog post: https://blogs.nvidia.com/blog/nvidia-acquires-schedmd/ Danny

1 0

Slurm 25.05 and 25.11 release video, slides from recent presentations
by Tim Wickberg 12 Dec '25

12 Dec '25

The release video covering what's new in Slurm 25.05 and 25.11 is online now on the SchedMDSlurm YouTube channel: https://youtu.be/gk42j2_kxFg The is the same content that we presented at the Slurm Community Birds-of-a-Feather session at SC'25 last month, but hopefully in a more broadly accessible location. The slides are in the publication archive as usual, as well as the other recent presentations from SC'25 and KubeCon NA 2025: https://www.schedmd.com/publications/ The direct link to the release sides is: http://slurm.schedmd.com/SC25/Slurm_BoF_SC25.pdf - Tim -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

GPU usage in fair share factor
by Massimo Sgaravatto 12 Dec '25

12 Dec '25

Dear all I have these settings in my slurm.conf: PriorityType=priority/multifactor PriorityDecayHalfLife=7-0 PriorityFavorSmall=NO PriorityMaxAge=10-0 PriorityWeightAge=100000 PriorityWeightFairshare=1000000 PriorityWeightJobSize=100000 PriorityWeightPartition=100000 PriorityWeightQOS=100000 PriorityFlags=ACCRUE_ALWAYS,FAIR_TREE AccountingStorageTRES=gres/gpu,gres/gpu:nvidia-h100,gres/gpu:nvidia-l40s If I have got it right, with these settings, when calculating the Job_priority (I am referring to the formula in https://slurm.schedmd.com/priority_multifactor.html) the fair-share_factor is calculated only considering cores*seconds. So if a job also used some GPUs, this is not taken into account. Am I right ? If I want to take memory and GPUs into account as well (with different weights for different GPU models), my understanding is that I should use the TRESBillingWeights attribute, setting e.g: TRESBillingWeights="CPU=1.0,Mem=0.25G,gres/gpu=nvidia-h100=10.0,gres/gpu=nvidia-l40s=5.0" Is this correct ? Thanks, Massimo

2 4

Problems with salloc
by Gestió Servidors 10 Dec '25

10 Dec '25

Hello, I am getting some problems with "salloc" execution. My cluster is running SLURM v23.11.0. I'm going to explain problems: 1. My first problem is that until some previous version (I can't remember what version exactly), when I ran a "salloc", I got a console inside the assigned host. Then, I could execute whatever I wanted. However, now, after execute "salloc" (without forcing host), SLURM always tries to assign me the last host where a job has been executed. But if that host is now powered off, salloc waits and waits but not tries to connect to other host 2. Second problem is that previous execution gives me a host (the host is powered on), my session continues in the allocation host, not in the assigned host. So if I want to execute some program, I need to run with "srun". Is there anyway to go directly to the assigned host? 3. Third problem is a question. How can I limit the hosts where I (as cluster admin) allow to run a "salloc"? For example, I would like that when a user runs a "salloc", it were execute always in a partition with "x" hosts, avoiding SLURM to assign me a host of the list of hosts. Thanks.

1 0

Invalid generic resource (gres) specification after RMA
by Lee 04 Dec '25

04 Dec '25

Good afternoon, I have a cluster that is managed by Base Command Manager (v10) and it has several Nvidia DGXs. dgx09 is a problem child. The entire node was RMA'd and now it no longer behaves the same as my other DGXs. I think the below symptoms are caused by a single underlying issue. *Symptoms : * 1. When I look at our 8 non-MIG DGXs, via `scontrol show node=dgxXY | grep Gres`, 7/8 DGXs report "Gres=gpu:*H100*:8(S:0-1)" while dgx09 reports " Gres=gpu:*h100*:8(S:0-1)" 2. When I submit a job to this node, I get : $ srun --reservation=g09_test --gres=gpu:2 --pty bash srun: error: Unable to create step for job 105035: Invalid generic resource (gres) specification ### No job is running on the node, yet AllocTRES shows consumed resources... $ scontrol show node=dgx09 | grep -i AllocTRES *AllocTRES=gres/gpu=2* ### dgx09 : /var/log/slurmd contains no information ### slurmctld shows : root@h01:# grep 105035 /var/log/slurmctld [2025-11-13T07:44:56.380] sched: _slurm_rpc_allocate_resources JobId=105035 NodeList=dgx09 usec=3420 [2025-11-13T07:44:56.482] _job_complete: JobId=105035 WTERMSIG 1 [2025-11-13T07:44:56.483] _job_complete: JobId=105035 done *Configuration : * 1. gres.conf : # This section of this file was automatically generated by cmd. Do not edit manually! # BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE AutoDetect=NVML NodeName=dgx[01,02] Name=gpu Type=1g.20gb Count=32 AutoDetect=NVML NodeName=dgx[03-10] Name=gpu Type=h100 Count=8 AutoDetect=NVML # END AUTOGENERATED SECTION -- DO NOT REMOVE 2. grep NodeName slurm.conf root@h01:# grep NodeName slurm.conf NodeName=dgx[01,02] RealMemory=2063937 Boards=1 SocketsPerBoard=2 CoresPerSocket=56 ThreadsPerCore=2 MemSpecLimit=30017 Gres=gpu:1g.20gb:32 Feature=location=local NodeName=dgx03-10] RealMemory=2063937 Boards=1 SocketsPerBoard=2 CoresPerSocket=56 ThreadsPerCore=2 MemSpecLimit=30017 Gres=gpu:H100:8 Feature=location=local 3. What slurmd detects on dgx09 root@dgx09:~# slurmd -C NodeName=dgx09 CPUs=224 Boards=1 SocketsPerBoard=2 CoresPerSocket=56 ThreadsPerCore=2 RealMemory=2063937 UpTime=8-00:39:10 root@dgx09:~# slurmd -G slurmd: gpu/nvml: _get_system_gpu_list_nvml: 8 GPU system device(s) detected slurmd: Gres Name=gpu Type=h100 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-55 CoreCnt=224 Links=-1,0,0,0,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-55 CoreCnt=224 Links=0,-1,0,0,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0-55 CoreCnt=224 Links=0,0,-1,0,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=0-55 CoreCnt=224 Links=0,0,0,-1,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=56-111 CoreCnt=224 Links=0,0,0,0,-1,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=5 ID=7696487 File=/dev/nvidia5 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,-1,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=6 ID=7696487 File=/dev/nvidia6 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,0,-1,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML slurmd: Gres Name=gpu Type=h100 Count=1 Index=7 ID=7696487 File=/dev/nvidia7 Cores=56-111 CoreCnt=224 Links=0,0,0,0,0,0,0,-1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML *Questions : * 1. As far as I can tell, dgx09 is identical to all my non-MIG DGX nodes in terms of configuration and hardware. Why does scontrol report it having 'h100' with a lower case 'h' unlike the other dgxs which report with an upper case 'H'? 2. Why is dgx09 not accepting GPU jobs and afterwards it artificially thinks that there are GPUs allocated even though no jobs are on the node? 3. Are there additional tests / configurations that I can do to probe the differences between dgx09 and all my other nodes? Best regards, Lee

6 12

Doubts Regarding Definition of Metrics AveRSS and AveDiskWrite
by Manuel G. Marciani 03 Dec '25

03 Dec '25

Hello, fellow users: I have been using Slurm for the past three years, but recently I bumped into a doubt. I am using Slurm's (version 23.02.7) collected metrics (jobacct_gather/linux) to do a performance analysis of an application. I have read the documentation regarding the metrics (https://slurm.schedmd.com/sacct.html) but still find the Ave* metrics confusing, and more specifically the AveRSS and AveDiskWrite. AveDiskWrite is defined as "Average number of bytes written by all tasks in job." So, if I double the workload, which say that it had x avediskwrite, I should observe 2x. So far it is what I observed. But, then, if I double the resources while maintaining the workload I observe again x, and not 2x. So my suspicion is that the metric is the sum of written bytes across time, then divided by the number of nodes. But then with AveRSS, defined as "Average resident set size of all tasks in job," I observe what I expected with AveDiskWrite. That is, that this metric scales with the workload irrespective of the resources it has available. So I am not sure what the "Ave" references here. I would be thankful if someone could clarify the behavior, and even more grateful if someone could point me where in the code these metrics are aggregated and processed to be stored in the database. Many thanks, Manu. -- Manuel G. Marciani - マルシアニ·マヌエル First Stage Reseacher at Computational Earth Sciences (CES) - Earth Sciences Department Barcelona Supercomputing Center - Centro Nacional de Supercomputación Ph.D Student at Departament d'Arquitectura de Computadors (DAC) - Facultat d'Informàtica de Barcelona (FIB) Universitat Politècnica de Catalunya (UPC) BSC building Plaça Eusebi Güell, 1-3, 08034 Barcelona, Spain Desk BSC-PL0-6-20/8 mail to: manuel.gimenez(a)bsc.es mail to: manuel.gimenez.de.castro(a)upc.edu mail to: manuel.marciani(a)a.riken.jp mail to: manuel.gimenez.de.castro.marciani(a)hu-berlin.de

1 0

MIG and eBPF issues (Slurm 24.11.6)
by wk5ng＠uwaterloo.ca 28 Nov '25

28 Nov '25

Hi all, I'm having some trouble getting Slurm 24.11.6 to work with MIG, and the slurmd logs seem to point to an issue with eBPF. For some context, this is an LXD unprivileged container where I'm trying to get MIG to work with Slurm. Other compute nodes without MIG work fine and isolate the GPUs accordingly. What I'm seeing in slurmd logs: [2025-11-24T23:32:50.197] [331.interactive] cgroup/v2: cgroup_p_constrain_apply: CGROUP: EBPF Closing and loading bpf program into /sys/fs/cgroup/system.slice/slurmstepd.scope/job_331 [2025-11-24T23:32:50.197] [331.interactive] error: load_ebpf_prog: BPF load error (Operation not permitted). Please check your system limits (MEMLOCK). I've tried increasing the system limits for MEMLOCK by setting DefaultLimitMEMLOCK=infinity in /etc/systemd/system.conf, and I've copied my slurmd.service file below where I've set Delegate=yes and LimitMEMLOCK=infinity. Previously only Delegate=yes wasn't set (I had rifled through the cgroupv2 documentaton for Slurm and found that setting), but in both cases I see the same BPF load error. Just wondering if this was something that other people had come across before and maybe I'm doing something silly here. I've checked that my slurm.conf has the corresponding parameters set according to Slurm's own documentation for cgroup.conf and my cgroup.conf is also copied below. Some portion of the gres.conf is also copied below, and even though I tried AutoDetect=nvml for this node, it's still doesn't work, which was why I changed to manually setting it based off the output of slurmd -G. Maybe I should try switching back to cgroupv1 and see if that helps fix things, but I'm not sure at this point if MIG and Slurm are compatible using cgroupv1. I can send other parts of logs, configuration files etc. Any help would be greatly appreciated! ###### slurmd.service file [Unit] Description=Slurm node daemon After=network.target munge.service ConditionPathExists=/etc/slurm/slurm.conf [Service] Type=forking EnvironmentFile=-/etc/sysconfig/slurmd ExecStart=/usr/sbin/slurmd -d /usr/sbin/slurmstepd $SLURMD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID PIDFile=/var/run/slurmd.pid KillMode=process LimitNOFILE=51200 Delegate=yes LimitMEMLOCK=infinity LimitSTACK=infinity [Install] WantedBy=multi-user.target ###### cgroup.conf CgroupPlugin=autodetect ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ##### gres.conf NodeName=gpu-3 AutoDetect=nvml Name=gpu NodeName=gpu-4 Name=gpu MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap30,/dev/nvidia-caps/nvidia-cap31

3 4

Slinky version 1.0.0 is now available
by Marrlow Warnicke 20 Nov '25

20 Nov '25

We are pleased to announce the availability of Slinky version 1.0.0! Slinky is SchedMD's set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.0.0: https://github.com/SlinkyProject/slurm-operator/tree/release-1.0 New features include hybrid support, integration points for external tooling, and workload protection and isolation. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge handles the cases where you want Slurm scheduling on your cluster and to be able to run either Kubernetes or Slurm jobs. Release v1.0.0: https://github.com/SlinkyProject/slurm-bridge/tree/release-1.0 New features include support for DRA Extended Resources, support for TaintToleration and VolumeBinding plugins, and integration of new Slurm 25.11 support for granular node resource allocation assignments with GRES. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… The SlinkyProject registry now has containers that support both amd64 (x86_64) and arm64 (aarch64) architectures. You may find these here: https://github.com/orgs/SlinkyProject/packages Apologies for the redundant email-we needed to clarify the release version. -- Marlow Warnicke Principal Cloud Engineer Commercial Slurm - and Slinky - Development and Support

1 0

Slinky version 1.0.0 is now available
by Marrlow Warnicke 20 Nov '25

20 Nov '25

We are pleased to announce the availability of Slinky version 1.0.0-rc1. Slinky is SchedMD’s set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-operator/tree/release-1.0 New features include hybrid support, integration points for external tooling, and workload protection and isolation. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge handles the cases where you want Slurm scheduling on your cluster and to be able to run either Kubernetes or Slurm jobs. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-bridge/tree/release-1.0 New features include support for DRA Extended Resources, support for TaintToleration and VolumeBinding plugins, and integration of new Slurm 25.11 support for granular node resource allocation assignments with GRES. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… The SlinkyProject registry now has containers that support both amd64 (x86_64) and arm64 (aarch64) architectures. You may find these here: https://github.com/orgs/SlinkyProject/packages -- Marlow Warnicke Principal Cloud Engineer, SchedMD LLC Commercial Slurm - and Slinky - Development and Support

1 0

Recommended Stable Slurm Version for >100P Scale Clusters
by KK 18 Nov '25

18 Nov '25

We are currently planning to deploy a new HPC system with a total compute capacity exceeding 100 PF. As part of our preparation, we would like to understand which Slurm versions are considered stable and widely used at this scale. Could you please share your recommendations or experience regarding: 1. Which Slurm version is currently running reliably on very large-scale clusters (>100 PF or >10k nodes)? 2. Whether there are any versions we should avoid due to known issues at large scale. 3. Any best practices or configuration considerations for Slurm deployments of this size.

3 2

Slurm versions 25.05.5 and 24.11.7 are now available
by Marshall Garey 11 Nov '25

11 Nov '25

We are pleased to announce the availability of Slurm versions 25.05.5 and 24.11.7. These releases fix some mild to moderate severity issues and potential crashes. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md https://github.com/SchedMD/slurm/blob/slurm-24.11/CHANGELOG/slurm-24.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Split node into 2 partitions
by Ratnasamy, Fritz 11 Nov '25

11 Nov '25

Hi, Is there a straightforward way to split a gpu node with 8 GPUs into 2 partitions with 4 GPUs on each? How do we edit the slurm.conf and gres.conf in this case? Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Slurm Community Survey 2025
by Tim Wickberg 10 Nov '25

10 Nov '25

With the Slurm 25.11 release out, and the 1.0.0rc1 for the Slinky Project done, we're quickly shifting into conference season. SchedMD staff are presenting at KubeCon North America on Slinky [1] this week. We don't have a booth here, but feel free to say hi if you see any of us, or send me a message on the Kubernetes or CNCF Slack channels (@wickberg) if you have something you'd like to discuss in person. Next week we'll be manning the Slurm Booth at SC25 [2], as well as hosting the annual Slurm Community Birds-of-a-Feather session [3] on Thursday from 12:15-1:15pm. For this year we wanted to to make the Slurm Community BoF survey available ahead of time, and to open it to a wider audience. This'll let us prepare some initial results ahead of the BoF (while still trying to update live data during the BoF), and the results from this are invaluable as we plan for future Slurm releases. The survey is available now: https://schedmd.com/survey For those not at SC25, we'll have a brief set of highlights from this survey included in the Slurm 25.11 release overview video on our YouTube channel in December. - Tim [1] https://kccncna2025.sched.com/event/27FW5/ [2] The Slurm Booth is #1641. We have a new halo banner this year as well. [3] https://sc25.conference-program.com/presentation/?id=bof101&sess=sess471 [4] https://www.youtube.com/SchedMDSlurm -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

sreport and AccountUtilizationByUser
by Gestió Servidors 08 Nov '25

08 Nov '25

Hello, I woud like to get a full report that includes CPU usage of an account (detalied by user). I have run "sreport cluster AccountUtilizationByUser start=9/1 end=11/2 account=mygr" and the output is this: -------------------------------------------------------------------------------- Cluster/Account/User Utilization 2025-09-01T00:00:00 - 2025-11-01T23:59:59 (5360400 secs) Usage reported in CPU Minutes -------------------------------------------------------------------------------- Cluster Account Login Proper Name Used Energy --------- --------------- --------- --------------- --------- -------- mycluster mygr 3979368 0 mycluster mygr mygr-1 3979368 0 mycluster mygr mygr-10 3979368 0 mycluster mygr mygr-11 3979368 0 mycluster mygr mygr-12 3979368 0 mycluster mygr mygr-13 3979368 0 mycluster mygr mygr-14 3979368 0 mycluster mygr mygr-15 3979368 0 mycluster mygr mygr-16 3979368 0 mycluster mygr mygr-17 3979368 0 mycluster mygr mygr-18 3979368 0 mycluster mygr mygr-19 3979368 0 mycluster mygr mygr-2 3979368 0 mycluster mygr mygr-20 3979368 0 mycluster mygr mygr-21 3979368 0 mycluster mygr mygr-22 3979368 0 mycluster mygr mygr-23 3979368 0 mycluster mygr mygr-24 3979368 0 mycluster mygr mygr-25 3979368 0 mycluster mygr mygr-26 3979368 0 mycluster mygr mygr-27 3979368 0 mycluster mygr mygr-28 3979368 0 mycluster mygr mygr-29 3979368 0 mycluster mygr mygr-3 3979368 0 mycluster mygr mygr-30 3979368 0 mycluster mygr mygr-31 3979368 0 mycluster mygr mygr-32 3979368 0 mycluster mygr mygr-33 3979368 0 mycluster mygr mygr-34 3979368 0 mycluster mygr mygr-35 3979368 0 mycluster mygr mygr-36 3979368 0 mycluster mygr mygr-37 3979368 0 mycluster mygr mygr-38 3979368 0 mycluster mygr mygr-39 3979368 0 mycluster mygr mygr-4 3979368 0 mycluster mygr mygr-40 3979368 0 mycluster mygr mygr-5 3979368 0 mycluster mygr mygr-6 3979368 0 mycluster mygr mygr-7 3979368 0 mycluster mygr mygr-8 3979368 0 mycluster mygr mygr-9 3979368 0 I don't understand why column "Used" shows always the same value, because each user "mygr-XX" has used the cluster in differents ways, times and each used has submitted very different number of jobs, as you can see in this "sshare" output: [root@login bin]# sshare -l -A mygr Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS -------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- mygr 1 0.250000 3941705 0.337402 0.961033 0.260137 mygr mygr-1 1 0.024390 32099 0.002748 0.008144 0.095146 2.995008 mygr mygr-10 1 0.024390 98343 0.008418 0.024949 0.067961 0.977586 mygr mygr-11 1 0.024390 213838 0.018304 0.054250 0.052427 0.449588 mygr mygr-12 1 0.024390 134149 0.011483 0.034033 0.056311 0.716657 mygr mygr-13 1 0.024390 89304 0.007644 0.022656 0.071845 1.076537 mygr mygr-14 1 0.024390 524836 0.044925 0.133150 0.040777 0.183179 mygr mygr-15 1 0.024390 303494 0.025979 0.076996 0.044660 0.316774 mygr mygr-16 1 0.024390 99020 0.008476 0.025121 0.066019 0.970897 mygr mygr-17 1 0.024390 296341 0.025366 0.075181 0.046602 0.324421 mygr mygr-18 1 0.024390 37809 0.003236 0.009592 0.085437 2.542696 mygr mygr-19 1 0.024390 72158 0.006177 0.018306 0.077670 1.332337 mygr mygr-2 1 0.024390 225658 0.019316 0.057249 0.050485 0.426039 mygr mygr-20 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-21 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-22 1 0.024390 115042 0.009847 0.029186 0.062136 0.835683 mygr mygr-23 1 0.024390 54169 0.004637 0.013743 0.081553 1.774775 mygr mygr-24 1 0.024390 125767 0.010765 0.031907 0.058252 0.764422 mygr mygr-25 1 0.024390 243088 0.020808 0.061671 0.048544 0.395490 mygr mygr-26 1 0.024390 35516 0.003040 0.009010 0.089320 2.706902 mygr mygr-27 1 0.024390 65281 0.005588 0.016562 0.079612 1.472682 mygr mygr-28 1 0.024390 118203 0.010118 0.029988 0.060194 0.813335 mygr mygr-29 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-3 1 0.024390 107543 0.009206 0.027284 0.064078 0.893956 mygr mygr-30 1 0.024390 33946 0.002906 0.008612 0.091262 2.832099 mygr mygr-31 1 0.024390 183059 0.015670 0.046442 0.054369 0.525180 mygr mygr-32 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-33 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-34 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-35 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-36 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-37 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-38 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-39 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-4 1 0.024390 37605 0.003219 0.009541 0.087379 2.556486 mygr mygr-40 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-5 1 0.024390 81114 0.006943 0.020578 0.073786 1.185232 mygr mygr-6 1 0.024390 91873 0.007864 0.023308 0.069903 1.046428 mygr mygr-7 1 0.024390 80513 0.006892 0.020426 0.075728 1.194082 mygr mygr-8 1 0.024390 367822 0.031485 0.093316 0.042718 0.261374 mygr mygr-9 1 0.024390 33630 0.002879 0.008532 0.093204 2.858713 mygr mygr-tutor 1 0.024390 40472 0.003464 0.010268 0.083495 2.375427 What am I doing wrong? What I need to get is a detalied report from all "mygr" users (mygr account) from CPU usage and, if it is possible, other values. Thanks.

2 1

Slinky version 1.0.0rc1 is now now available
by Marrlow Warnicke 07 Nov '25

07 Nov '25

We are pleased to announce the availability of Slinky version 1.0.0-rc1. Slinky is SchedMD’s set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-operator/tree/release-1.0 New features include hybrid support, integration points for external tooling, and workload protection and isolation. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge handles the cases where you want Slurm scheduling on your cluster and to be able to run either Kubernetes or Slurm jobs. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-bridge/tree/release-1.0 New features include support for DRA Extended Resources, support for TaintToleration and VolumeBinding plugins, and integration of new Slurm 25.11 support for granular node resource allocation assignments with GRES. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… The SlinkyProject registry now has containers that support both amd64 (x86_64) and arm64 (aarch64) architectures. You may find these here: https://github.com/orgs/SlinkyProject/packages -- Marlow Warnicke Principal Cloud Engineer, SchedMD LLC Commercial Slurm - and Slinky - Development and Support

1 0

submit host DNS lookup errors stall slurmctld, fail nodes, kill jobs
by moorehfl＠amazon.com 06 Nov '25

06 Nov '25

As we've scaled up our slurm usage, we've noticed that short, moderate bursts of DNS lookup failures are enough to regularly stall slurmctld: _xgetaddrinfo: getaddrinfo(runner-t35knco5d-project-54-concurrent-0:37251) failed ...this has a cascading effect where, when stalled, the controlled can't always communicate with nodes: error: Error connecting, bad data: family = 0, port = 0 ...and immediately the controller will mark the nodes as unhealthy, and kill jobs: slurmctld: Killing JobId=3120751 on failed node slurm-0f6cacdc1 The reason for the DNS failures is not unreliable DNS server or network, but rather the jobs are submitted by containers that don't have resolvable hostnames. This traditionally hasn't disrupted functionality, but we've noticed if 8-10 jobs all terminate at the same time (submitter container SIGTERMs the srun process) that the controller can be easily overloaded for several seconds, despite having significant free system resources. gdb confirms the process is hanging on DNS. We also can see "Socket timed out on send/recv operation" from clients attempting to interact with the controller during the issue. slurm 24.11.0 RHEL 8.10 kernel 4.18.0-553.58.1.el8_10.x86_64 We're looking into ways to get our ephemeral job submitter containers resolvable in DNS to prevent lookup failures (either by giving them resolvable hostnames, or by blackholing the records to 0.0.0.0 to allow for fast local failure on the slurmctld server). However, it does seem unusual for a handful of bad DNS lookups to cause so much disruption in slurmctld. Is this a known weak point of ctld? The slurmctld host is a single-purpose 16 vCPU 30GB EC2 instance with minimal load. We have ~150 nodes, all nodes have valid IPs in slurm.conf to remove the need for ctld to perform lookups for nodes, but apparently there is still a need to lookup the submit host as well, and we can reliably reproduce these cascading failures. Another possiblity might be to extend SlurmdTimeout to something very long and hope that the controller recovers from its stall in enough time to prevent from marking nodes as unhealthy and killing jobs, but it's not clear if that will have any effect since the first occurrence of "error: Error connecting, bad data: family = 0, port = 0" immediately drains nodes and kills jobs. Thanks

1 0

Slurm version 25.11 is now available
by Marshall Garey 06 Nov '25

06 Nov '25

We are pleased to announce the availability of Slurm 25.11. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md The Slurm documentation has also been updated to the 25.11 release: https://slurm.schedmd.com Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

CPU frequency setting not configured for this node
by Felix 06 Nov '25

06 Nov '25

Hello I get this error: "CPU frequency setting not configured for this node" in some of my node. I do not know what to do, can you please advice. Thank you Felix -- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

1 0

Move to higher qos when cluster idle
by Ratnasamy, Fritz 06 Nov '25

06 Nov '25

Hi, Is there an option to automatically have users move to a higher qos when the cluster is x% idle? I was reading about customizing the job_submit.lua script in order to do that but I was wondering if it makes sense. We have a gpu cluster and many users complain they can not allocate more than their current qos when the cluster is not very busy. Best, *Fritz Ratnasamy*Data Scientist Information Technology

1 0

Slurm release candidate version 25.11.0rc1 is available for testing
by Tim Wickberg 28 Oct '25

28 Oct '25

We are pleased to announce the availability of Slurm release candidate 25.11.0rc1. To highlight some new features coming in 25.11: * Added new "Expedited Requeue" mode for batch jobs. Batch jobs with --requeue=expedite will automatically requeue on node failure, or if the batch script returns a non-zero exit code and one or more Epilog scripts fail. Expedited requeue jobs are eligible to restart immediately, are treated as the highest priority job in the system, and their previously allocated set of nodes will be prevented from launching other work. * Added a new "Mode 3" of operation to Hierarchical Resources. This mode complements the existing Mode 1 and Mode 2 by summing usage from lower levels automatically. This can be used, e.g., to implement a power-capping mode modeling power distribution between the datacenter, local distribution, and individual racks. * Added direct support for exporting OpenMetrics (Prometheus) telemetry from slurmctld. This is accessible on SlurmctldPort on SlurmctldHost by default, or can be disabled if desired. * Added an experimental asynchronous-reply mode to slurmctld. If enabled with "SlurmctldParameters=enable_async_reply", RPC responses are offloaded to the kernel for further processing, freeing individual worker threads for new traffic. This is the first release candidate of the upcoming 25.11 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 25.11.x version and we will address them before the first production 25.11.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/download-slurm/. The changelog for 25.11 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-25.11.md -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

AuthInfo broken in 25.05.1 ?
by michael＠mayer.cx 27 Oct '25

27 Oct '25

I am using the AuthInfo in both slurm.conf and slurmdbd.conf to point to a different location for the munge socket. This works as expected in slurmdbd, but it does not work for slurmctld. In slurmctld it seems to ignore this setting and try to locate the munge socket at the default location and then fail. When using the same approach in 23.11.11, for example, AuthInfo is working as expected for both slurmctld and slurmdbd. Sample config from slurm.conf and slurmdbd.conf AuthType=auth/munge AuthInfo=socket=/my/non-default/location/of/munge.socket I'd be grateful for any pointer whether this is a genuine bug/regression or not. Many thanks, Michael.

3 3

2025

2024