- slurm-users - lists.schedmd.com

Can't clear the REASON
by Berg, Stephen P CIV USN NRL DET SSC MS (USA) 17 Apr '26

17 Apr '26

Just upgraded my grid to 25.11 a couple days ago. Got two nodes that were down for a bit and when I set them to resume and set the reason="" they are still showing up with the reason I had set while they were down. Nothing I do seems to unset that reason flag for these two nodes. I've set the nodes to down with a reason and then back to idle with the blank reason but it never goes back to "none" like the other nodes. Whatever the last reason set was persists. The behavior seems different since I updated to 25.11. Was running the 20.11 available from the epel-8 repos previously.

2 4

Jobs aborting after slurmctld reload on Intel nodes - AMD unaffected
by Pharthiphan Asokan 16 Apr '26

16 Apr '26

Hi team, We’re observing job aborts on Intel-based nodes immediately after a slurmctld reload. AMD nodes remain stable and jobs continue unaffected. No system or Slurm configuration changes were made before the issue started. Error observed: error: Aborting JobID=1288 due to change in socket/core configuration of allocated nodes Relevant node configuration (Intel node example): NodeName=smc-h4-u19 CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=1 RealMemory=510976 State=UNKNOWN Feature=model_SSG-1228-B From logs: error: valid_job_resources: smc-h4-u19 sockets:2:2, cores 64,32 error: Node configuration differs from hardware: CPUs=128:128(hw) Boards=1:1(hw) SocketsPerBoard=2:2(hw) What we’ve verified: * No changes in BIOS, firmware, or hardware topology * No edits to slurm.conf or slurm_nodes.conf * Reload (scontrol reconfigure) triggers job aborts only on Intel nodes * AMD nodes remain intact through reloads

3 2

We are pleased to announce the availability of Slurm version 25.11.5
by Marshall Garey 14 Apr '26

14 Apr '26

We are pleased to announce the availability of Slurm version 25.11.5. Some of the changes in 25.11.0 are highlighted below: * Fix a regression in 25.11.0 that could cause daemons to hang after reaching conmgr_max_connections. * Fix a regression added in 25.05.0 concerning how the slurmctld inherits /run/slurmctld/sack.socket when using AuthType=auth/slurm to prevent clients that connected during a reconfigure from hanging indefinitely. * Fix a race in tree forwarding that could cause a crash during shutdown * Fix several bugs in the namespace/linux plugin that caused instability The full list of changes are available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Why my job can't start (backfill reservation issue)
by Massimo Sgaravatto 14 Apr '26

14 Apr '26

Dear all I (try to) manage a slurm cluster composed by some CPU-only nodes and some worker nodes which have also GPUs: NodeName=cld-ter-[01-06] Sockets=2 CoresPerSocket=96 ThreadsPerCore=2 RealMemory=1536000 State=UNKNOWN NodeName=cld-ter-gpu-[01-05] Sockets=2 CoresPerSocket=96 ThreadsPerCore=2 Gres=gpu:nvidia-h100:4 RealMemory=1536000 State=UNKNOWN The GPU nodes are exposed through multiple partitions: PartitionName=gpus Nodes=cld-ter-gpu-[01-02] State=UP PriorityTier=20 PartitionName=sparch Nodes=cld-ter-gpu-03 AllowAccounts=sparch,operators QoS=sparch State=UP PriorityTier=20 PartitionName=geant4 Nodes=cld-ter-gpu-03 AllowAccounts=geant4,operators QoS=geant4 State=UP PriorityTier=20 PartitionName=enipred Nodes=cld-ter-gpu-04 AllowAccounts=enipred,operators QoS=enipred State=UP PriorityTier=20 PartitionName=enipiml Nodes=cld-ter-gpu-05 AllowAccounts=enipiml,operators QoS=enipiml State=UP PriorityTier=20 We also set a partition to allow cpu-only jobs on the GPU nodes, but these jobs should be preempted (killed and requeued) if jobs submitted to partitions with higher priorities require those resources: PreemptType=preempt/partition_prio PreemptMode=REQUEUE PartitionName=onlycpus-opp Nodes=cld-ter-gpu-[01-05],cld-dfa-gpu-06,btc-dfa-gpu-02 State=UP PriorityTier=10 Now, I don't understand why this job [*] submitted on the onlycpus-opp partition can't start running e.g. on the cld-ter-gpu-01, since it has a lot of free resources: [sgaravat@cld-ter-ui-01 ~]$ scontrol show node cld-ter-gpu-01 NodeName=cld-ter-gpu-01 Arch=x86_64 CoresPerSocket=96 CPUAlloc=8 CPUEfctv=384 CPUTot=384 CPULoad=5.93 AvailableFeatures=(null) ActiveFeatures=(null) Gres=gpu:nvidia-h100:4 NodeAddr=cld-ter-gpu-01 NodeHostName=cld-ter-gpu-01 Version=25.11.3 OS=Linux 5.14.0-611.45.1.el9_7.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 1 05:56:53 EDT 2026 RealMemory=1536000 AllocMem=560000 FreeMem=1192357 Sockets=2 Boards=1 State=MIXED+PLANNED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpus,onlycpus-opp BootTime=2026-04-09T10:39:35 SlurmdStartTime=2026-04-09T10:40:01 LastBusyTime=2026-04-09T11:54:46 ResumeAfterTime=None CfgTRES=cpu=384,mem=1500G,billing=839,gres/gpu=4,gres/gpu:nvidia-h100=4 AllocTRES=cpu=8,mem=560000M,gres/gpu=4,gres/gpu:nvidia-h100=4 CurrentWatts=0 AveWatts=0 I guess the "MIXED+PLANNED" is the answer, but as far as I can see only a job (283469) is planned for this worker node: sgaravat@cld-ter-ui-01 ~]$ squeue --start | grep ter-gpu-01 JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON) 283469 gpus vllm-pod ciangott PD 2026-04-13T14:31:40 1 cld-ter-gpu-01 (Resources) But job 283469 doesn't require too many resources [**], so the 2 jobs could run together. Why job 283534 can't start ? Any hints ? Thanks, Massimo [*] [sgaravat@cld-ter-ui-01 ~]$ scontrol show job=283534 JobId=283534 JobName=myscript.sh UserId=sgaravat(5008) GroupId=tbadmin(5001) MCS_label=N/A Priority=542954 Nice=0 Account=operators QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:41 TimeLimit=1-00:00:00 TimeMin=N/A SubmitTime=2026-04-13T11:10:13 EligibleTime=2026-04-13T11:10:13 AccrueTime=2026-04-13T11:10:13 StartTime=2026-04-13T11:58:39 EndTime=2026-04-14T11:58:39 Deadline=N/A PreemptEligibleTime=2026-04-13T11:58:39 PreemptTime=None SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-13T11:58:39 Scheduler=Backfill Partition=onlycpus-opp AllocNode:Sid=cld-ter-ui-01:3035857 ReqNodeList=(null) ExcNodeList=(null) NodeList=btc-dfa-gpu-02 BatchHost=btc-dfa-gpu-02 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=100G,node=1,billing=26 AllocTRES=cpu=1,mem=100G,node=1,billing=26 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null) Command=/shared/home/sgaravat/myscript.sh SubmitLine=sbatch myscript.sh WorkDir=/shared/home/sgaravat StdErr=/shared/home/sgaravat/JOB-myscript.sh.283534.4294967294.err StdIn=/dev/null StdOut=/shared/home/sgaravat/JOB-myscript.sh.283534.4294967294.out MailUser=massimo.sgaravatto(a)pd.infn.it MailType=INVALID_DEPEND,BEGIN,END,FAIL,REQUEUE,STAGE_OUT [**] sgaravat@cld-ter-ui-01 ~]$ scontrol show job=283469 JobId=283469 JobName=vllm-pod UserId=ciangott(6054) GroupId=tbuser(6000) MCS_label=N/A Priority=499703 Nice=0 Account=cms QOS=normal JobState=PENDING Reason=Resources Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A SubmitTime=2026-04-13T06:48:37 EligibleTime=2026-04-13T06:48:37 AccrueTime=2026-04-13T06:48:37 StartTime=2026-04-13T14:31:40 EndTime=2026-04-14T14:31:40 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-13T11:59:48 Scheduler=Main Partition=gpus AllocNode:Sid=cld-ter-ui-01:3015801 ReqNodeList=(null) ExcNodeList=(null) NodeList= SchedNodeList=cld-ter-gpu-01 NumNodes=1-1 NumCPUs=32 NumTasks=1 CPUs/Task=32 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=32,mem=190734M,node=1,billing=118,gres/gpu=2,gres/gpu:nvidia-h100=2 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=32 MinMemoryNode=190734M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null) Command=.interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.slurm SubmitLine=sbatch .interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.slurm WorkDir=/shared/home/ciangott StdErr= StdIn=/dev/null StdOut=/shared/home/ciangott/.interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.out TresPerNode=gres/gpu:nvidia-h100:2 TresPerTask=cpu=32

5 10

pam_slurm_adopt
by Ratnasamy, Fritz 10 Apr '26

10 Apr '26

Hi, We have blocked ssh to nodes where users do not have a running job. However, when there is a running job, a user can ssh to that node and use more than the slurm resources allocated on that node. Am I missing something? I usually request an interactive session of 2GB. I then open another terminal and ssh to the node where the interactive session runs. I allocated a 4GB R variable on the node where I just ssh but this does not error out. Additionally, if I end my interactive session, the ssh connection still lives which I believe should not happen. Best, *Fritz Ratnasamy*Data Scientist Information Technology

7 6

Moving a Slurm database from one slurm database host to another
by James A Allsopp 09 Apr '26

09 Apr '26

Hi, We'd like to move our Slurm database from its existing machine running mariadb-server-5.5.68 to another running mariadb-server-10.5.29-3, both of which are running slurm 24.11.7. We dumped the database successfully from the old server, reimported it into the new server using mysqldump and *mysql < db_dump.sql*. The new slurmdbd service was then started. The clusters have different names so there were a few inconsistencies, but when we started the new slurm cluster, we could see the jobs according to sacct (using the -M option) matched those of the original cluster, but none of the associated job steps were shown. I manually logged into the new database and could see the jobs in the job_table but it looked like the foreign key link had been broken in the step table as there were entries in that table. Am I missing some information about how to export and import properly? I was wondering if the import had triggered auto-increment on keys and that was breaking things? Ideally, we'd like to change the name of the cluster as well. Is that just a matter of altering the prefix on the relevant tables, which is something I can do by hand on the database dump before reimporting? For instance, changing the table <old_name>_event_table to <new_cluster_name>_event_table. Thanks James -- Dr James Allsopp | Research Platforms team IT Services | University of Sheffield I do not work Tuesday and Wednesday 3-5pm. I also send emails at odd times, but respond at your convenience.

1 0

How to delete my defaultwckey ?
by Faraz Hussain 07 Apr '26

07 Apr '26

I want every submitted job to have some value for the wckey, i.e: #SBATCH --wckey=myproject I made the appropriate changes to slurm.conf and slurmdb.conf and it works great. I can track how many hours people are using with those wckeys. But now I want to make it mandatory to use a wckey. To do that I need to delete the default wckey associated with the user's account. I tried doing it as follows, but it still lets me submit jobs without a wckey. It probably thinks I have an "empty" default wckey. sacctmgr mod user fhussa set defaultwckey= [root@mas01 ~]# sacctmgr list user fhussa format=user,defaultwckey User Def WCKey ---------- ---------- fhussa

1 0

Difference between sreport and sacct
by hany.ibrahim＠kit.edu 07 Apr '26

07 Apr '26

Hello, I am trying to understand how sreport work. So I run sreport and obtain th estatistics about speciic project. On teh other side, I collected the jobs related to this account using sacct. When I try to sum the numbers, sometimes it don't sum up. I can't see any reservations. This is the sreport I used: sreport --tres cpu cluster AccountUtilizationByUser Start=2021-01-01 account=<account> -t hours -np This is jobs I collected by using sacct: sudo -u slurm sacct -a -S "2021-01-01" -E NOW --format=JobID%100,User,Account%100,State,AllocCPUs,CPUTime%100,CPUTIMERAW%100,ElapsedRaw%100,AllocTres%100,AllocNodes,NodeList,NNodes --noheader --parsable2 | grep <account> I tried to exclude the steps from the main Jobs, so I summed only the main jobs whatever its state (canceled, completed, timeout, fail..). However the sreport occasionally is larger. What I am missing exactly? Warm regards, Hany.

1 0

Shared queue: how to request full node with all resources?
by Dustin Lang 31 Mar '26

31 Mar '26

Hi, With a partition with "OverSubscribe=FORCE" set, is there a way to request all the node resources? I see "--mem=0" does that for memory. But I do not see an option to request all the CPUs and GRES/TRES such as GPUs. I tried "--nodes=1 --ntasks=1 --cpus-per-task=0", but "--cpus-per-task=0" does not do the same thing as "--mem=0". In other words, is it possible to have a queue where nodes can be shared between jobs, but have a simple way for an sbatch script to request the full node with all its memory, cpus, and other resources? We have a heterogeneous cluster, so telling users (or jupyterhub scripts) to list exactly the resources they want doesn't really work. thanks, dustin

3 4

How do you handle GPU node failures during long jobs?
by Antonio Jose Alonso-Stepanov 26 Mar '26

26 Mar '26

Hi all, I'm a Stanford CS student looking into how sites handle GPU node failures during long-running jobs. A couple questions: When a GPU node goes down mid-job, do most sites use Slurm's requeue or --no-kill to handle it, or is it mostly manual drain and resubmit? Is anyone using HealthCheckProgram to catch GPU issues (like ECC errors via DCGM), or do you handle GPU health monitoring outside of Slurm? Curious what's worked and what hasn't. Thanks. Antonio

3 2