July 2026 - slurm-users - lists.schedmd.com

sreport changes between Slurm 24 and 26?
by Kamil Wilczek 20 Jul '26

20 Jul '26

Dear All, I recently upgraded my cluster to Slurm 26.05.2 (a completely fresh installation, including the operating system), replacing a previous Slurm 24.11 installation. On the previous installation (Slurm 24.11), I used sreport from a Python script to retrieve TRES usage per QOS: slurm_sreport = subprocess.run( [ "sreport", "--tres=cpu,gres/gpu", "cluster", "AccountUtilizationByQOS", "Start=2025-01-01", "End=now", "Accounts=mim", "Format=QOS,TresName,Used", f"Users={os.environ['USER']}", "--noheader", "--parsable2" ], ... The script generated a nice report for each user who ran it. The "Users=" parameter was respected, and the command returned only QOSes associated with the calling user. With Slurm 26.05.2, however, Users= appears to be ignored, and the command returns the complete list of QOSes in the cluster instead. Is this a confirmed change in behaviour? Kind regards, -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

1 0

Dummy Slurm jobs?
by Grigory Shamov 18 Jul '26

18 Jul '26

HI All, Is it possible, or have it been done, to create dummy Slurm jobs that would start, block the resource, execute prologue and epilogue, but never execute the batch script body? The idea is to start something in prologue, like a user/job/credential-specific instance of a proxy (say, nginx) controlled/scheduled by Slurm, but without giving users ability to run an arbitrary script in the job's script body. Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

3 2

Slurm versions 26.05.2 and 25.11.7 are now available
by Tim McMullan 14 Jul '26

14 Jul '26

We are pleased to announce the availability of Slurm versions 26.05.2 and 25.11.7. These releases fix some mild to moderate severity issues and potential crashes. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Tim McMullan Senior System Software Engineer NVIDIA

1 0

Cluster ID changed in Slurm DB
by Burian, John 10 Jul '26

10 Jul '26

I was working on adding some partitions to our test cluster to test preemption. This is Slurm 24.11.1, we have a single slurmctld and slurmdbd in this cluster. I’m using partition based preemption, not QOS based preemption. I added a QOS to use with the preempting partition to enforce per-job resource limits, and then added the new partitions to slurm.conf. It took me many tries to get the slurm.conf correct, so many attempts to start slurmctld that failed due to conf file parsing errors. However, once I got the slurm.conf correct, slurmctld would not start, complaining that the cluster ID in the saved state information was wrong: [2026-07-02T14:55:38.573] fatal: CLUSTER ID MISMATCH. slurmctld has been started with "ClusterID=3013" from the state files in StateSaveLocation, but the DBD thinks it should be "3391". Running multiple clusters from a shared StateSaveLocation WILL CAUSE CORRUPTION. Remove /export/slurm/state/clustername to override this safety check if this is intentional. The clustername file in the save state location shows the cluster ID 3013. In the Slurm DB, in clusters_table, I see the row for my test cluster shows the (wrong) ID that appears in the logs, 3391, and the mod_time field has a timestamp in the same time frame as when I was doing the work on adding the QOS and partition configuration. Any thoughts on what could have caused the Slurm DB to be updated with a different cluster ID than what is saved in state? I’d like to avoid something similar happening when I go to deploy the new partitions in production. John

1 0

Expected behavior for preempt/partition_prio
by Massimo Sgaravatto 10 Jul '26

10 Jul '26

Dear all I have these settings in slurm.conf: . ... PreemptType=preempt/partition_prio PreemptMode=REQUEUE PartitionName=gpus-opp Nodes=btc-ter-gpu-[03-05],btc-dfa-gpu-06,btc-dfa-gpu-02 State=UP PriorityTier=15 PartitionName=mpicms Nodes=btc-ter-03,btc-ter-gpu-05 AllowAccounts=cms,operators State=UP PriorityTier=20 .... My understanding was that the expected behavior is that a job submitted to the gpus-opp partition is always preempted (requeued) if the resources allocated by this job are requested by a job submitted to the mpicms partition (since 20 > 15) Did I get it right? I am asking because I see that sometimes this doesn't happen. I am still trying to debug the issue, but it looks like also the QoS is taken into account (and this doesn't make sense to me unless I am missing something) Thanks, Massimo

3 3

Slinky 1.2.0 released
by Marlow Warnicke 02 Jul '26

02 Jul '26

We are pleased to announce the availability of Slinky version 1.2.0! Slinky is SchedMD (now part of NVIDIA)'s set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.2.0: https://github.com/SlinkyProject/slurm-operator/releases/tag/v1.2.0 New features include a ScheduledUpdate strategy for NodeSets to control when updates are applied, support for Google GKE A3 Mega instance types, and configurable pod security contexts for operator and webhook deployments. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge brings Kubernetes and Slurm jobs together under one scheduler. Both types of workloads share the same cluster resources. Release v1.2.0: https://github.com/SlinkyProject/slurm-bridge/releases/tag/v1.2.0 New features include support for LeaderWorkerSet workloads introduced in Kubernetes 1.37, priority annotations for workloads to control Slurm job scheduling order, and dynamic topology reconciliation for external and hybrid Slurm nodes. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… #kubernetes #slurm #dra #ai #training #inference #hpc #nvidia #gpu #cloudnative #slinky

1 0