- slurm-users - lists.schedmd.com

Can we avoid jobs near wall time from being preempted?
by iazaceta＠vicomtech.org 06 Feb '26

06 Feb '26

We have an HPC where the average job length is measured in days, not hours. Users are careful to add checkpoints to their jobs but even in that case, preempting a job that is close to its walltime (max: 14 days) can be very disruptive. I checked what options preemption offers but none seem to protect jobs near their finishing line. PreemptExempTime ensures a minimum job runtime and GraceTime allows for a grace time period after the job has been selected for preemption. Is there anything I am missing to achieve what I want? Thank you!

2 1

systemd unit ordering for dbd host
by Michael Gutteridge 05 Feb '26

05 Feb '26

We have a host dedicated to slurmdbd and mysql (Ubuntu 24.04, Slurm 24.11.4, mysql 8.0.45). A couple days ago unattended-upgrades upgraded the MySQL package which triggered a restart of mysql.service which looks to have caused some errors in slurmdbd. I don't think those errors were permanent, everything looks to be running as per usual currently. Enough backstory- the question is should we configure the slurmdbd systemd unit to stop slurmdbd before mysql is stopped? Currently there are minimal dependencies built into the unit: After=network-online.target remote-fs.target munge.service mysql.service mysqld.service mariadb.service sssd.service Wants=network-online.target But nothing to tell systemd to take down slurmdbd when the mysql unit is stopped. Is there *any* reason we'd want slurmdbd running if mysql wasn't? Are there any problems I'd cause by creating such a dependency? I do plan to exclude mysql from future unattended upgrading, but I'd also like to keep our systemd units sensible. Thanks - Michael

3 3

How to get strigger event information?
by Mauricio Pinto 03 Feb '26

03 Feb '26

Greetings, I am managing a cluster using SLURM, and want to use strigger to automate notifications about certain events like jobs or nodes failing or changing states, slurmctld failing, etc. I want the notifications I send to include information about the event, like which node, which state, which job ID, and so on. However, so far I haven't found documentation on how to get these values. I did find a related thread <https://groups.google.com/g/slurm-users/c/NhrlL32q0Io/m/HKVd9l1yDgAJ> in the google groups archive. One of the replies includes a script which uses environment variables such as $SLURM_JOB_ID and $SLURM_JOB_NAME. How are these environment variables available? Is there documentation on which environment variables are available to strigger scripts? I'd appreciate any guidance you can provide. On a separate note, I just joined this mailing list and haven't seen any community guidelines, preferred email format, etc. Please feel free to point them out to me or make suggestions if you have any. Thanks, Mauricio

2 2

Configless Slurm Error: failed to fetch remote configs
by Xaver Stiensmeier 30 Jan '26

30 Jan '26

Hey everyone, in the past we set up clusters with configs on each node. Now we want to explore configless. Without changing anything else, we therefore followed: https://slurm.schedmd.com/configless_slurm.html and added 'enable_configless' in the config on the master: SlurmctldParameters=cloud_dns,idle_on_node_suspend,enable_configless,reconfig_on_restart and start each worker's slurmd with the conf-server parameter: # Override systemd service to set conditional path [Service] ExecStart= ExecStart=/usr/sbin/slurmd --conf-server=master However, this leads to: slurmd: error: _fetch_child: failed to fetch remote configs: Protocol authentication error slurmd: error: _establish_configuration: failed to load configs. Retrying in 10 seconds. on the workers and on the master (/var/log/slurm/slurmctld) to: [2026-01-16T10:00:06.681] error: Munge decode failed: Invalid credential [2026-01-16T10:00:06.681] auth/munge: _print_cred: ENCODED: Thu Jan 01 00:00:00 1970 [2026-01-16T10:00:06.681] auth/munge: _print_cred: DECODED: Thu Jan 01 00:00:00 1970 [2026-01-16T10:00:06.681] error: slurm_unpack_received_msg: [[worker]:24295] auth_g_verify: REQUEST_CONFIG has authentication error: Unspecified error [2026-01-16T10:00:06.681] error: slurm_unpack_received_msg: [[worker]:24295] Protocol authentication error The munge key setup is the same as before so I don't think there is anything wrong with it unless something changes with configless (slurm.conf): AuthType=auth/munge CryptoType=crypto/munge AuthAltTypes=auth/jwt AuthAltParameters=jwt_key=/etc/slurm/jwt-secret.key I found https://groups.google.com/g/slurm-users/c/Q7FVkhx-bOs but this seems unrelated as both can talk fine with each other: worker:~$ nc -zv master 6817 Connection to master (192.168.20.169) 6817 port [tcp/*] succeeded! I tried adding more "-v" to the slurmd start, but that did not give more information. I am unsure how to debug this further. Somehow I think it must be a munge issue, but I am confused as this part hasn't changed. Best regards, Xaver

2 6

Nodes removed from topology after suspend in 25.11.2
by Franky Backeljauw 29 Jan '26

29 Jan '26

Hi all We have upgraded to version 25.11.2 and are now noticing that nodes are removed getting removed from the topology/tree when the node gets suspended. In the slurmctld.log we see things like this: [2026-01-29T23:31:32.517] POWER: power_save: suspending nodes node1 [2026-01-29T23:32:37.536] debug2: _update_node_features_post: nodes node1 active features set to: feature1 [2026-01-29T23:32:37.536] debug2: topology/tree: topology_p_add_rm_node: topology_p_add_rm_node: remove node1 from switch1 If we later on run a job with: srun -w node1 --pty bash then we see the following: [2026-01-30T00:08:04.004] debug2: NodeSet for JobId=1234 [2026-01-30T00:08:04.005] _pick_best_nodes: JobId=1234 never runnable in partition part1 [2026-01-30T00:08:04.005] sched: schedule: JobId=1234 non-runnable: Requested topology configuration is not available [2026-01-30T00:08:04.005] debug2: _schedule: setting JobId=1234 to "BadConstraints" (Requested topology configuration is not available) This worked without any issues in the previous version 25.05.2, so something has change in the newer version 25.11.2. My guess is something with dynamic nodes and/or active versus available features, but it’s just not clear :-( Ideas anyone??? Our setup includes in slurm.conf (trimmed): TopologyPlugin = topology/tree NodeName=node1 Features=feature1 NodeSet=set1 Feature=feature1 PartitionName=part1 Nodes=set1 [...] SuspendProgram = /power_save/suspend.sh ResumeProgram = /power_save/resume.sh ReconfigFlags = KeepPowerSaveSettings In topology.conf: SwitchName=switch1 Nodes=node1 The SuspendProgram and ResumeProgram scripts use tuned-adm to change the power profile (to powersave and hpc-compute resp.). When looking at the node itself, after suspend, it shows (trimmed): NodeName=node1 AvailableFeatures=feature1 ActiveFeatures=feature1 State=IDLE+POWERED_DOWN PARTITION=part1 I hope someone can shed a light on this issue... Thanks in advance for any reply! -- Kind regards Franky

1 0

sacct 25.11.2: --state filter returns no results
by Norman Gaywood 26 Jan '26

26 Jan '26

Hi all, I have recently upgraded my cluster to Slurm 25.11.2 (running on Rocky Linux 9). slurm-25.11.2-1.el9.x86_64 built locally. This time I built accounting in and so I'm new to sacct. When I run sacct without a state filter, I see the jobs correctly marked as COMPLETED: sacct -a -S 2026-01-26 --format=JobID,State | grep COMPLETED | head -10 9 COMPLETED 9.batch COMPLETED 9.extern COMPLETED 9.0 COMPLETED 39 COMPLETED 39.batch COMPLETED 39.extern COMPLETED 39.0 COMPLETED 40 COMPLETED 40.batch COMPLETED When I attempt to filter for those exact states (using either the short or long flag), the output is empty: sacct -a -S 2026-01-26 --format=JobID,State --state=completed JobID State ------------ ---------- (no output beyond the heading) Am I using it wrong? Cheers, Norm.

2 2

opposite of --array
by Herbert Fruchtl 21 Jan '26

21 Jan '26

Folks, When I use sacct, it always shows all tasks of an array job individually. This is supposed to be switched on by --array, but it seems to be the default. How do I switch it off? Thanks in advance, Herbert -- Herbert Fruchtl (he/him) Senior Scientific Computing Officer / HPC Administrator School of Chemistry, IT Services University of St Andrews -- The University of St Andrews is a charity registered in Scotland: No SC013532

2 3

building EL10 packages
by Brian Andrus 17 Jan '26

17 Jan '26

Hopefully someone has already busted through this wall... Currently trying to build slurm for the EL10 distros. Just tried 25.11.2 (been trying .1 as well) straight from tarball to rpms. The issue trying to build so far is that there seems to be no http-parser package in the repos (trying rocky 10 and centos stream 10). Ok, so I disable slurmrestd and rpmbuild runs. It still warns about http-parser missing, but packages come out. Now, when I try to run slurmctld/slurmdbd, I get an error: //usr/sbin/slurmctld: symbol lookup error: /usr/lib64/slurm/auth_slurm.so: undefined symbol: jwt_Base64encode / This is even on the build system... I've tried enabling developer mode even, with no luck. Starting to wonder if there is a breaking change in the EL10 distro line. Anyone have insight here? Thanks in advance, Brian Andrus

3 3

Best practice instructions and packages for Debian/Ubuntu full-upgrade?
by Steffen Grunewald 15 Jan '26

15 Jan '26

Good morning, long story: Every two years, some time after a release of Debian or Ubuntu LTS, when evaluations have ended and configurations adapted or rewritten and some reasonable maintenance date has been agreed upon by users' representatives (and directors!) and admins, we're in the situation to perform a full-upgrade (dist-upgrade) of all the machines that keep important data (while the compute nodes just get a fresh install). With the old 9-month release cycle of Slurm, in the past there was a slim chance that the upgrade from the version provided with the old release to the new release's one was small enough (since 3*9>24), but with the new 6-month cycle (4*6=24) there will always be one version too much in between. For a more or less painless transition, the manager machine running slurmdbd and slurmctld would need an intermediate version - which is not provided by Debian or Ubuntu at the time of the upgrade. Long story shorter: Provided you run Slurm on Debian/Ubuntu: Where do you get that necessary intermediate version from? I'll have to upgrade from 22.05.8-4+deb12u3 (Bookworm) to 24.11.5-4 (Trixie) (while I'm expecting the Slurm developers to urge me to proceed to 25.11.x, I couldn't find *any* deb packages for that, plus it would immediately break Ubuntu LTS integration since Noble is at 23.11.4-1.2ubuntu5). I'd need 23.11 for Debian (Bookworm *or* Trixie) for the half-step upgrade of slurmdbd's databases, and I'd like to bump Ubuntu's version to 24.11 to eventually add our GPU machines to the cluster at a later point. Long story short: Does someone keep those intermediate versions, and maybe builds ahead of the official distro ones? I can't be alone with this problem... Thanks for any pointer, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

4 3

We are pleased to announce the availability of Slurm versions 25.11.2 and 25.05.6.
by Marshall Garey 15 Jan '26

15 Jan '26

We are pleased to announce the availability of Slurm versions 25.11.2 and 25.05.6. Changes in 25.11.2 include fixing potential slurmctld state save file corruption while using the rpc_queue experimental feature, preventing PMIx 5.0.8 and 5.0.9 clients from hanging when connecting to the PMIx server, correctly expanding srun filename format identifiers, and various stability issues. Changes in 25.05.6 include fixing various stability issues. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

2026

2025

2024

slurm-users