January 2026 - slurm-users - lists.schedmd.com

slurmctld keeps segfaulting, possibly during or just after backfill
by Marcus Lauer 21 Jun '26

21 Jun '26

We are running into a problem where slurmctld is segfaulting a few times a day. We had this problem with SLURM 23.11.8 and now with 23.11.10 as well, though the problem only appears on one of the several SLURM clusters we have, and all of them use one of those versions of SLURM. I was wondering if anyone has encountered a similar issue and has any thoughts on how to prevent this. Obviously we use "SchedulerType=sched/backfill" but strangely when I switched to sched/builtin for a while there were still slurmctld segfaults. We also set "SchedulerParameters=enable_user_top,bf_max_job_test=2000". I have tried turning those off but it did not help. I have also tried tweaking several other settings to no avail. Most of the cluster runs Rocky Linux 8.10 (including the slurmctld system) though we still have some Scientific Linux 7.9 compute nodes (we compile SLURM separately for those). Here is the crash-time error from journalctl: Oct 02 06:31:20 our.host.name kernel: sched_agent[2048355]: segfault at 8 ip 00007fec755d7ea8 sp 00007fec6bffe7e8 error 4 in libslurmfull.so[7fec7555a000+1f4000] Oct 02 06:31:20 our.host.name kernel: Code: 48 39 c1 7e 19 48 c1 f8 06 ba 01 00 00 00 48 d3 e2 48 f7 da 48 0b 54 c6 10 48 21 54 c7 10 c3 b8 00 00 00 00 eb da 48 8b 4f 08 <48> 39 4e 08 48 0f 4e 4e 08 49 89 c9 48 83 f9 3f 76 4e ba 40 00 00 Oct 02 06:31:20 our.host.name systemd[1]: Started Process Core Dump (PID 2169426/UID 0). Oct 02 06:31:20 our.host.name systemd-coredump[2169427]: Process 2048344 ( slurmctld) of user 991 dumped core. This is followed by a list of each of the dozen or so related threads. The one which is dumping core is first and looks like this: Stack trace of thread 2048355: #0 0x00007fec755d7ea8 bit_and_not (libslurmfull.so) #1 0x000000000044531f _job_alloc (slurmctld) #2 0x000000000044576b _job_alloc_whole_node_internal (slurmctld) #3 0x0000000000446e6d gres_ctld_job_alloc_whole_node (slurmctld) #4 0x00007fec722e29b8 job_res_add_job (select_cons_tres.so) #5 0x00007fec722f7c32 select_p_select_nodeinfo_set (select_cons_tres.so) #6 0x00007fec756e7dc7 select_g_select_nodeinfo_set (libslurmfull.so) #7 0x0000000000496eb3 select_nodes (slurmctld) #8 0x0000000000480826 _schedule (slurmctld) #9 0x00007fec753421ca start_thread (libpthread.so.0) #10 0x00007fec745f78d3 __clone (libc.so.6) I have run slurmctld with "debug5" level logging and it appears that the error occurs right after backfill considers a large number of jobs. Slurmctld could be failing at the end of backfill or when doing something which happens just after backfill runs. Usually this is the last message before the crash: [2024-09-25T18:39:42.076] slurmscriptd: debug: _slurmscriptd_mainloop: finished If anyone has any thoughts or advice on this that would be appreciated. Thank you. -- Marcus Lauer Systems Administrator CETS Group, Research Support

4 4

Optimizing CPU allocation in Slurm with hyperthreading enabled
by Guillaume COCHARD 02 Mar '26

02 Mar '26

Hello, The Slurm documentation states that "The default allocation method within a node is cyclic allocation (allocate available CPUs in a round-robin fashion across the sockets within a node)." In our case, we are using hyperthreading, which means that if two jobs requesting one CPU each arrive at the same time on an empty node, they will be assigned to the two threads of the same physical core. To improve performance, we would like Slurm to allocate CPUs (in our case, threads) in a way that prioritizes using one thread per physical core first, and only once all cores are occupied, assign the second thread of each core. This would be similar to the Least Loaded Node (LLN) approach, but instead of distributing jobs across nodes, it would distribute them across physical cores (Least Loaded Core). We do not want to disable hyperthreading; we simply want to optimize core usage when a machine is not fully loaded. We also want this to be a cluster-wide setting, and not a user and/or job setting. Our current slurm.conf contains: ConstrainCores = no SelectType = select/cons_tres SelectTypeParameters = CR_CPU_MEMORY And our cgroup.conf contains: ConstrainDevices=yes ConstrainCores=yes Does anyone know how to achieve this? Thanks, Guillaume

3 2

Can I update when jobs are running?
by Gould, Ron (GRC-VBA0)[AEGIS] 11 Feb '26

11 Feb '26

The tl;dr is “This is my first upgrade since inheriting this Cluster, so I’m not sure what can or can’t be running during the upgrades.”. My Cluster is running an old version, 22.05.3. This is my first upgrade since inheriting the Cluster. As such, I’d like to install 22.05.4 because it’s a short jump, and it fixes the bug my users are seeing. The Cluster is composed of mostly Oracle Linux 8. I’m aware that I can upgrade within the two release compatibility window. I’ve read through the Upgrade guide and I’m unclear if downtime is required. Perhaps I’m unifying downtime requirements across different SLURM services where I should be interpreting that certain services have their own downtime requirements. https://slurm.schedmd.com/upgrades.html<https://slurm.schedmd.com/upgrades.html#procedure> In the Upgrade Procedure section<https://slurm.schedmd.com/upgrades.html#procedure>, there’re a couple questionable things. 1. Is downtime required? Does downtime == “all jobs must be halted”? “Downtime”, to me, seems like nothing should be running. This statement indicates that jobs can be running during the upgrade. Before considering the upgrade complete, wait for all jobs that were already running to finish. Any jobs started before the slurmd system was upgraded will be running with the old version of slurmstepd, so starting another upgrade or trying to use new features in the new version may cause problems. within a few paragraphs<https://slurm.schedmd.com/upgrades.html#downtime>, this message indicates I will need downtime: Refer to the expected downtime guidance in the following sections for each relevant Slurm daemon Further in the guide, in SLURMD (COMPUTE NODES)<https://slurm.schedmd.com/upgrades.html#slurmd>, I read Upgrades will not interrupt running jobs as long as SlurmdTimeout is not reached during the process This implies, at least, that existing running jobs can stay running. 2. There’re conflicting suggestions of using “rpm” to install the RPMs I built with “rpmbuild". Should I use “dnf localinstall ./*.rpm”? I’m inferring that dependencies will (not) be handled correctly. NOTE: If RPM/DEB packages are used, all packages present on each system must be upgraded together instead of piecewise. … Avoid using low-level package managers like rpm or dpkg as they may not properly enforce these dependencies However, in SLURMDBD (ACCOUNTING)<https://slurm.schedmd.com/upgrades.html#slurmdbd>, this statement Upgrade the slurmdbd daemon binaries, libraries, and its systemd unit file (if used). If using RPM/DEB packages, the package manager will take care of these indicates I should be using RPM packages. Lastly, to get to a current install, I need to step through multiple versions, with the condition that jobs started with a specific major version must finish within the compatibility window. GitLab has a tool where you plug in your current and intended versions and it tells you explicitly which versions are required along the upgrade path. I’d like a similarly explicit tool for SLURM, but I infer from the Compatibility Window<https://slurm.schedmd.com/upgrades.html#compatibility_window> that I can update like so: 1. Current = 22.05.3 2. 23.11 3. 25.05 4. 26.05 That feels like a big leapfrog between versions. I’d like the practice of upgrading. Is there any detriment to upgrading at a slower pace: 1. Current = 22.05.3 2. 22.05.11 3. 23.02.8 4. 23.11.11 5. 24.05.8 6. 24.11.7 7. 25.05.6 8. 25.11.2

9 26

How to get strigger event information?
by Mauricio Pinto 03 Feb '26

03 Feb '26

Greetings, I am managing a cluster using SLURM, and want to use strigger to automate notifications about certain events like jobs or nodes failing or changing states, slurmctld failing, etc. I want the notifications I send to include information about the event, like which node, which state, which job ID, and so on. However, so far I haven't found documentation on how to get these values. I did find a related thread <https://groups.google.com/g/slurm-users/c/NhrlL32q0Io/m/HKVd9l1yDgAJ> in the google groups archive. One of the replies includes a script which uses environment variables such as $SLURM_JOB_ID and $SLURM_JOB_NAME. How are these environment variables available? Is there documentation on which environment variables are available to strigger scripts? I'd appreciate any guidance you can provide. On a separate note, I just joined this mailing list and haven't seen any community guidelines, preferred email format, etc. Please feel free to point them out to me or make suggestions if you have any. Thanks, Mauricio

2 2

Configless Slurm Error: failed to fetch remote configs
by Xaver Stiensmeier 30 Jan '26

30 Jan '26

Hey everyone, in the past we set up clusters with configs on each node. Now we want to explore configless. Without changing anything else, we therefore followed: https://slurm.schedmd.com/configless_slurm.html and added 'enable_configless' in the config on the master: SlurmctldParameters=cloud_dns,idle_on_node_suspend,enable_configless,reconfig_on_restart and start each worker's slurmd with the conf-server parameter: # Override systemd service to set conditional path [Service] ExecStart= ExecStart=/usr/sbin/slurmd --conf-server=master However, this leads to: slurmd: error: _fetch_child: failed to fetch remote configs: Protocol authentication error slurmd: error: _establish_configuration: failed to load configs. Retrying in 10 seconds. on the workers and on the master (/var/log/slurm/slurmctld) to: [2026-01-16T10:00:06.681] error: Munge decode failed: Invalid credential [2026-01-16T10:00:06.681] auth/munge: _print_cred: ENCODED: Thu Jan 01 00:00:00 1970 [2026-01-16T10:00:06.681] auth/munge: _print_cred: DECODED: Thu Jan 01 00:00:00 1970 [2026-01-16T10:00:06.681] error: slurm_unpack_received_msg: [[worker]:24295] auth_g_verify: REQUEST_CONFIG has authentication error: Unspecified error [2026-01-16T10:00:06.681] error: slurm_unpack_received_msg: [[worker]:24295] Protocol authentication error The munge key setup is the same as before so I don't think there is anything wrong with it unless something changes with configless (slurm.conf): AuthType=auth/munge CryptoType=crypto/munge AuthAltTypes=auth/jwt AuthAltParameters=jwt_key=/etc/slurm/jwt-secret.key I found https://groups.google.com/g/slurm-users/c/Q7FVkhx-bOs but this seems unrelated as both can talk fine with each other: worker:~$ nc -zv master 6817 Connection to master (192.168.20.169) 6817 port [tcp/*] succeeded! I tried adding more "-v" to the slurmd start, but that did not give more information. I am unsure how to debug this further. Somehow I think it must be a munge issue, but I am confused as this part hasn't changed. Best regards, Xaver

2 6

Nodes removed from topology after suspend in 25.11.2
by Franky Backeljauw 29 Jan '26

29 Jan '26

Hi all We have upgraded to version 25.11.2 and are now noticing that nodes are removed getting removed from the topology/tree when the node gets suspended. In the slurmctld.log we see things like this: [2026-01-29T23:31:32.517] POWER: power_save: suspending nodes node1 [2026-01-29T23:32:37.536] debug2: _update_node_features_post: nodes node1 active features set to: feature1 [2026-01-29T23:32:37.536] debug2: topology/tree: topology_p_add_rm_node: topology_p_add_rm_node: remove node1 from switch1 If we later on run a job with: srun -w node1 --pty bash then we see the following: [2026-01-30T00:08:04.004] debug2: NodeSet for JobId=1234 [2026-01-30T00:08:04.005] _pick_best_nodes: JobId=1234 never runnable in partition part1 [2026-01-30T00:08:04.005] sched: schedule: JobId=1234 non-runnable: Requested topology configuration is not available [2026-01-30T00:08:04.005] debug2: _schedule: setting JobId=1234 to "BadConstraints" (Requested topology configuration is not available) This worked without any issues in the previous version 25.05.2, so something has change in the newer version 25.11.2. My guess is something with dynamic nodes and/or active versus available features, but it’s just not clear :-( Ideas anyone??? Our setup includes in slurm.conf (trimmed): TopologyPlugin = topology/tree NodeName=node1 Features=feature1 NodeSet=set1 Feature=feature1 PartitionName=part1 Nodes=set1 [...] SuspendProgram = /power_save/suspend.sh ResumeProgram = /power_save/resume.sh ReconfigFlags = KeepPowerSaveSettings In topology.conf: SwitchName=switch1 Nodes=node1 The SuspendProgram and ResumeProgram scripts use tuned-adm to change the power profile (to powersave and hpc-compute resp.). When looking at the node itself, after suspend, it shows (trimmed): NodeName=node1 AvailableFeatures=feature1 ActiveFeatures=feature1 State=IDLE+POWERED_DOWN PARTITION=part1 I hope someone can shed a light on this issue... Thanks in advance for any reply! -- Kind regards Franky

1 0

sacct 25.11.2: --state filter returns no results
by Norman Gaywood 26 Jan '26

26 Jan '26

Hi all, I have recently upgraded my cluster to Slurm 25.11.2 (running on Rocky Linux 9). slurm-25.11.2-1.el9.x86_64 built locally. This time I built accounting in and so I'm new to sacct. When I run sacct without a state filter, I see the jobs correctly marked as COMPLETED: sacct -a -S 2026-01-26 --format=JobID,State | grep COMPLETED | head -10 9 COMPLETED 9.batch COMPLETED 9.extern COMPLETED 9.0 COMPLETED 39 COMPLETED 39.batch COMPLETED 39.extern COMPLETED 39.0 COMPLETED 40 COMPLETED 40.batch COMPLETED When I attempt to filter for those exact states (using either the short or long flag), the output is empty: sacct -a -S 2026-01-26 --format=JobID,State --state=completed JobID State ------------ ---------- (no output beyond the heading) Am I using it wrong? Cheers, Norm.

2 2

opposite of --array
by Herbert Fruchtl 21 Jan '26

21 Jan '26

Folks, When I use sacct, it always shows all tasks of an array job individually. This is supposed to be switched on by --array, but it seems to be the default. How do I switch it off? Thanks in advance, Herbert -- Herbert Fruchtl (he/him) Senior Scientific Computing Officer / HPC Administrator School of Chemistry, IT Services University of St Andrews -- The University of St Andrews is a charity registered in Scotland: No SC013532

2 3

building EL10 packages
by Brian Andrus 17 Jan '26

17 Jan '26

Hopefully someone has already busted through this wall... Currently trying to build slurm for the EL10 distros. Just tried 25.11.2 (been trying .1 as well) straight from tarball to rpms. The issue trying to build so far is that there seems to be no http-parser package in the repos (trying rocky 10 and centos stream 10). Ok, so I disable slurmrestd and rpmbuild runs. It still warns about http-parser missing, but packages come out. Now, when I try to run slurmctld/slurmdbd, I get an error: //usr/sbin/slurmctld: symbol lookup error: /usr/lib64/slurm/auth_slurm.so: undefined symbol: jwt_Base64encode / This is even on the build system... I've tried enabling developer mode even, with no luck. Starting to wonder if there is a breaking change in the EL10 distro line. Anyone have insight here? Thanks in advance, Brian Andrus

3 3

Best practice instructions and packages for Debian/Ubuntu full-upgrade?
by Steffen Grunewald 15 Jan '26

15 Jan '26

Good morning, long story: Every two years, some time after a release of Debian or Ubuntu LTS, when evaluations have ended and configurations adapted or rewritten and some reasonable maintenance date has been agreed upon by users' representatives (and directors!) and admins, we're in the situation to perform a full-upgrade (dist-upgrade) of all the machines that keep important data (while the compute nodes just get a fresh install). With the old 9-month release cycle of Slurm, in the past there was a slim chance that the upgrade from the version provided with the old release to the new release's one was small enough (since 3*9>24), but with the new 6-month cycle (4*6=24) there will always be one version too much in between. For a more or less painless transition, the manager machine running slurmdbd and slurmctld would need an intermediate version - which is not provided by Debian or Ubuntu at the time of the upgrade. Long story shorter: Provided you run Slurm on Debian/Ubuntu: Where do you get that necessary intermediate version from? I'll have to upgrade from 22.05.8-4+deb12u3 (Bookworm) to 24.11.5-4 (Trixie) (while I'm expecting the Slurm developers to urge me to proceed to 25.11.x, I couldn't find *any* deb packages for that, plus it would immediately break Ubuntu LTS integration since Noble is at 23.11.4-1.2ubuntu5). I'd need 23.11 for Debian (Bookworm *or* Trixie) for the half-step upgrade of slurmdbd's databases, and I'd like to bump Ubuntu's version to 24.11 to eventually add our GPU machines to the cluster at a later point. Long story short: Does someone keep those intermediate versions, and maybe builds ahead of the official distro ones? I can't be alone with this problem... Thanks for any pointer, Steffen -- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~

4 3