- slurm-users - lists.schedmd.com

Can I update when jobs are running?
by Gould, Ron (GRC-VBA0)[AEGIS] 09 Feb '26

09 Feb '26

The tl;dr is “This is my first upgrade since inheriting this Cluster, so I’m not sure what can or can’t be running during the upgrades.”. My Cluster is running an old version, 22.05.3. This is my first upgrade since inheriting the Cluster. As such, I’d like to install 22.05.4 because it’s a short jump, and it fixes the bug my users are seeing. The Cluster is composed of mostly Oracle Linux 8. I’m aware that I can upgrade within the two release compatibility window. I’ve read through the Upgrade guide and I’m unclear if downtime is required. Perhaps I’m unifying downtime requirements across different SLURM services where I should be interpreting that certain services have their own downtime requirements. https://slurm.schedmd.com/upgrades.html<https://slurm.schedmd.com/upgrades.html#procedure> In the Upgrade Procedure section<https://slurm.schedmd.com/upgrades.html#procedure>, there’re a couple questionable things. 1. Is downtime required? Does downtime == “all jobs must be halted”? “Downtime”, to me, seems like nothing should be running. This statement indicates that jobs can be running during the upgrade. Before considering the upgrade complete, wait for all jobs that were already running to finish. Any jobs started before the slurmd system was upgraded will be running with the old version of slurmstepd, so starting another upgrade or trying to use new features in the new version may cause problems. within a few paragraphs<https://slurm.schedmd.com/upgrades.html#downtime>, this message indicates I will need downtime: Refer to the expected downtime guidance in the following sections for each relevant Slurm daemon Further in the guide, in SLURMD (COMPUTE NODES)<https://slurm.schedmd.com/upgrades.html#slurmd>, I read Upgrades will not interrupt running jobs as long as SlurmdTimeout is not reached during the process This implies, at least, that existing running jobs can stay running. 2. There’re conflicting suggestions of using “rpm” to install the RPMs I built with “rpmbuild". Should I use “dnf localinstall ./*.rpm”? I’m inferring that dependencies will (not) be handled correctly. NOTE: If RPM/DEB packages are used, all packages present on each system must be upgraded together instead of piecewise. … Avoid using low-level package managers like rpm or dpkg as they may not properly enforce these dependencies However, in SLURMDBD (ACCOUNTING)<https://slurm.schedmd.com/upgrades.html#slurmdbd>, this statement Upgrade the slurmdbd daemon binaries, libraries, and its systemd unit file (if used). If using RPM/DEB packages, the package manager will take care of these indicates I should be using RPM packages. Lastly, to get to a current install, I need to step through multiple versions, with the condition that jobs started with a specific major version must finish within the compatibility window. GitLab has a tool where you plug in your current and intended versions and it tells you explicitly which versions are required along the upgrade path. I’d like a similarly explicit tool for SLURM, but I infer from the Compatibility Window<https://slurm.schedmd.com/upgrades.html#compatibility_window> that I can update like so: 1. Current = 22.05.3 2. 23.11 3. 25.05 4. 26.05 That feels like a big leapfrog between versions. I’d like the practice of upgrading. Is there any detriment to upgrading at a slower pace: 1. Current = 22.05.3 2. 22.05.11 3. 23.02.8 4. 23.11.11 5. 24.05.8 6. 24.11.7 7. 25.05.6 8. 25.11.2

8 24

for srun jobs, if we try to get UserCPU time by sacct. it always shows 0.
by squallabc＠126.com 09 Feb '26

09 Feb '26

Hi we have a slurm v25.05.2 setup. we found any srun jobs are always shows 0 on UserCPU . for sbatch jobs shoiws normal. is this expected or it is a bug ? Thanks. sacct -j 2948 --format=AveCPU,UserCPU AveCPU UserCPU ---------- ---------- 00:00:00 00:02:00 00:00:00

2 1

Adding FPGAs for Slurm Tracking
by Raven Hallsby 08 Feb '26

08 Feb '26

Hey all, We have a very heterogeneous cluster with a mix of Intel & AMD machines, GPUs, and FPGAs. We use Slurm to allocate time on the CPUs right now, but do not manage the GPU or FPGAs with Slurm yet. Some of our work involves hardware/SoC design, and we have been using Firesim (https://fires.im/) for a good amount of our work. Firesim is a bunch of Python that sits on top of invoking programs to provide "seamless" functionality and orchestration. However, Firesim was designed for AWS where a single user controls all of the resources (FPGAs in our case) used for the simulation. We have gone through the effort to pull out the important bits from Firesim to write a shell script that we can execute locally to achieve the same thing as Firesim' orchestration, without the need for an orchestrator. We have two FPGAs in two separate but identical machines and multiple people that want to use them for very long periods of time (days or longer) and want to let people freely move between the two FPGAs. All of our other machines in the cluster use Slurm, so we are looking to have Slurm multiplex these FPGA jobs. We already have the NFS stuff set up, so shared disk is fine. The important thing in our desired Slurm setup is that no user is allocated the same FPGA at the same time for the same job (you cannot have multiple FPGA jobs running the same design in parallel, partly because of bitstreams, partly because of disk images). Slurm does not track FPGAs by default. I have looked into how to teach Slurm about the FPGAs and know that I need to tell Slurm about the FPGAs as a GRES through gres.conf. I just want to confirm that I have the correct steps below: 1. Mark the 2 FPGAs as GRES with gres.conf. What should they be marked as? Do I need anything special? 2. Then enable tracking on the FPGA GRES to make a TRES in slurm.conf. What should this be labeled as? I see other groups using GresTypes=gpu in their configuration. Would that apply here? Since FPGA usage is exclusive already, I could see the GPU GRES type applying. 3. Then the TRES can be used by a QoS to enforce limits? The other important question, will I need to write a Slurm plugin (GRES or otherwise) to manage and detect the FPGAs? I saw the files stuff for GRES-es, but want to be sure. Writing a plugin is not a big deal for us, but it is future maintenance that I would want to document.

1 0

Re: Dynamic MIG Question
by Davide DelVento 06 Feb '26

06 Feb '26

Aaron (or anyone else), Did you manage to get Dynamic MIG working in Slurm? I'm actually surprised that after these many years SchedMD has not implemented this feature yet, especially now that newer GPUs allow MIG repartitioning without being root. The only mention of this in their ticketing system is at https://support.schedmd.com/show_bug.cgi?id=11091#c8 (and subsequent c10) which say that it's not on their roadmap, but that was 5 years ago. I have heard that some users manage dynamic changes by draining nodes, running scripts to reconfigure MIG via nvidia-smi, bringing the node back and then submitting the job. Anybody here has tried that and with what success? I speculate that now that NVIDIA owns SchedMD perhaps this feature will be at a higher priority, but maybe not? Anybody knows anything about it and is not bound by an NDA to keep mum? Thanks On Wed, Nov 22, 2023 at 1:22 PM Davide DelVento <davide.quantum(a)gmail.com> wrote: > I assume you mean the sentence about dynamic MIG at > https://slurm.schedmd.com/gres.html#MIG_Management > Could it be supported? I think so, but only if one of their paying > customers (that could be you) asks for it. > > On Wed, Nov 22, 2023 at 11:24 AM Aaron Kollmann < > aaron.kollmann(a)student.hpi.de> wrote: > >> Hello All, >> >> I am currently working in a research project and we are trying to find >> out whether we can use NVIDIAs multi-instance GPU (MIG) dynamically in >> SLURM. >> >> For instance: >> >> - a user requests a job and wants a GPU but none is available >> >> - now SLURM will reconfigure a MIG GPU to create a partition (e.g. >> 1g.5gb) which becomes available and allocated immediately >> >> I can already reconfigure MIG + SLURM within a few seconds to start jobs >> on newly partitioned resources, but Jobs get killed when I restart slurmd >> on nodes with a changed MIG config. (see script example below) >> >> *Do you think it is possible to develop a plugin or change SLURM to the >> extent that dynamic MIG will be supported one day? * >> >> (The website says it is not supported) >> >> >> >> Best >> >> - Aaron >> >> >> >> >> #!/usr/bin/bash >> >> # Generate Start Config >> killall slurmd >> killall slurmctld >> nvidia-smi mig -dci >> nvidia-smi mig -dgi >> nvidia-smi mig -cgi 19,14,5 -i 0 -C >> nvidia-smi mig -cgi 0 -i 1 -C >> cp -f ./slurm-19145-0.conf /etc/slurm/slurm.conf >> slurmd -c >> slurmctld -c >> sleep 5 >> >> # Start a running and a pending job (the first job gets killed by slurm) >> srun -w gx06 -c 2 --mem 1G --gres=gpu:a100_1g.5gb:1 sleep 300 & >> srun -w gx06 -c 2 --mem 1G --gres=gpu:a100_1g.5gb:1 sleep 300 & >> sleep 5 >> >> # Simulate MIG Config Change >> nvidia-smi mig -i 1 -dci >> nvidia-smi mig -i 1 -dgi >> nvidia-smi mig -cgi 19,14,5 -i 1 -C >> cp -f ./slurm-2x19145.conf /etc/slurm/slurm.conf >> killall slurmd >> killall slurmctld >> slurmd >> slurmctld >> >

2 3

Can we avoid jobs near wall time from being preempted?
by iazaceta＠vicomtech.org 06 Feb '26

06 Feb '26

We have an HPC where the average job length is measured in days, not hours. Users are careful to add checkpoints to their jobs but even in that case, preempting a job that is close to its walltime (max: 14 days) can be very disruptive. I checked what options preemption offers but none seem to protect jobs near their finishing line. PreemptExempTime ensures a minimum job runtime and GraceTime allows for a grace time period after the job has been selected for preemption. Is there anything I am missing to achieve what I want? Thank you!

2 1

systemd unit ordering for dbd host
by Michael Gutteridge 05 Feb '26

05 Feb '26

We have a host dedicated to slurmdbd and mysql (Ubuntu 24.04, Slurm 24.11.4, mysql 8.0.45). A couple days ago unattended-upgrades upgraded the MySQL package which triggered a restart of mysql.service which looks to have caused some errors in slurmdbd. I don't think those errors were permanent, everything looks to be running as per usual currently. Enough backstory- the question is should we configure the slurmdbd systemd unit to stop slurmdbd before mysql is stopped? Currently there are minimal dependencies built into the unit: After=network-online.target remote-fs.target munge.service mysql.service mysqld.service mariadb.service sssd.service Wants=network-online.target But nothing to tell systemd to take down slurmdbd when the mysql unit is stopped. Is there *any* reason we'd want slurmdbd running if mysql wasn't? Are there any problems I'd cause by creating such a dependency? I do plan to exclude mysql from future unattended upgrading, but I'd also like to keep our systemd units sensible. Thanks - Michael

3 3

How to get strigger event information?
by Mauricio Pinto 03 Feb '26

03 Feb '26

Greetings, I am managing a cluster using SLURM, and want to use strigger to automate notifications about certain events like jobs or nodes failing or changing states, slurmctld failing, etc. I want the notifications I send to include information about the event, like which node, which state, which job ID, and so on. However, so far I haven't found documentation on how to get these values. I did find a related thread <https://groups.google.com/g/slurm-users/c/NhrlL32q0Io/m/HKVd9l1yDgAJ> in the google groups archive. One of the replies includes a script which uses environment variables such as $SLURM_JOB_ID and $SLURM_JOB_NAME. How are these environment variables available? Is there documentation on which environment variables are available to strigger scripts? I'd appreciate any guidance you can provide. On a separate note, I just joined this mailing list and haven't seen any community guidelines, preferred email format, etc. Please feel free to point them out to me or make suggestions if you have any. Thanks, Mauricio

2 2

Configless Slurm Error: failed to fetch remote configs
by Xaver Stiensmeier 30 Jan '26

30 Jan '26

Hey everyone, in the past we set up clusters with configs on each node. Now we want to explore configless. Without changing anything else, we therefore followed: https://slurm.schedmd.com/configless_slurm.html and added 'enable_configless' in the config on the master: SlurmctldParameters=cloud_dns,idle_on_node_suspend,enable_configless,reconfig_on_restart and start each worker's slurmd with the conf-server parameter: # Override systemd service to set conditional path [Service] ExecStart= ExecStart=/usr/sbin/slurmd --conf-server=master However, this leads to: slurmd: error: _fetch_child: failed to fetch remote configs: Protocol authentication error slurmd: error: _establish_configuration: failed to load configs. Retrying in 10 seconds. on the workers and on the master (/var/log/slurm/slurmctld) to: [2026-01-16T10:00:06.681] error: Munge decode failed: Invalid credential [2026-01-16T10:00:06.681] auth/munge: _print_cred: ENCODED: Thu Jan 01 00:00:00 1970 [2026-01-16T10:00:06.681] auth/munge: _print_cred: DECODED: Thu Jan 01 00:00:00 1970 [2026-01-16T10:00:06.681] error: slurm_unpack_received_msg: [[worker]:24295] auth_g_verify: REQUEST_CONFIG has authentication error: Unspecified error [2026-01-16T10:00:06.681] error: slurm_unpack_received_msg: [[worker]:24295] Protocol authentication error The munge key setup is the same as before so I don't think there is anything wrong with it unless something changes with configless (slurm.conf): AuthType=auth/munge CryptoType=crypto/munge AuthAltTypes=auth/jwt AuthAltParameters=jwt_key=/etc/slurm/jwt-secret.key I found https://groups.google.com/g/slurm-users/c/Q7FVkhx-bOs but this seems unrelated as both can talk fine with each other: worker:~$ nc -zv master 6817 Connection to master (192.168.20.169) 6817 port [tcp/*] succeeded! I tried adding more "-v" to the slurmd start, but that did not give more information. I am unsure how to debug this further. Somehow I think it must be a munge issue, but I am confused as this part hasn't changed. Best regards, Xaver

2 6

Nodes removed from topology after suspend in 25.11.2
by Franky Backeljauw 29 Jan '26

29 Jan '26

Hi all We have upgraded to version 25.11.2 and are now noticing that nodes are removed getting removed from the topology/tree when the node gets suspended. In the slurmctld.log we see things like this: [2026-01-29T23:31:32.517] POWER: power_save: suspending nodes node1 [2026-01-29T23:32:37.536] debug2: _update_node_features_post: nodes node1 active features set to: feature1 [2026-01-29T23:32:37.536] debug2: topology/tree: topology_p_add_rm_node: topology_p_add_rm_node: remove node1 from switch1 If we later on run a job with: srun -w node1 --pty bash then we see the following: [2026-01-30T00:08:04.004] debug2: NodeSet for JobId=1234 [2026-01-30T00:08:04.005] _pick_best_nodes: JobId=1234 never runnable in partition part1 [2026-01-30T00:08:04.005] sched: schedule: JobId=1234 non-runnable: Requested topology configuration is not available [2026-01-30T00:08:04.005] debug2: _schedule: setting JobId=1234 to "BadConstraints" (Requested topology configuration is not available) This worked without any issues in the previous version 25.05.2, so something has change in the newer version 25.11.2. My guess is something with dynamic nodes and/or active versus available features, but it’s just not clear :-( Ideas anyone??? Our setup includes in slurm.conf (trimmed): TopologyPlugin = topology/tree NodeName=node1 Features=feature1 NodeSet=set1 Feature=feature1 PartitionName=part1 Nodes=set1 [...] SuspendProgram = /power_save/suspend.sh ResumeProgram = /power_save/resume.sh ReconfigFlags = KeepPowerSaveSettings In topology.conf: SwitchName=switch1 Nodes=node1 The SuspendProgram and ResumeProgram scripts use tuned-adm to change the power profile (to powersave and hpc-compute resp.). When looking at the node itself, after suspend, it shows (trimmed): NodeName=node1 AvailableFeatures=feature1 ActiveFeatures=feature1 State=IDLE+POWERED_DOWN PARTITION=part1 I hope someone can shed a light on this issue... Thanks in advance for any reply! -- Kind regards Franky

1 0

sacct 25.11.2: --state filter returns no results
by Norman Gaywood 26 Jan '26

26 Jan '26

Hi all, I have recently upgraded my cluster to Slurm 25.11.2 (running on Rocky Linux 9). slurm-25.11.2-1.el9.x86_64 built locally. This time I built accounting in and so I'm new to sacct. When I run sacct without a state filter, I see the jobs correctly marked as COMPLETED: sacct -a -S 2026-01-26 --format=JobID,State | grep COMPLETED | head -10 9 COMPLETED 9.batch COMPLETED 9.extern COMPLETED 9.0 COMPLETED 39 COMPLETED 39.batch COMPLETED 39.extern COMPLETED 39.0 COMPLETED 40 COMPLETED 40.batch COMPLETED When I attempt to filter for those exact states (using either the short or long flag), the output is empty: sacct -a -S 2026-01-26 --format=JobID,State --state=completed JobID State ------------ ---------- (no output beyond the heading) Am I using it wrong? Cheers, Norm.

2 2

2026

2025

2024

slurm-users