- slurm-users - lists.schedmd.com

How do you handle GPU node failures during long jobs?
by Antonio Jose Alonso-Stepanov 14 Mar '26

14 Mar '26

Hi all, I'm a Stanford CS student looking into how sites handle GPU node failures during long-running jobs. A couple questions: When a GPU node goes down mid-job, do most sites use Slurm's requeue or --no-kill to handle it, or is it mostly manual drain and resubmit? Is anyone using HealthCheckProgram to catch GPU issues (like ECC errors via DCGM), or do you handle GPU health monitoring outside of Slurm? Curious what's worked and what hasn't. Thanks. Antonio

1 0

We are pleased to announce the availability of Slurm versions 25.11.4 and 25.05.7.
by Marshall Garey 12 Mar '26

12 Mar '26

We are pleased to announce the availability of Slurm versions 25.11.4 and 25.05.7. Changes in both versions include preventing SIGHUP from terminating slurmrestd. Changes in 25.11.4 include fixing several issues with the switch/nvidia_imex plugin, several memory leaks, a race that could cause jobs to be stuck in the completing state or be requeued and held if the --get-user-env option was used, flex reservations taking nodes from other reservations that do not request full nodes, and a variety of other issues. Changes in 25.05.7 include fixing the gpu/nrt plugin, and fixing some other regressions and stability issues. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Senior System Software Engineer NVIDIA

1 0

Server with two GPUs sharing each GPU to different partitions
by Gestió Servidors 09 Mar '26

09 Mar '26

Hello, First of all, sorry if my question is about something easy, but in my environment, this is the first time I need to do this. I have one server that have 2 identical GPUs. I have two different partitions: one is accessible from some allocation nodes and the other from other allocation nodes. My question is if it's possible to share one GPU with one partition and the other GPU with the other partition. For example, something like this: * gres.conf: NodeName=gpu-node AutoDetect=off Name=gpu Type=RTX5070 File=/dev/nvidia0 Cores=0-127 NodeName=gpu-node AutoDetect=off Name=gpu Type=RTX5070 File=/dev/nvidia1 Cores=128-255 * slurm.conf: NodeName=gpu-node CPUs=256 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=260000 TmpDisk=47000 Gres= RTX5070:2 PartitionName=gpu-int.q Nodes=gpu-node OverSubscribe=No State=UP AllocNodes=submit-node-1 PriorityTier=3 GraceTime=900 PartitionName=gpu-ext.q Nodes=gpu-node OverSubscribe=No State=UP AllocNodes=submit-node-1 PriorityTier=3 GraceTime=900 With this configuration, if simultaneously two different users tries to execute in one of the RTX5070 (one in gpu-int.q and the other in gpu-ext.q), could both of them execute with no problems? If user from gpu-int.q uses device #0 and user from gpu-ext.q uses device #1, could SLURM work without problems if a second simultaneously exection assigns device #1 to user from gpu-int.q and device #0 to user from gpu-ext.q? Thanks.

1 0

Optimizing CPU allocation in Slurm with hyperthreading enabled
by Guillaume COCHARD 02 Mar '26

02 Mar '26

Hello, The Slurm documentation states that "The default allocation method within a node is cyclic allocation (allocate available CPUs in a round-robin fashion across the sockets within a node)." In our case, we are using hyperthreading, which means that if two jobs requesting one CPU each arrive at the same time on an empty node, they will be assigned to the two threads of the same physical core. To improve performance, we would like Slurm to allocate CPUs (in our case, threads) in a way that prioritizes using one thread per physical core first, and only once all cores are occupied, assign the second thread of each core. This would be similar to the Least Loaded Node (LLN) approach, but instead of distributing jobs across nodes, it would distribute them across physical cores (Least Loaded Core). We do not want to disable hyperthreading; we simply want to optimize core usage when a machine is not fully loaded. We also want this to be a cluster-wide setting, and not a user and/or job setting. Our current slurm.conf contains: ConstrainCores = no SelectType = select/cons_tres SelectTypeParameters = CR_CPU_MEMORY And our cgroup.conf contains: ConstrainDevices=yes ConstrainCores=yes Does anyone know how to achieve this? Thanks, Guillaume

3 2

Missing mounts.h in Ubuntu Bionic build
by Michael Gutteridge 25 Feb '26

25 Feb '26

Hi We're running into a problem with building 25.11.3 for Ubuntu Bionic. When the cgroup/v2 plugin is building, ebpf throws an error: ``` ../../../../../src/plugins/cgroup/v2/ebpf.c:38:10: fatal error: linux/mount.h: No such file or directory #include <linux/mount.h> ^~~~~~~~~~~~~~~ compilation terminated. Makefile:638: recipe for target 'ebpf.lo' failed make[6]: *** [ebpf.lo] Error 1 make[6]: Leaving directory '/build/slurm-25.11.3/obj-x86_64-linux-gnu/src/plugins/cgroup/v2' ``` I think this is because on Ubuntu, the `mounts.h` header is in the path `sys`- the manpage for mounts(2) describes including this file via `#include <sys/mounts.h>`. `ebpf.c` seems to be the only place where the `linux/mounts.h` path is used, everywhere else using `sys`. Curiously it seems to build OK on Noble (Ubuntu 24). I have no explanation for the difference. Has anyone run into this problem building on Ubuntu? This include looks to have been a recent add so might be a bug, but I feel like someone else would have run into this already if it were. Thanks - Michael

1 0

Issue with coordinator increasing limits
by Groner, Rob 24 Feb '26

24 Feb '26

I'm seeing a weird issue when I try to increase limits on a child account that a coordinator has created. Here is the setup: 1. Create an account named slurm_parent, set coordinator1 as a coordinator of that account, and give that account a grptresmins=cpu=100 limit. 2. As coordinator1, create a child account named slurm_child whose parent is slurm_parent. 3. As coordinator1, set the grptresmins=cpu=10 limit on the child account. That all works fine. The problem is when I try to INCREASE the cpu limit on the child account by any amount. $coordinator1> sacctmgr modify account test_crch_slurmaccessch set grptresmins=cpu=11 sacctmgr: error: Coordinators can not increase job limits beyond the parent ones Modified account associations... Error with request: Coordinators can not increase job limits beyond the parent ones The first issue is that the limit of cpu=11 is NOT beyond the parent limit of cpu=100. The second issue is that (as a workaround), I can remove the cpu=10 limit on the child account as coordinator (by setting it to -1), and then I can set grptresmins=cpu=1000. That command works. So, I can very clearly set a grptresmins=cpu limit that is beyond the parent limit (I understand that the child account in reality won't be able to access more than the parent account, no matter what the limit is). Note that as coordinator, I can REDUCE the limit on the child account all I want. So I can set it to cpu=9 just fine. But if I try to set it back to 10, I'll again get the error saying I can't go above parent limits. So here are my issues: 1. Inability of coordinator to increase the child account grptresmins limits once set. 2. Incorrect error message about exceeding the parent limit. I originally saw this issue on 24.11.6, and built and tested on 25.11.3, and still see the issue. Thanks for any help figuring out what I'm doing wrong. Rob

1 1

Reliable, Atomic, Idempotent, or Transactional Job Submission?
by Adam Novak 24 Feb '26

24 Feb '26

I'm working on the Slurm integration in our Toil workflow runner project. I'm having a problem where an `sbatch` command to submit a job to Slurm can fail (with exit code 1 and message "sbatch: error: Batch job submission failed: Socket timed out on send/recv operation", in my case, but possibly in other ways), but the job can still actually have been submitted, and can still execute. This causes a problem for Toil because right now, when it sees a submission attempt fail, it backs off and submits the job again a little later. But Toil can't handle multiple copies of the same job running at once, and if a submission appears to the client to have failed but actually succeeded, it's possible to get into that situation if you just submit again. When an sbatch command fails, is it possible to detect cases where the cluster will still execute the job? (For example, is it guaranteed that the job ID will be available on standard output on the client if the job is going to execute on the cluster, no matter when in the client process it might get a socket disconnection, allowing the job to be inquired about later?) Do I maybe need to tag my jobs with unique identifiers myself so I can poll for them in the queue after a supposedly-failed submission? Is it possible to write an idempotent sbatch command, where it can be run any number of times but will only actually submit one copy of the job? Is the Slurm REST API somehow more transactional, or able to promise somehow that a job will not actually go into the queue without the client having acknowledged receipt of the job's assigned ID? Thanks, -Adam -- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code." Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5

4 7

Get the list of added dynamic nodes
by Edmundo Carmona Antoranz 23 Feb '26

23 Feb '26

Hi! Is there a way to get the list of dynamic nodes that have been created but that are pending to get slurmd to run on them? I fail to see them with with either scontrol or sinfo: root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol create NodeName=node004 .... state=FUTURE feature=dyn1 root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol version slurm 25.11.2 Only when I start slurmd from node004 does it become visible (through scontrol only): root@ea-u24-dynamic-2511:~# ssh node004 /cm/local/apps/slurm/current/sbin/slurmd -Z -f /cm/shared/apps/slurm/etc/slurm/slurm.conf root@ea-u24-dynamic-2511:~# sinfo -N -a NODELIST NODES PARTITION STATE node001 1 defq* idle node002 1 defq* idle node003 1 defq* idle root@ea-u24-dynamic-2511:~# scontrol show nodes | grep NodeName NodeName=node001 Arch=x86_64 CoresPerSocket=1 NodeName=node002 Arch=x86_64 CoresPerSocket=1 NodeName=node003 Arch=x86_64 CoresPerSocket=1 NodeName=node004 Arch=x86_64 CoresPerSocket=1 Thanks in advance. Edmundo Carmona Antoranz Software Engineer NVIDIA Corporation

3 4

We are pleased to announce the availability of Slurm version 25.11.3
by Tim Wickberg 19 Feb '26

19 Feb '26

We are pleased to announce the availability of Slurm version 25.11.3. 25.11.3 fixes a number of stability issues, regressions, and a variety of other bugs, including the following: * Fix multiple issues with dynamic topology when power saving is enabled * Fix a backfill performance regression when using bf_licenses * Fix race conditions when shutting down slurmctld or slurmdbd that cause crashes or hangs * Fix slurmctld crashing when using enable_async_reply and TLS * Fix regression from 25.05 which caused usercpu and systemcpu to be missing for job steps * Fix regression that caused thread limits in slurmd to not be respected for incoming RPCs * Prevent the double-removal of accounting usage for jobs being requeued that are in the COMPLETED or COMPLETING state The full list of changes is available in the CHANGELOG file: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases One note on some updates to our release process - we've started publishing the releases directly on GitHub, in parallel to their usual location on the SchedMD web server. If you 'star' the Slurm repository you can automatically receive notices when new releases are published there. You can also subscribe to the RSS/Atom feed through: https://github.com/SchedMD/slurm/releases.atom - Tim -- Tim Wickberg Director, System Software, Slurm and Slinky, NVIDIA

1 0

preempt/qos is not working as expected
by Marchand Aurélia 19 Feb '26

19 Feb '26

Hello, I want too use QOS preemption. my configuration is : preemptMode = GANG,SUSPEND PreemptParameters = (null) PreemptType = preempt/qos PriorityType = priority/multifactor SelectType = select/cons_tres SelectTypeParameters = CR_CPU I define 3 partitions : PartitionName=DEFAULT Nodes=ALL Default=NO MaxTime=15-0 State=UP OverSubscribe=FORCE:1 PreemptMode=suspend PartitionName=veryhi PriorityTier=30 AllowQos=veryhi_short,gaia_veryhi_short PartitionName=hi PriorityTier=20 AllowQos=hi_short,gaia_hi_short PartitionName=def PriorityTier=10 OverSubscribe=No Default=YES AllowQos=def_short I define 5 QOS : sacctmgr show qos format=name%20,Priority,Preempt%40,PreemptMode Name Priority Preempt PreemptMode ----------------------- ------------ ---------------------------------- --------------- normal 0 cluster veryhi_short 20000 def_short suspend hi_short 10000 def_short suspend def_short 0 normal suspend gaia_veryhi_short 20000 def_short,gaia_hi_short suspend gaia_hi_short 10000 def_short suspend I define account : sacctmgr show ass format=account%20,user,qos%60 Account User QOS -------------------- -------- -------------------------------------------- gaia_hi user1 gaia_hi_short gaia_veryhi user1 gaia_veryhi_short mis_hi user1 hi_short def user1 def_short I submit 2 jobs : JOBID PARTITION QOS ACCOUNT NAME USER STATE PRIORITY 159 hi hi_short mis_hi test user1 SUSPENDE 637 160 veryhi gaia_veryhi_short gaia_veryhi test user1 RUNNING 1150 I don't understand why job 160 suspend job 159, the partition priority is higher, but I want gaia_veryhi_short QOS preempt only def_short and gaia_hi_short QOS. The QOS of job 159 is hi_short. Where is my configuration error ? Thanks for your time, Aurélia

2 4

for srun jobs, if we try to get UserCPU time by sacct. it always shows 0.
by squallabc＠126.com 16 Feb '26

16 Feb '26

Hi we have a slurm v25.05.2 setup. we found any srun jobs are always shows 0 on UserCPU . for sbatch jobs shoiws normal. is this expected or it is a bug ? Thanks. sacct -j 2948 --format=AveCPU,UserCPU AveCPU UserCPU ---------- ---------- 00:00:00 00:02:00 00:00:00

4 4

Sharding GPUs
by alessandr.dauria＠gmail.com 13 Feb '26

13 Feb '26

Dear Community, We are trying to activate sharding. Our Compute Node are configured with 64 cores, 4 phisical GPU MI250x ( 8 logical ) 4 Numa Domain. 1 Phisical Gpu / 2 logical GPU for each Numa Domain. 1 Logical GPU for each l3 cache domain gres.conf AutoDetect=rsmi NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD128 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD129 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD130 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD131 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD132 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD133 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD134 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD135 Count=4 If I ask 2 cores with block:cyclic I have the expected result srun -N1 -n2 -c1 --cpu-bind=cores -m block:cyclic --pty bash cpuset cgroup is 1,17 But if I add 2 shard in the request I don't expect this result srun -N1 -n2 -c1 --cpu-bind=cores --gres=shard:2 -m block:cyclic --pty bash cpuset cgroup is 1-2 ROCR_VISIBILE_DEVICES=0 Is it possibile request 2 sharding in round robin fashion, in order to run a multigpu job on different GPUs? srun -N1 -n2 -c1 --cpu-bind=cores --gres=shard:2 -m block:cyclic --pty bash Practically, I would to have this result cpuset cgroup is 1-17 ROCR_VISIBILE_DEVICES=0,1 Thank you in advance, Alessandro

1 1

Restarting slurmd kills still-running jobs
by Griebel, Christian 13 Feb '26

13 Feb '26

Dear community, Trying to implement the latest fix/patch for munged, we restarted the updated munged locally on the compute nodes with "systemctl restart munged", resulting in the sudden death of a lot of compute nodes' slurmd. Checking the jobs on the affected nodes, we saw a lot of user processes/jobs still running, which was good - yet "systemctl restart slurmd" cancelled all of them, eg. [2026-02-12T17:08:00.325] Cleaning up stray StepId=49695760.extern [2026-02-12T17:08:00.325] [49695760.extern] Sent signal 9 to StepId=49695760.extern [2026-02-12T17:08:00.325] slurmd version 25.05.5 started and all affected user jobs (even though having survived the death of their parent slurmd) were killed and re-queued... We have cgroups v2 (only, no hybrid), "Delegate=yes" in the slurmd unit and "ProctrackType=proctrack/cgroup" configured. Other sites do not see the same behavior (their user jobs survive a slurmd restart without issues), so now we are at a loss figuring out why the h.... this happens within our setup. Anyone experienced similar problems and got them solved...? Thanks in advance - -- ___________________________ Christian Griebel/HPC

6 5

sstat is not able check others job.
by squallabc＠126.com 12 Feb '26

12 Feb '26

Hi slurm version is v25.05.2. when I tried to query others job by sstat, it returns error. is there any configuration we can setup to fix this ? Thanks. sstat -j 3023.batch JobID AllocTRES MaxVMSize MaxVMSizeNode MaxVMSizeTask AveVMSize MaxRSS MaxRSSNode MaxRSSTask AveRSS MaxPages MaxPagesNode MaxPagesTask AvePages MinCPU MinCPUNode MinCPUTask AveCPU NTasks AveCPUFreq ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov ConsumedEnergy MaxDiskRead MaxDiskReadNode MaxDiskReadTask AveDiskRead MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask AveDiskWrite TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask TRESUsageInMin TRESUsageInMinNode TRESUsageInMinTask TRESUsageInTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot ------------ ---------- ---------- -------------- -------------- ---------- ---------- ---------- ---------- ---------- -------- ------------ -------------- ---------- ---------- ---------- ---------- ---------- -------- ---------- ------------- ------------- ------------- -------------- ------------ --------------- --------------- ------------ ------------ ---------------- ---------------- ------------ -------------- -------------- ------------------ ------------------ -------------- ------------------ ------------------ -------------- --------------- --------------- ------------------- ------------------- --------------- ------------------- ------------------- --------------- sstat: error: slurm_job_step_stat: there was an error with the request to atlsde8-reg0001 rc = Invalid user id sstat: error: problem getting step_layout for StepId=3023.batch: Invalid user id

3 2

MUNGE security issue (CVE-2026-25506)
by Tim Wickberg 12 Feb '26

12 Feb '26

MUNGE just announced a serious security issue as CVE-2026-25506: https://github.com/dun/munge/security/advisories/GHSA-r9cr-jf4v-75gh The issue is entirely within MUNGE - specifically in the munged daemon. However, as most Slurm installations explicitly trust MUNGE as the source of authentication, Slurm's security is immediately undermined if an attacker were to exploit that issue and gain access to the MUNGE key. On Slurm clusters you should treat that security issue as if it were local-root exploit. If you have switched to Slurm's built-in authentication ("auth/slurm"), you are not impacted by this issue. (However, we do not recommend switching over as a mitigation - you cannot safely change authentication types in Slurm while jobs are currently running.) I strongly encourage sites to get install the fixed packages from the upstream distros and restart the munged daemon on all nodes. If you install and restart munged promptly then Slurm should be unaffected. You do not need to rebuild Slurm for this issue. The MUNGE ABI/API remain unchanged, so there is no need to recompile Slurm, and there are no patches to apply to Slurm related to this issue. If you are concerned that an attacker could have compromised the MUNGE key itself, you may want to rotate that key. Unfortunately there's no mechanism to safely handle this key rotation with the cluster operational, and I do not recommend key rotation as long as you're installing and restarting MUNGE ASAP. - Tim -- Tim Wickberg Director, System Software, Slurm and Slinky, NVIDIA

11 16

Can I update when jobs are running?
by Gould, Ron (GRC-VBA0)[AEGIS] 11 Feb '26

11 Feb '26

The tl;dr is “This is my first upgrade since inheriting this Cluster, so I’m not sure what can or can’t be running during the upgrades.”. My Cluster is running an old version, 22.05.3. This is my first upgrade since inheriting the Cluster. As such, I’d like to install 22.05.4 because it’s a short jump, and it fixes the bug my users are seeing. The Cluster is composed of mostly Oracle Linux 8. I’m aware that I can upgrade within the two release compatibility window. I’ve read through the Upgrade guide and I’m unclear if downtime is required. Perhaps I’m unifying downtime requirements across different SLURM services where I should be interpreting that certain services have their own downtime requirements. https://slurm.schedmd.com/upgrades.html<https://slurm.schedmd.com/upgrades.html#procedure> In the Upgrade Procedure section<https://slurm.schedmd.com/upgrades.html#procedure>, there’re a couple questionable things. 1. Is downtime required? Does downtime == “all jobs must be halted”? “Downtime”, to me, seems like nothing should be running. This statement indicates that jobs can be running during the upgrade. Before considering the upgrade complete, wait for all jobs that were already running to finish. Any jobs started before the slurmd system was upgraded will be running with the old version of slurmstepd, so starting another upgrade or trying to use new features in the new version may cause problems. within a few paragraphs<https://slurm.schedmd.com/upgrades.html#downtime>, this message indicates I will need downtime: Refer to the expected downtime guidance in the following sections for each relevant Slurm daemon Further in the guide, in SLURMD (COMPUTE NODES)<https://slurm.schedmd.com/upgrades.html#slurmd>, I read Upgrades will not interrupt running jobs as long as SlurmdTimeout is not reached during the process This implies, at least, that existing running jobs can stay running. 2. There’re conflicting suggestions of using “rpm” to install the RPMs I built with “rpmbuild". Should I use “dnf localinstall ./*.rpm”? I’m inferring that dependencies will (not) be handled correctly. NOTE: If RPM/DEB packages are used, all packages present on each system must be upgraded together instead of piecewise. … Avoid using low-level package managers like rpm or dpkg as they may not properly enforce these dependencies However, in SLURMDBD (ACCOUNTING)<https://slurm.schedmd.com/upgrades.html#slurmdbd>, this statement Upgrade the slurmdbd daemon binaries, libraries, and its systemd unit file (if used). If using RPM/DEB packages, the package manager will take care of these indicates I should be using RPM packages. Lastly, to get to a current install, I need to step through multiple versions, with the condition that jobs started with a specific major version must finish within the compatibility window. GitLab has a tool where you plug in your current and intended versions and it tells you explicitly which versions are required along the upgrade path. I’d like a similarly explicit tool for SLURM, but I infer from the Compatibility Window<https://slurm.schedmd.com/upgrades.html#compatibility_window> that I can update like so: 1. Current = 22.05.3 2. 23.11 3. 25.05 4. 26.05 That feels like a big leapfrog between versions. I’d like the practice of upgrading. Is there any detriment to upgrading at a slower pace: 1. Current = 22.05.3 2. 22.05.11 3. 23.02.8 4. 23.11.11 5. 24.05.8 6. 24.11.7 7. 25.05.6 8. 25.11.2

9 26

Adding FPGAs for Slurm Tracking
by Raven Hallsby 08 Feb '26

08 Feb '26

Hey all, We have a very heterogeneous cluster with a mix of Intel & AMD machines, GPUs, and FPGAs. We use Slurm to allocate time on the CPUs right now, but do not manage the GPU or FPGAs with Slurm yet. Some of our work involves hardware/SoC design, and we have been using Firesim (https://fires.im/) for a good amount of our work. Firesim is a bunch of Python that sits on top of invoking programs to provide "seamless" functionality and orchestration. However, Firesim was designed for AWS where a single user controls all of the resources (FPGAs in our case) used for the simulation. We have gone through the effort to pull out the important bits from Firesim to write a shell script that we can execute locally to achieve the same thing as Firesim' orchestration, without the need for an orchestrator. We have two FPGAs in two separate but identical machines and multiple people that want to use them for very long periods of time (days or longer) and want to let people freely move between the two FPGAs. All of our other machines in the cluster use Slurm, so we are looking to have Slurm multiplex these FPGA jobs. We already have the NFS stuff set up, so shared disk is fine. The important thing in our desired Slurm setup is that no user is allocated the same FPGA at the same time for the same job (you cannot have multiple FPGA jobs running the same design in parallel, partly because of bitstreams, partly because of disk images). Slurm does not track FPGAs by default. I have looked into how to teach Slurm about the FPGAs and know that I need to tell Slurm about the FPGAs as a GRES through gres.conf. I just want to confirm that I have the correct steps below: 1. Mark the 2 FPGAs as GRES with gres.conf. What should they be marked as? Do I need anything special? 2. Then enable tracking on the FPGA GRES to make a TRES in slurm.conf. What should this be labeled as? I see other groups using GresTypes=gpu in their configuration. Would that apply here? Since FPGA usage is exclusive already, I could see the GPU GRES type applying. 3. Then the TRES can be used by a QoS to enforce limits? The other important question, will I need to write a Slurm plugin (GRES or otherwise) to manage and detect the FPGAs? I saw the files stuff for GRES-es, but want to be sure. Writing a plugin is not a big deal for us, but it is future maintenance that I would want to document.

1 0

Re: Dynamic MIG Question
by Davide DelVento 06 Feb '26

06 Feb '26

Aaron (or anyone else), Did you manage to get Dynamic MIG working in Slurm? I'm actually surprised that after these many years SchedMD has not implemented this feature yet, especially now that newer GPUs allow MIG repartitioning without being root. The only mention of this in their ticketing system is at https://support.schedmd.com/show_bug.cgi?id=11091#c8 (and subsequent c10) which say that it's not on their roadmap, but that was 5 years ago. I have heard that some users manage dynamic changes by draining nodes, running scripts to reconfigure MIG via nvidia-smi, bringing the node back and then submitting the job. Anybody here has tried that and with what success? I speculate that now that NVIDIA owns SchedMD perhaps this feature will be at a higher priority, but maybe not? Anybody knows anything about it and is not bound by an NDA to keep mum? Thanks On Wed, Nov 22, 2023 at 1:22 PM Davide DelVento <davide.quantum(a)gmail.com> wrote: > I assume you mean the sentence about dynamic MIG at > https://slurm.schedmd.com/gres.html#MIG_Management > Could it be supported? I think so, but only if one of their paying > customers (that could be you) asks for it. > > On Wed, Nov 22, 2023 at 11:24 AM Aaron Kollmann < > aaron.kollmann(a)student.hpi.de> wrote: > >> Hello All, >> >> I am currently working in a research project and we are trying to find >> out whether we can use NVIDIAs multi-instance GPU (MIG) dynamically in >> SLURM. >> >> For instance: >> >> - a user requests a job and wants a GPU but none is available >> >> - now SLURM will reconfigure a MIG GPU to create a partition (e.g. >> 1g.5gb) which becomes available and allocated immediately >> >> I can already reconfigure MIG + SLURM within a few seconds to start jobs >> on newly partitioned resources, but Jobs get killed when I restart slurmd >> on nodes with a changed MIG config. (see script example below) >> >> *Do you think it is possible to develop a plugin or change SLURM to the >> extent that dynamic MIG will be supported one day? * >> >> (The website says it is not supported) >> >> >> >> Best >> >> - Aaron >> >> >> >> >> #!/usr/bin/bash >> >> # Generate Start Config >> killall slurmd >> killall slurmctld >> nvidia-smi mig -dci >> nvidia-smi mig -dgi >> nvidia-smi mig -cgi 19,14,5 -i 0 -C >> nvidia-smi mig -cgi 0 -i 1 -C >> cp -f ./slurm-19145-0.conf /etc/slurm/slurm.conf >> slurmd -c >> slurmctld -c >> sleep 5 >> >> # Start a running and a pending job (the first job gets killed by slurm) >> srun -w gx06 -c 2 --mem 1G --gres=gpu:a100_1g.5gb:1 sleep 300 & >> srun -w gx06 -c 2 --mem 1G --gres=gpu:a100_1g.5gb:1 sleep 300 & >> sleep 5 >> >> # Simulate MIG Config Change >> nvidia-smi mig -i 1 -dci >> nvidia-smi mig -i 1 -dgi >> nvidia-smi mig -cgi 19,14,5 -i 1 -C >> cp -f ./slurm-2x19145.conf /etc/slurm/slurm.conf >> killall slurmd >> killall slurmctld >> slurmd >> slurmctld >> >

2 3

Can we avoid jobs near wall time from being preempted?
by iazaceta＠vicomtech.org 06 Feb '26

06 Feb '26

We have an HPC where the average job length is measured in days, not hours. Users are careful to add checkpoints to their jobs but even in that case, preempting a job that is close to its walltime (max: 14 days) can be very disruptive. I checked what options preemption offers but none seem to protect jobs near their finishing line. PreemptExempTime ensures a minimum job runtime and GraceTime allows for a grace time period after the job has been selected for preemption. Is there anything I am missing to achieve what I want? Thank you!

2 1

systemd unit ordering for dbd host
by Michael Gutteridge 05 Feb '26

05 Feb '26

We have a host dedicated to slurmdbd and mysql (Ubuntu 24.04, Slurm 24.11.4, mysql 8.0.45). A couple days ago unattended-upgrades upgraded the MySQL package which triggered a restart of mysql.service which looks to have caused some errors in slurmdbd. I don't think those errors were permanent, everything looks to be running as per usual currently. Enough backstory- the question is should we configure the slurmdbd systemd unit to stop slurmdbd before mysql is stopped? Currently there are minimal dependencies built into the unit: After=network-online.target remote-fs.target munge.service mysql.service mysqld.service mariadb.service sssd.service Wants=network-online.target But nothing to tell systemd to take down slurmdbd when the mysql unit is stopped. Is there *any* reason we'd want slurmdbd running if mysql wasn't? Are there any problems I'd cause by creating such a dependency? I do plan to exclude mysql from future unattended upgrading, but I'd also like to keep our systemd units sensible. Thanks - Michael

3 3

How to get strigger event information?
by Mauricio Pinto 03 Feb '26

03 Feb '26

Greetings, I am managing a cluster using SLURM, and want to use strigger to automate notifications about certain events like jobs or nodes failing or changing states, slurmctld failing, etc. I want the notifications I send to include information about the event, like which node, which state, which job ID, and so on. However, so far I haven't found documentation on how to get these values. I did find a related thread <https://groups.google.com/g/slurm-users/c/NhrlL32q0Io/m/HKVd9l1yDgAJ> in the google groups archive. One of the replies includes a script which uses environment variables such as $SLURM_JOB_ID and $SLURM_JOB_NAME. How are these environment variables available? Is there documentation on which environment variables are available to strigger scripts? I'd appreciate any guidance you can provide. On a separate note, I just joined this mailing list and haven't seen any community guidelines, preferred email format, etc. Please feel free to point them out to me or make suggestions if you have any. Thanks, Mauricio

2 2

Configless Slurm Error: failed to fetch remote configs
by Xaver Stiensmeier 30 Jan '26

30 Jan '26

Hey everyone, in the past we set up clusters with configs on each node. Now we want to explore configless. Without changing anything else, we therefore followed: https://slurm.schedmd.com/configless_slurm.html and added 'enable_configless' in the config on the master: SlurmctldParameters=cloud_dns,idle_on_node_suspend,enable_configless,reconfig_on_restart and start each worker's slurmd with the conf-server parameter: # Override systemd service to set conditional path [Service] ExecStart= ExecStart=/usr/sbin/slurmd --conf-server=master However, this leads to: slurmd: error: _fetch_child: failed to fetch remote configs: Protocol authentication error slurmd: error: _establish_configuration: failed to load configs. Retrying in 10 seconds. on the workers and on the master (/var/log/slurm/slurmctld) to: [2026-01-16T10:00:06.681] error: Munge decode failed: Invalid credential [2026-01-16T10:00:06.681] auth/munge: _print_cred: ENCODED: Thu Jan 01 00:00:00 1970 [2026-01-16T10:00:06.681] auth/munge: _print_cred: DECODED: Thu Jan 01 00:00:00 1970 [2026-01-16T10:00:06.681] error: slurm_unpack_received_msg: [[worker]:24295] auth_g_verify: REQUEST_CONFIG has authentication error: Unspecified error [2026-01-16T10:00:06.681] error: slurm_unpack_received_msg: [[worker]:24295] Protocol authentication error The munge key setup is the same as before so I don't think there is anything wrong with it unless something changes with configless (slurm.conf): AuthType=auth/munge CryptoType=crypto/munge AuthAltTypes=auth/jwt AuthAltParameters=jwt_key=/etc/slurm/jwt-secret.key I found https://groups.google.com/g/slurm-users/c/Q7FVkhx-bOs but this seems unrelated as both can talk fine with each other: worker:~$ nc -zv master 6817 Connection to master (192.168.20.169) 6817 port [tcp/*] succeeded! I tried adding more "-v" to the slurmd start, but that did not give more information. I am unsure how to debug this further. Somehow I think it must be a munge issue, but I am confused as this part hasn't changed. Best regards, Xaver

2 6

Nodes removed from topology after suspend in 25.11.2
by Franky Backeljauw 29 Jan '26

29 Jan '26

Hi all We have upgraded to version 25.11.2 and are now noticing that nodes are removed getting removed from the topology/tree when the node gets suspended. In the slurmctld.log we see things like this: [2026-01-29T23:31:32.517] POWER: power_save: suspending nodes node1 [2026-01-29T23:32:37.536] debug2: _update_node_features_post: nodes node1 active features set to: feature1 [2026-01-29T23:32:37.536] debug2: topology/tree: topology_p_add_rm_node: topology_p_add_rm_node: remove node1 from switch1 If we later on run a job with: srun -w node1 --pty bash then we see the following: [2026-01-30T00:08:04.004] debug2: NodeSet for JobId=1234 [2026-01-30T00:08:04.005] _pick_best_nodes: JobId=1234 never runnable in partition part1 [2026-01-30T00:08:04.005] sched: schedule: JobId=1234 non-runnable: Requested topology configuration is not available [2026-01-30T00:08:04.005] debug2: _schedule: setting JobId=1234 to "BadConstraints" (Requested topology configuration is not available) This worked without any issues in the previous version 25.05.2, so something has change in the newer version 25.11.2. My guess is something with dynamic nodes and/or active versus available features, but it’s just not clear :-( Ideas anyone??? Our setup includes in slurm.conf (trimmed): TopologyPlugin = topology/tree NodeName=node1 Features=feature1 NodeSet=set1 Feature=feature1 PartitionName=part1 Nodes=set1 [...] SuspendProgram = /power_save/suspend.sh ResumeProgram = /power_save/resume.sh ReconfigFlags = KeepPowerSaveSettings In topology.conf: SwitchName=switch1 Nodes=node1 The SuspendProgram and ResumeProgram scripts use tuned-adm to change the power profile (to powersave and hpc-compute resp.). When looking at the node itself, after suspend, it shows (trimmed): NodeName=node1 AvailableFeatures=feature1 ActiveFeatures=feature1 State=IDLE+POWERED_DOWN PARTITION=part1 I hope someone can shed a light on this issue... Thanks in advance for any reply! -- Kind regards Franky

1 0

sacct 25.11.2: --state filter returns no results
by Norman Gaywood 26 Jan '26

26 Jan '26

Hi all, I have recently upgraded my cluster to Slurm 25.11.2 (running on Rocky Linux 9). slurm-25.11.2-1.el9.x86_64 built locally. This time I built accounting in and so I'm new to sacct. When I run sacct without a state filter, I see the jobs correctly marked as COMPLETED: sacct -a -S 2026-01-26 --format=JobID,State | grep COMPLETED | head -10 9 COMPLETED 9.batch COMPLETED 9.extern COMPLETED 9.0 COMPLETED 39 COMPLETED 39.batch COMPLETED 39.extern COMPLETED 39.0 COMPLETED 40 COMPLETED 40.batch COMPLETED When I attempt to filter for those exact states (using either the short or long flag), the output is empty: sacct -a -S 2026-01-26 --format=JobID,State --state=completed JobID State ------------ ---------- (no output beyond the heading) Am I using it wrong? Cheers, Norm.

2 2

opposite of --array
by Herbert Fruchtl 21 Jan '26

21 Jan '26

Folks, When I use sacct, it always shows all tasks of an array job individually. This is supposed to be switched on by --array, but it seems to be the default. How do I switch it off? Thanks in advance, Herbert -- Herbert Fruchtl (he/him) Senior Scientific Computing Officer / HPC Administrator School of Chemistry, IT Services University of St Andrews -- The University of St Andrews is a charity registered in Scotland: No SC013532

2 3

2026

2025

2024

slurm-users