- slurm-users - lists.schedmd.com

Reliable, Atomic, Idempotent, or Transactional Job Submission?
by Adam Novak 18 Feb '26

18 Feb '26

I'm working on the Slurm integration in our Toil workflow runner project. I'm having a problem where an `sbatch` command to submit a job to Slurm can fail (with exit code 1 and message "sbatch: error: Batch job submission failed: Socket timed out on send/recv operation", in my case, but possibly in other ways), but the job can still actually have been submitted, and can still execute. This causes a problem for Toil because right now, when it sees a submission attempt fail, it backs off and submits the job again a little later. But Toil can't handle multiple copies of the same job running at once, and if a submission appears to the client to have failed but actually succeeded, it's possible to get into that situation if you just submit again. When an sbatch command fails, is it possible to detect cases where the cluster will still execute the job? (For example, is it guaranteed that the job ID will be available on standard output on the client if the job is going to execute on the cluster, no matter when in the client process it might get a socket disconnection, allowing the job to be inquired about later?) Do I maybe need to tag my jobs with unique identifiers myself so I can poll for them in the queue after a supposedly-failed submission? Is it possible to write an idempotent sbatch command, where it can be run any number of times but will only actually submit one copy of the job? Is the Slurm REST API somehow more transactional, or able to promise somehow that a job will not actually go into the queue without the client having acknowledged receipt of the job's assigned ID? Thanks, -Adam -- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code." Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5

3 3

preempt/qos is not working as expected
by Marchand Aurélia 17 Feb '26

17 Feb '26

Hello, I want too use QOS preemption. my configuration is : preemptMode = GANG,SUSPEND PreemptParameters = (null) PreemptType = preempt/qos PriorityType = priority/multifactor SelectType = select/cons_tres SelectTypeParameters = CR_CPU I define 3 partitions : PartitionName=DEFAULT Nodes=ALL Default=NO MaxTime=15-0 State=UP OverSubscribe=FORCE:1 PreemptMode=suspend PartitionName=veryhi PriorityTier=30 AllowQos=veryhi_short,gaia_veryhi_short PartitionName=hi PriorityTier=20 AllowQos=hi_short,gaia_hi_short PartitionName=def PriorityTier=10 OverSubscribe=No Default=YES AllowQos=def_short I define 5 QOS : sacctmgr show qos format=name%20,Priority,Preempt%40,PreemptMode Name Priority Preempt PreemptMode ----------------------- ------------ ---------------------------------- --------------- normal 0 cluster veryhi_short 20000 def_short suspend hi_short 10000 def_short suspend def_short 0 normal suspend gaia_veryhi_short 20000 def_short,gaia_hi_short suspend gaia_hi_short 10000 def_short suspend I define account : sacctmgr show ass format=account%20,user,qos%60 Account User QOS -------------------- -------- -------------------------------------------- gaia_hi user1 gaia_hi_short gaia_veryhi user1 gaia_veryhi_short mis_hi user1 hi_short def user1 def_short I submit 2 jobs : JOBID PARTITION QOS ACCOUNT NAME USER STATE PRIORITY 159 hi hi_short mis_hi test user1 SUSPENDE 637 160 veryhi gaia_veryhi_short gaia_veryhi test user1 RUNNING 1150 I don't understand why job 160 suspend job 159, the partition priority is higher, but I want gaia_veryhi_short QOS preempt only def_short and gaia_hi_short QOS. The QOS of job 159 is hi_short. Where is my configuration error ? Thanks for your time, Aurélia

2 2

for srun jobs, if we try to get UserCPU time by sacct. it always shows 0.
by squallabc＠126.com 16 Feb '26

16 Feb '26

Hi we have a slurm v25.05.2 setup. we found any srun jobs are always shows 0 on UserCPU . for sbatch jobs shoiws normal. is this expected or it is a bug ? Thanks. sacct -j 2948 --format=AveCPU,UserCPU AveCPU UserCPU ---------- ---------- 00:00:00 00:02:00 00:00:00

4 4

Sharding GPUs
by alessandr.dauria＠gmail.com 13 Feb '26

13 Feb '26

Dear Community, We are trying to activate sharding. Our Compute Node are configured with 64 cores, 4 phisical GPU MI250x ( 8 logical ) 4 Numa Domain. 1 Phisical Gpu / 2 logical GPU for each Numa Domain. 1 Logical GPU for each l3 cache domain gres.conf AutoDetect=rsmi NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD128 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD129 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD130 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD131 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD132 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD133 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD134 Count=4 NodeName=c6n[3339-3348] Name=shard File=/dev/dri/renderD135 Count=4 If I ask 2 cores with block:cyclic I have the expected result srun -N1 -n2 -c1 --cpu-bind=cores -m block:cyclic --pty bash cpuset cgroup is 1,17 But if I add 2 shard in the request I don't expect this result srun -N1 -n2 -c1 --cpu-bind=cores --gres=shard:2 -m block:cyclic --pty bash cpuset cgroup is 1-2 ROCR_VISIBILE_DEVICES=0 Is it possibile request 2 sharding in round robin fashion, in order to run a multigpu job on different GPUs? srun -N1 -n2 -c1 --cpu-bind=cores --gres=shard:2 -m block:cyclic --pty bash Practically, I would to have this result cpuset cgroup is 1-17 ROCR_VISIBILE_DEVICES=0,1 Thank you in advance, Alessandro

1 1

Restarting slurmd kills still-running jobs
by Griebel, Christian 13 Feb '26

13 Feb '26

Dear community, Trying to implement the latest fix/patch for munged, we restarted the updated munged locally on the compute nodes with "systemctl restart munged", resulting in the sudden death of a lot of compute nodes' slurmd. Checking the jobs on the affected nodes, we saw a lot of user processes/jobs still running, which was good - yet "systemctl restart slurmd" cancelled all of them, eg. [2026-02-12T17:08:00.325] Cleaning up stray StepId=49695760.extern [2026-02-12T17:08:00.325] [49695760.extern] Sent signal 9 to StepId=49695760.extern [2026-02-12T17:08:00.325] slurmd version 25.05.5 started and all affected user jobs (even though having survived the death of their parent slurmd) were killed and re-queued... We have cgroups v2 (only, no hybrid), "Delegate=yes" in the slurmd unit and "ProctrackType=proctrack/cgroup" configured. Other sites do not see the same behavior (their user jobs survive a slurmd restart without issues), so now we are at a loss figuring out why the h.... this happens within our setup. Anyone experienced similar problems and got them solved...? Thanks in advance - -- ___________________________ Christian Griebel/HPC

6 5

sstat is not able check others job.
by squallabc＠126.com 12 Feb '26

12 Feb '26

Hi slurm version is v25.05.2. when I tried to query others job by sstat, it returns error. is there any configuration we can setup to fix this ? Thanks. sstat -j 3023.batch JobID AllocTRES MaxVMSize MaxVMSizeNode MaxVMSizeTask AveVMSize MaxRSS MaxRSSNode MaxRSSTask AveRSS MaxPages MaxPagesNode MaxPagesTask AvePages MinCPU MinCPUNode MinCPUTask AveCPU NTasks AveCPUFreq ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov ConsumedEnergy MaxDiskRead MaxDiskReadNode MaxDiskReadTask AveDiskRead MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask AveDiskWrite TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask TRESUsageInMin TRESUsageInMinNode TRESUsageInMinTask TRESUsageInTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot ------------ ---------- ---------- -------------- -------------- ---------- ---------- ---------- ---------- ---------- -------- ------------ -------------- ---------- ---------- ---------- ---------- ---------- -------- ---------- ------------- ------------- ------------- -------------- ------------ --------------- --------------- ------------ ------------ ---------------- ---------------- ------------ -------------- -------------- ------------------ ------------------ -------------- ------------------ ------------------ -------------- --------------- --------------- ------------------- ------------------- --------------- ------------------- ------------------- --------------- sstat: error: slurm_job_step_stat: there was an error with the request to atlsde8-reg0001 rc = Invalid user id sstat: error: problem getting step_layout for StepId=3023.batch: Invalid user id

3 2

MUNGE security issue (CVE-2026-25506)
by Tim Wickberg 12 Feb '26

12 Feb '26

MUNGE just announced a serious security issue as CVE-2026-25506: https://github.com/dun/munge/security/advisories/GHSA-r9cr-jf4v-75gh The issue is entirely within MUNGE - specifically in the munged daemon. However, as most Slurm installations explicitly trust MUNGE as the source of authentication, Slurm's security is immediately undermined if an attacker were to exploit that issue and gain access to the MUNGE key. On Slurm clusters you should treat that security issue as if it were local-root exploit. If you have switched to Slurm's built-in authentication ("auth/slurm"), you are not impacted by this issue. (However, we do not recommend switching over as a mitigation - you cannot safely change authentication types in Slurm while jobs are currently running.) I strongly encourage sites to get install the fixed packages from the upstream distros and restart the munged daemon on all nodes. If you install and restart munged promptly then Slurm should be unaffected. You do not need to rebuild Slurm for this issue. The MUNGE ABI/API remain unchanged, so there is no need to recompile Slurm, and there are no patches to apply to Slurm related to this issue. If you are concerned that an attacker could have compromised the MUNGE key itself, you may want to rotate that key. Unfortunately there's no mechanism to safely handle this key rotation with the cluster operational, and I do not recommend key rotation as long as you're installing and restarting MUNGE ASAP. - Tim -- Tim Wickberg Director, System Software, Slurm and Slinky, NVIDIA

11 16

Can I update when jobs are running?
by Gould, Ron (GRC-VBA0)[AEGIS] 11 Feb '26

11 Feb '26

The tl;dr is “This is my first upgrade since inheriting this Cluster, so I’m not sure what can or can’t be running during the upgrades.”. My Cluster is running an old version, 22.05.3. This is my first upgrade since inheriting the Cluster. As such, I’d like to install 22.05.4 because it’s a short jump, and it fixes the bug my users are seeing. The Cluster is composed of mostly Oracle Linux 8. I’m aware that I can upgrade within the two release compatibility window. I’ve read through the Upgrade guide and I’m unclear if downtime is required. Perhaps I’m unifying downtime requirements across different SLURM services where I should be interpreting that certain services have their own downtime requirements. https://slurm.schedmd.com/upgrades.html<https://slurm.schedmd.com/upgrades.html#procedure> In the Upgrade Procedure section<https://slurm.schedmd.com/upgrades.html#procedure>, there’re a couple questionable things. 1. Is downtime required? Does downtime == “all jobs must be halted”? “Downtime”, to me, seems like nothing should be running. This statement indicates that jobs can be running during the upgrade. Before considering the upgrade complete, wait for all jobs that were already running to finish. Any jobs started before the slurmd system was upgraded will be running with the old version of slurmstepd, so starting another upgrade or trying to use new features in the new version may cause problems. within a few paragraphs<https://slurm.schedmd.com/upgrades.html#downtime>, this message indicates I will need downtime: Refer to the expected downtime guidance in the following sections for each relevant Slurm daemon Further in the guide, in SLURMD (COMPUTE NODES)<https://slurm.schedmd.com/upgrades.html#slurmd>, I read Upgrades will not interrupt running jobs as long as SlurmdTimeout is not reached during the process This implies, at least, that existing running jobs can stay running. 2. There’re conflicting suggestions of using “rpm” to install the RPMs I built with “rpmbuild". Should I use “dnf localinstall ./*.rpm”? I’m inferring that dependencies will (not) be handled correctly. NOTE: If RPM/DEB packages are used, all packages present on each system must be upgraded together instead of piecewise. … Avoid using low-level package managers like rpm or dpkg as they may not properly enforce these dependencies However, in SLURMDBD (ACCOUNTING)<https://slurm.schedmd.com/upgrades.html#slurmdbd>, this statement Upgrade the slurmdbd daemon binaries, libraries, and its systemd unit file (if used). If using RPM/DEB packages, the package manager will take care of these indicates I should be using RPM packages. Lastly, to get to a current install, I need to step through multiple versions, with the condition that jobs started with a specific major version must finish within the compatibility window. GitLab has a tool where you plug in your current and intended versions and it tells you explicitly which versions are required along the upgrade path. I’d like a similarly explicit tool for SLURM, but I infer from the Compatibility Window<https://slurm.schedmd.com/upgrades.html#compatibility_window> that I can update like so: 1. Current = 22.05.3 2. 23.11 3. 25.05 4. 26.05 That feels like a big leapfrog between versions. I’d like the practice of upgrading. Is there any detriment to upgrading at a slower pace: 1. Current = 22.05.3 2. 22.05.11 3. 23.02.8 4. 23.11.11 5. 24.05.8 6. 24.11.7 7. 25.05.6 8. 25.11.2

9 26

Adding FPGAs for Slurm Tracking
by Raven Hallsby 08 Feb '26

08 Feb '26

Hey all, We have a very heterogeneous cluster with a mix of Intel & AMD machines, GPUs, and FPGAs. We use Slurm to allocate time on the CPUs right now, but do not manage the GPU or FPGAs with Slurm yet. Some of our work involves hardware/SoC design, and we have been using Firesim (https://fires.im/) for a good amount of our work. Firesim is a bunch of Python that sits on top of invoking programs to provide "seamless" functionality and orchestration. However, Firesim was designed for AWS where a single user controls all of the resources (FPGAs in our case) used for the simulation. We have gone through the effort to pull out the important bits from Firesim to write a shell script that we can execute locally to achieve the same thing as Firesim' orchestration, without the need for an orchestrator. We have two FPGAs in two separate but identical machines and multiple people that want to use them for very long periods of time (days or longer) and want to let people freely move between the two FPGAs. All of our other machines in the cluster use Slurm, so we are looking to have Slurm multiplex these FPGA jobs. We already have the NFS stuff set up, so shared disk is fine. The important thing in our desired Slurm setup is that no user is allocated the same FPGA at the same time for the same job (you cannot have multiple FPGA jobs running the same design in parallel, partly because of bitstreams, partly because of disk images). Slurm does not track FPGAs by default. I have looked into how to teach Slurm about the FPGAs and know that I need to tell Slurm about the FPGAs as a GRES through gres.conf. I just want to confirm that I have the correct steps below: 1. Mark the 2 FPGAs as GRES with gres.conf. What should they be marked as? Do I need anything special? 2. Then enable tracking on the FPGA GRES to make a TRES in slurm.conf. What should this be labeled as? I see other groups using GresTypes=gpu in their configuration. Would that apply here? Since FPGA usage is exclusive already, I could see the GPU GRES type applying. 3. Then the TRES can be used by a QoS to enforce limits? The other important question, will I need to write a Slurm plugin (GRES or otherwise) to manage and detect the FPGAs? I saw the files stuff for GRES-es, but want to be sure. Writing a plugin is not a big deal for us, but it is future maintenance that I would want to document.

1 0

Re: Dynamic MIG Question
by Davide DelVento 06 Feb '26

06 Feb '26

Aaron (or anyone else), Did you manage to get Dynamic MIG working in Slurm? I'm actually surprised that after these many years SchedMD has not implemented this feature yet, especially now that newer GPUs allow MIG repartitioning without being root. The only mention of this in their ticketing system is at https://support.schedmd.com/show_bug.cgi?id=11091#c8 (and subsequent c10) which say that it's not on their roadmap, but that was 5 years ago. I have heard that some users manage dynamic changes by draining nodes, running scripts to reconfigure MIG via nvidia-smi, bringing the node back and then submitting the job. Anybody here has tried that and with what success? I speculate that now that NVIDIA owns SchedMD perhaps this feature will be at a higher priority, but maybe not? Anybody knows anything about it and is not bound by an NDA to keep mum? Thanks On Wed, Nov 22, 2023 at 1:22 PM Davide DelVento <davide.quantum(a)gmail.com> wrote: > I assume you mean the sentence about dynamic MIG at > https://slurm.schedmd.com/gres.html#MIG_Management > Could it be supported? I think so, but only if one of their paying > customers (that could be you) asks for it. > > On Wed, Nov 22, 2023 at 11:24 AM Aaron Kollmann < > aaron.kollmann(a)student.hpi.de> wrote: > >> Hello All, >> >> I am currently working in a research project and we are trying to find >> out whether we can use NVIDIAs multi-instance GPU (MIG) dynamically in >> SLURM. >> >> For instance: >> >> - a user requests a job and wants a GPU but none is available >> >> - now SLURM will reconfigure a MIG GPU to create a partition (e.g. >> 1g.5gb) which becomes available and allocated immediately >> >> I can already reconfigure MIG + SLURM within a few seconds to start jobs >> on newly partitioned resources, but Jobs get killed when I restart slurmd >> on nodes with a changed MIG config. (see script example below) >> >> *Do you think it is possible to develop a plugin or change SLURM to the >> extent that dynamic MIG will be supported one day? * >> >> (The website says it is not supported) >> >> >> >> Best >> >> - Aaron >> >> >> >> >> #!/usr/bin/bash >> >> # Generate Start Config >> killall slurmd >> killall slurmctld >> nvidia-smi mig -dci >> nvidia-smi mig -dgi >> nvidia-smi mig -cgi 19,14,5 -i 0 -C >> nvidia-smi mig -cgi 0 -i 1 -C >> cp -f ./slurm-19145-0.conf /etc/slurm/slurm.conf >> slurmd -c >> slurmctld -c >> sleep 5 >> >> # Start a running and a pending job (the first job gets killed by slurm) >> srun -w gx06 -c 2 --mem 1G --gres=gpu:a100_1g.5gb:1 sleep 300 & >> srun -w gx06 -c 2 --mem 1G --gres=gpu:a100_1g.5gb:1 sleep 300 & >> sleep 5 >> >> # Simulate MIG Config Change >> nvidia-smi mig -i 1 -dci >> nvidia-smi mig -i 1 -dgi >> nvidia-smi mig -cgi 19,14,5 -i 1 -C >> cp -f ./slurm-2x19145.conf /etc/slurm/slurm.conf >> killall slurmd >> killall slurmctld >> slurmd >> slurmctld >> >

2 3

2026

2025

2024

slurm-users