- slurm-users - lists.schedmd.com

Limit number of allocated GPUs
by Gestió Servidors 23 Oct '25

23 Oct '25

Hello, I have three nodes, serving each one 2 GPUs. I would like to limit (qos??) that a user could user only one GPU from earch server, but user could user simultaneously three GPUs if each GPU belongs to different servers. With this QoS "sacctmgr add qos test-limit-GPUs MaxJobsPerUser=3 MaxTRESPerUser=gres/gpu=1" I can limit to one GPU, but then user can't run other job in a GPU from other server. How must I configure QoS (or other method) to allow more than one job requesting GPUs but never in the same server? Thanks.

3 2

Job remains "PENDING" with reason "QOSMaxGRESPerUser"
by Gestió Servidors 23 Oct '25

23 Oct '25

Hello, I have added a new "qos" with these parameters: sacctmgr add qos test-GPUs MaxJobsPerUser=6 MaxTRESPerUser=gres/gpu=1 MaxSubmitJobsPerUser=25. With it, I only allow 6 running jobs per user, a total of 25 pending+running job per user and only 1 GPU. I have applied this qos directly to a partition in slurm.conf. When a user submits to that partition requesting 2 or more GPUs, job remains "PD" (pending) and notifies "QOSMaxGRESPerUser" in NODELIST column, but I would like to know if it would be possible to direcly reject job and avoid that job remains at queue? For example, if I submit 50 jobs, after number 25 I get message "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) sbatch: error: QOSMaxSubmitJobPerUserLimit" 25 times) Thanks.

2 1

dynamic node slurm node list order
by Assmann, Greta Marie 23 Oct '25

23 Oct '25

Hello, we would like to understand how the internal SLURM node list order works. More detailed info: Our setup: We have a Slurm cluster with N dynamic nodes (heterogeneous node types) in a partition and we observe that if all resources are free, jobs always get submitted to one specific node first. This node is also the node that turns up at the top of the list when doing an $ scontrol show nodes . As we do not have a defined node list in slurm.conf as it is done for non-dynamic nodes, we were wondering how the order is set up. Interestingly, when deleting (and unregistering) this node from the cluster and re-registering it again, the node is still at the same position when doing scontrol show nodes. Is there some internal node list order caching or similar? How is the node list order defined ? Thanks , Greta Dr. Greta Assmann Data Analysis and Research Infrastructure OBBA 230 Forschungsstrasse 111 5232 Villigen-PSI email: greta.assmann(a)psi.ch

4 5

Slurm and Slinky events this fall, update events page
by Tim Wickberg 14 Oct '25

14 Oct '25

We'll have a bit more details as conference season quickly approaches this November, but SchedMD staff are presenting at KubeCon NA on Slinky [1]. We'll be manning the Slurm Booth at SC25 [2], as well as hosting the annual Slurm Community Birds-of-a-Feather session [3]. I'll also send a link out to the survey questions for the BoF to the slurm-users list ahead of the conference, and we'll be going into a bit more depth on the answers during the BoF this year. The events page on the SchedMD website has more detail on future events as well: https://www.schedmd.com/events/ A few folks had asked, and apparently we never mentioned this more publicly, but: SchedMD does not plan to hold an in-person SLUG in 2025 or 2026. We are working to bring some of the same content to our YouTube channel [4] as a way to more broadly disseminate some of that same content, starting with the Slurm 25.11 release overview in December. - Tim [1] https://kccncna2025.sched.com/event/27FW5/ [2] The Slurm Booth is #1641. [3] https://sc25.conference-program.com/presentation/?id=bof101&sess=sess471 [4] https://www.youtube.com/SchedMDSlurm -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

Slurm version 25.05.4 is now available
by Marshall Garey 14 Oct '25

14 Oct '25

We are pleased to announce the availability of Slurm version 25.05.4. This version increases the default number of maximum connections to slurmctld from 50 to 512, fixes a regression added in 25.05.2 that broke compatibility with PMIx v2.x through v3.1.0rc1, and fixes other minor to moderate bugs. The full list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Preemption Scheduler Tuning Knobs
by Reed Dier 13 Oct '25

13 Oct '25

Hoping someone may be able to help demystify some questions around the scheduler and preemption decisions. I'm trying to help better predict the scheduler behavior as it pertains to preemption to create more predictable scheduling. Version is 24.11.5, OS U22.04 I have a fairly simple lo/hi partition which the same set of nodes are assigned > PartitionName=partition-lo Nodes=foo[00..NN] Default=YES MaxTime=INFINITE OverSubscribe=FORCE:1 State=UP > PartitionName=partition-hi Nodes=foo[00..NN] Default=YES MaxTime=INFINITE OverSubscribe=NO State=UP PreemptMode=OFF And then I have 2 QOSs per partition > Name Priority GraceTime Preempt PreemptMode UsageFactor MaxJobsPU MaxTRESPA MaxJobsPA > ---------------- ---------- ---------- ---------------------- ----------- ----------- --------- ------------- --------- > qos-stateless-lo 1 00:00:00 requeue 1.000000 > qos-stateful-lo 1 00:00:00 suspend 1.000000 NN cpu=NNN NNN > qos-stateful-hi 5 00:00:00 qos-state[ful,less]-lo cluster 1.000000 cpu=NNN > qos-stateless-hi 5 00:00:00 qos-state[ful,less]-lo cluster 1.000000 And the general way that it works out is that stateful jobs will typically spawn stateless jobs. Stateful jobs will get suspended, while stateless jobs will get requeued. And then there are some general guard rails around stateful jobs clogging the queue and preventing stateless jobs from scheduling and creating a deadlock, but thats not the specific issue here. > SchedulerType=sched/backfill > SelectType=select/cons_tres > SelectTypeParameters=CR_CPU_Memory > SchedulerParameters=max_rpc_cnt=500,\ > sched_min_interval=50000,\ > sched_max_job_start=300,\ > batch_sched_delay=6 > PriorityType=priority/multifactor > PreemptType=preempt/qos > PreemptMode=SUSPEND,GANG Workload is rather "high throughput", so a few settings were influenced by that <https://slurm.schedmd.com/high_throughput.html> guide. What I end up seeing is that it can take sometimes, but not always, take a while for lo jobs to be preempted by hi jobs, an example below, but if the mailing list eats images, a link to it here: https://imgur.com/a/7xVFC8a This is a rather low resolution view, as it is just a scraper running on 5 minute increments, but the blue filled area are "hi" jobs running, where the yellow are "lo" jobs running. Second image (dashed lines) are pending jobs for the same partitions. Oddly, this specific instance did not show any preemption events in the slurmctld logs, but users/admins were a bit perplexed as to why this drug on for so long, and without preemption kicking in. I was considering looking deeper into to try to better understand and predict preemption decisions: default_queue_depth partition_job_depth sched_interval sched_min_interval defer Hopefully someone can point me to some nuggets of information around this? Appreciate any pointers, Reed

1 0

About tpology.conf
by ysaiki18＠gmail.com 12 Oct '25

12 Oct '25

Must topology.conf, like slurm.conf, be accessible to all nodes?

3 6

Prolog error causing node to drain
by Ratnasamy, Fritz 11 Oct '25

11 Oct '25

Hi, We have a running slurm cluster and users have been submitting jobs for the past 3 months without any issues. Recently, some nodes jobs are getting drained randomly due to the reason "prolog error" Our slurm.conf has these 2 lines regarding prolog: PrologFlags=Contain,Alloc,X11 Prolog=/slurm_stuff/bin/prolog.d/prolog* Inside the prolog.d folder, there are 2 scripts which run with no errors as far as I can see but is there a way to debug why the nodes are going in draining mode once in a while because of "prolog error"? That seems to happen at random times and on random nodes. From the log file, I can see only this: Oct 06 00:57:43 pgpu008.chicagobooth.edu slurmd[3709622]: slurmd: error: prolog failed: rc:230 output:Successfully started proces> Oct 06 00:57:43 pgpu008.chicagobooth.edu slurmd[3709622]: slurmd: error: [job 20398] prolog failed status=230:0 Oct 06 00:57:43 pgpu008 slurmd[3709622]: slurmd: Job 20398 already killed, do not launch batch job Oct 06 13:06:23 pgpu008 systemd[1]: Stopping Slurm node daemon... Oct 06 13:06:23 pgpu008 slurmd[3709622]: slurmd: Caught SIGTERM. Shutting down. Oct 06 13:06:23 pgpu008 slurmd[3709622]: slurmd: Slurmd shutdown completing Currently, now the job 20398 that is getting killed in the picture above is in the state "Launch failed requeue held" after I resume the node. *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Testing Slurm on FreeBSD – Offering Feedback and Collaboration
by Rikka Göring 08 Oct '25

08 Oct '25

Hello everyone, I’d like to briefly introduce myself and share some good news. I maintain the FreeBSD port of Slurm (sysutils/slurm-wlm) and have just completed the upgrade to the latest release. Alongside this, several FreeBSD-related fixes I had been maintaining locally have now been merged upstream by SchedMD engineers, which is an encouraging step for portability. With the updated port now running reliably on FreeBSD, I can offer: - Regular build and runtime testing of Slurm on FreeBSD, - Feedback on portability patches and non-Linux code paths, - Collaboration with anyone interested in running or experimenting with Slurm on BSD systems. My goal is to help ensure that Slurm remains portable and usable on FreeBSD, and to contribute back testing and fixes where possible. If you’re working on anything that may affect portability, feel free to reach out — I’d be glad to test and provide results from the FreeBSD side. Best regards, Rikka Göring

1 0

Unexpected missing socket error
by Ozeryan, Vladimir 07 Oct '25

07 Oct '25

Hello everyone, Not sure if you guys have heard this tune already but did anyone come across a solution for "Unexpected missing socket error". There is nothing useful in the logs but the message appears on compute nodes and slurm controller node. Thank you, Vlad Ozeryan AMDS - AB1 Linux-Support Vladimir.Ozeryan(a)jhuapl.edu Ext. 23966

3 2