- slurm-users - lists.schedmd.com

Slurm Community Survey 2025
by Tim Wickberg 10 Nov '25

10 Nov '25

With the Slurm 25.11 release out, and the 1.0.0rc1 for the Slinky Project done, we're quickly shifting into conference season. SchedMD staff are presenting at KubeCon North America on Slinky [1] this week. We don't have a booth here, but feel free to say hi if you see any of us, or send me a message on the Kubernetes or CNCF Slack channels (@wickberg) if you have something you'd like to discuss in person. Next week we'll be manning the Slurm Booth at SC25 [2], as well as hosting the annual Slurm Community Birds-of-a-Feather session [3] on Thursday from 12:15-1:15pm. For this year we wanted to to make the Slurm Community BoF survey available ahead of time, and to open it to a wider audience. This'll let us prepare some initial results ahead of the BoF (while still trying to update live data during the BoF), and the results from this are invaluable as we plan for future Slurm releases. The survey is available now: https://schedmd.com/survey For those not at SC25, we'll have a brief set of highlights from this survey included in the Slurm 25.11 release overview video on our YouTube channel in December. - Tim [1] https://kccncna2025.sched.com/event/27FW5/ [2] The Slurm Booth is #1641. We have a new halo banner this year as well. [3] https://sc25.conference-program.com/presentation/?id=bof101&sess=sess471 [4] https://www.youtube.com/SchedMDSlurm -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

sreport and AccountUtilizationByUser
by Gestió Servidors 08 Nov '25

08 Nov '25

Hello, I woud like to get a full report that includes CPU usage of an account (detalied by user). I have run "sreport cluster AccountUtilizationByUser start=9/1 end=11/2 account=mygr" and the output is this: -------------------------------------------------------------------------------- Cluster/Account/User Utilization 2025-09-01T00:00:00 - 2025-11-01T23:59:59 (5360400 secs) Usage reported in CPU Minutes -------------------------------------------------------------------------------- Cluster Account Login Proper Name Used Energy --------- --------------- --------- --------------- --------- -------- mycluster mygr 3979368 0 mycluster mygr mygr-1 3979368 0 mycluster mygr mygr-10 3979368 0 mycluster mygr mygr-11 3979368 0 mycluster mygr mygr-12 3979368 0 mycluster mygr mygr-13 3979368 0 mycluster mygr mygr-14 3979368 0 mycluster mygr mygr-15 3979368 0 mycluster mygr mygr-16 3979368 0 mycluster mygr mygr-17 3979368 0 mycluster mygr mygr-18 3979368 0 mycluster mygr mygr-19 3979368 0 mycluster mygr mygr-2 3979368 0 mycluster mygr mygr-20 3979368 0 mycluster mygr mygr-21 3979368 0 mycluster mygr mygr-22 3979368 0 mycluster mygr mygr-23 3979368 0 mycluster mygr mygr-24 3979368 0 mycluster mygr mygr-25 3979368 0 mycluster mygr mygr-26 3979368 0 mycluster mygr mygr-27 3979368 0 mycluster mygr mygr-28 3979368 0 mycluster mygr mygr-29 3979368 0 mycluster mygr mygr-3 3979368 0 mycluster mygr mygr-30 3979368 0 mycluster mygr mygr-31 3979368 0 mycluster mygr mygr-32 3979368 0 mycluster mygr mygr-33 3979368 0 mycluster mygr mygr-34 3979368 0 mycluster mygr mygr-35 3979368 0 mycluster mygr mygr-36 3979368 0 mycluster mygr mygr-37 3979368 0 mycluster mygr mygr-38 3979368 0 mycluster mygr mygr-39 3979368 0 mycluster mygr mygr-4 3979368 0 mycluster mygr mygr-40 3979368 0 mycluster mygr mygr-5 3979368 0 mycluster mygr mygr-6 3979368 0 mycluster mygr mygr-7 3979368 0 mycluster mygr mygr-8 3979368 0 mycluster mygr mygr-9 3979368 0 I don't understand why column "Used" shows always the same value, because each user "mygr-XX" has used the cluster in differents ways, times and each used has submitted very different number of jobs, as you can see in this "sshare" output: [root@login bin]# sshare -l -A mygr Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS -------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- mygr 1 0.250000 3941705 0.337402 0.961033 0.260137 mygr mygr-1 1 0.024390 32099 0.002748 0.008144 0.095146 2.995008 mygr mygr-10 1 0.024390 98343 0.008418 0.024949 0.067961 0.977586 mygr mygr-11 1 0.024390 213838 0.018304 0.054250 0.052427 0.449588 mygr mygr-12 1 0.024390 134149 0.011483 0.034033 0.056311 0.716657 mygr mygr-13 1 0.024390 89304 0.007644 0.022656 0.071845 1.076537 mygr mygr-14 1 0.024390 524836 0.044925 0.133150 0.040777 0.183179 mygr mygr-15 1 0.024390 303494 0.025979 0.076996 0.044660 0.316774 mygr mygr-16 1 0.024390 99020 0.008476 0.025121 0.066019 0.970897 mygr mygr-17 1 0.024390 296341 0.025366 0.075181 0.046602 0.324421 mygr mygr-18 1 0.024390 37809 0.003236 0.009592 0.085437 2.542696 mygr mygr-19 1 0.024390 72158 0.006177 0.018306 0.077670 1.332337 mygr mygr-2 1 0.024390 225658 0.019316 0.057249 0.050485 0.426039 mygr mygr-20 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-21 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-22 1 0.024390 115042 0.009847 0.029186 0.062136 0.835683 mygr mygr-23 1 0.024390 54169 0.004637 0.013743 0.081553 1.774775 mygr mygr-24 1 0.024390 125767 0.010765 0.031907 0.058252 0.764422 mygr mygr-25 1 0.024390 243088 0.020808 0.061671 0.048544 0.395490 mygr mygr-26 1 0.024390 35516 0.003040 0.009010 0.089320 2.706902 mygr mygr-27 1 0.024390 65281 0.005588 0.016562 0.079612 1.472682 mygr mygr-28 1 0.024390 118203 0.010118 0.029988 0.060194 0.813335 mygr mygr-29 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-3 1 0.024390 107543 0.009206 0.027284 0.064078 0.893956 mygr mygr-30 1 0.024390 33946 0.002906 0.008612 0.091262 2.832099 mygr mygr-31 1 0.024390 183059 0.015670 0.046442 0.054369 0.525180 mygr mygr-32 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-33 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-34 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-35 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-36 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-37 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-38 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-39 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-4 1 0.024390 37605 0.003219 0.009541 0.087379 2.556486 mygr mygr-40 1 0.024390 0 0.000000 0.000000 0.118447 inf mygr mygr-5 1 0.024390 81114 0.006943 0.020578 0.073786 1.185232 mygr mygr-6 1 0.024390 91873 0.007864 0.023308 0.069903 1.046428 mygr mygr-7 1 0.024390 80513 0.006892 0.020426 0.075728 1.194082 mygr mygr-8 1 0.024390 367822 0.031485 0.093316 0.042718 0.261374 mygr mygr-9 1 0.024390 33630 0.002879 0.008532 0.093204 2.858713 mygr mygr-tutor 1 0.024390 40472 0.003464 0.010268 0.083495 2.375427 What am I doing wrong? What I need to get is a detalied report from all "mygr" users (mygr account) from CPU usage and, if it is possible, other values. Thanks.

2 1

Slinky version 1.0.0rc1 is now now available
by Marrlow Warnicke 07 Nov '25

07 Nov '25

We are pleased to announce the availability of Slinky version 1.0.0-rc1. Slinky is SchedMD’s set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-operator/tree/release-1.0 New features include hybrid support, integration points for external tooling, and workload protection and isolation. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge handles the cases where you want Slurm scheduling on your cluster and to be able to run either Kubernetes or Slurm jobs. Release v1.0.0-rc1: https://github.com/SlinkyProject/slurm-bridge/tree/release-1.0 New features include support for DRA Extended Resources, support for TaintToleration and VolumeBinding plugins, and integration of new Slurm 25.11 support for granular node resource allocation assignments with GRES. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… The SlinkyProject registry now has containers that support both amd64 (x86_64) and arm64 (aarch64) architectures. You may find these here: https://github.com/orgs/SlinkyProject/packages -- Marlow Warnicke Principal Cloud Engineer, SchedMD LLC Commercial Slurm - and Slinky - Development and Support

1 0

submit host DNS lookup errors stall slurmctld, fail nodes, kill jobs
by moorehfl＠amazon.com 06 Nov '25

06 Nov '25

As we've scaled up our slurm usage, we've noticed that short, moderate bursts of DNS lookup failures are enough to regularly stall slurmctld: _xgetaddrinfo: getaddrinfo(runner-t35knco5d-project-54-concurrent-0:37251) failed ...this has a cascading effect where, when stalled, the controlled can't always communicate with nodes: error: Error connecting, bad data: family = 0, port = 0 ...and immediately the controller will mark the nodes as unhealthy, and kill jobs: slurmctld: Killing JobId=3120751 on failed node slurm-0f6cacdc1 The reason for the DNS failures is not unreliable DNS server or network, but rather the jobs are submitted by containers that don't have resolvable hostnames. This traditionally hasn't disrupted functionality, but we've noticed if 8-10 jobs all terminate at the same time (submitter container SIGTERMs the srun process) that the controller can be easily overloaded for several seconds, despite having significant free system resources. gdb confirms the process is hanging on DNS. We also can see "Socket timed out on send/recv operation" from clients attempting to interact with the controller during the issue. slurm 24.11.0 RHEL 8.10 kernel 4.18.0-553.58.1.el8_10.x86_64 We're looking into ways to get our ephemeral job submitter containers resolvable in DNS to prevent lookup failures (either by giving them resolvable hostnames, or by blackholing the records to 0.0.0.0 to allow for fast local failure on the slurmctld server). However, it does seem unusual for a handful of bad DNS lookups to cause so much disruption in slurmctld. Is this a known weak point of ctld? The slurmctld host is a single-purpose 16 vCPU 30GB EC2 instance with minimal load. We have ~150 nodes, all nodes have valid IPs in slurm.conf to remove the need for ctld to perform lookups for nodes, but apparently there is still a need to lookup the submit host as well, and we can reliably reproduce these cascading failures. Another possiblity might be to extend SlurmdTimeout to something very long and hope that the controller recovers from its stall in enough time to prevent from marking nodes as unhealthy and killing jobs, but it's not clear if that will have any effect since the first occurrence of "error: Error connecting, bad data: family = 0, port = 0" immediately drains nodes and kills jobs. Thanks

1 0

Slurm version 25.11 is now available
by Marshall Garey 06 Nov '25

06 Nov '25

We are pleased to announce the availability of Slurm 25.11. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md The Slurm documentation has also been updated to the 25.11 release: https://slurm.schedmd.com Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

CPU frequency setting not configured for this node
by Felix 06 Nov '25

06 Nov '25

Hello I get this error: "CPU frequency setting not configured for this node" in some of my node. I do not know what to do, can you please advice. Thank you Felix -- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

1 0

Move to higher qos when cluster idle
by Ratnasamy, Fritz 06 Nov '25

06 Nov '25

Hi, Is there an option to automatically have users move to a higher qos when the cluster is x% idle? I was reading about customizing the job_submit.lua script in order to do that but I was wondering if it makes sense. We have a gpu cluster and many users complain they can not allocate more than their current qos when the cluster is not very busy. Best, *Fritz Ratnasamy*Data Scientist Information Technology

1 0

Slurm release candidate version 25.11.0rc1 is available for testing
by Tim Wickberg 28 Oct '25

28 Oct '25

We are pleased to announce the availability of Slurm release candidate 25.11.0rc1. To highlight some new features coming in 25.11: * Added new "Expedited Requeue" mode for batch jobs. Batch jobs with --requeue=expedite will automatically requeue on node failure, or if the batch script returns a non-zero exit code and one or more Epilog scripts fail. Expedited requeue jobs are eligible to restart immediately, are treated as the highest priority job in the system, and their previously allocated set of nodes will be prevented from launching other work. * Added a new "Mode 3" of operation to Hierarchical Resources. This mode complements the existing Mode 1 and Mode 2 by summing usage from lower levels automatically. This can be used, e.g., to implement a power-capping mode modeling power distribution between the datacenter, local distribution, and individual racks. * Added direct support for exporting OpenMetrics (Prometheus) telemetry from slurmctld. This is accessible on SlurmctldPort on SlurmctldHost by default, or can be disabled if desired. * Added an experimental asynchronous-reply mode to slurmctld. If enabled with "SlurmctldParameters=enable_async_reply", RPC responses are offloaded to the kernel for further processing, freeing individual worker threads for new traffic. This is the first release candidate of the upcoming 25.11 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 25.11.x version and we will address them before the first production 25.11.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/download-slurm/. The changelog for 25.11 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-25.11.md -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

AuthInfo broken in 25.05.1 ?
by michael＠mayer.cx 27 Oct '25

27 Oct '25

I am using the AuthInfo in both slurm.conf and slurmdbd.conf to point to a different location for the munge socket. This works as expected in slurmdbd, but it does not work for slurmctld. In slurmctld it seems to ignore this setting and try to locate the munge socket at the default location and then fail. When using the same approach in 23.11.11, for example, AuthInfo is working as expected for both slurmctld and slurmdbd. Sample config from slurm.conf and slurmdbd.conf AuthType=auth/munge AuthInfo=socket=/my/non-default/location/of/munge.socket I'd be grateful for any pointer whether this is a genuine bug/regression or not. Many thanks, Michael.

3 3

Possible to require jobs to use NVLink-ed pairs of GPUs?
by Marcus Lauer 27 Oct '25

27 Oct '25

One of our researchers asked whether it was possible to require a job to use NVLink-ed pairs of GPUs. I see that there is a support ticket on the SchedMD site which covers this (https://support.schedmd.com/show_bug.cgi?id=15995). That ticket is a few years old though. Does anyone happen to know whether support for this has been added in newer releases of SLURM? The cluster in question does use "AutoDetect=nvml" in its gres.conf and the output of "slurmd -G" shows that SLURM is aware of the NVLink pairs. I assume the scheduler is trying to use that information. What I want to know is whether there is some way for an end-user to add a constraint (for example) to a job such that it only runs on an NVLink-ed pair of GPUs. I do know that there are other ways to implement this such as requiring jobs to run with even numbers of GPUs, perhaps just on some nodes to allow single GPU jobs to run on the remaining nodes. I'm specifically asking about a flag or setting a user could apply to their jobs. If there is such a thing maybe someone here knows about it. If so I'd love to hear about it. Thanks! -- *Mr. Marcus Lauer* Systems Administrator Penn Engineering University of Pennsylvania https://www.seas.upenn.edu/

1 0

Limit number of allocated GPUs
by Gestió Servidors 23 Oct '25

23 Oct '25

Hello, I have three nodes, serving each one 2 GPUs. I would like to limit (qos??) that a user could user only one GPU from earch server, but user could user simultaneously three GPUs if each GPU belongs to different servers. With this QoS "sacctmgr add qos test-limit-GPUs MaxJobsPerUser=3 MaxTRESPerUser=gres/gpu=1" I can limit to one GPU, but then user can't run other job in a GPU from other server. How must I configure QoS (or other method) to allow more than one job requesting GPUs but never in the same server? Thanks.

3 2

Job remains "PENDING" with reason "QOSMaxGRESPerUser"
by Gestió Servidors 23 Oct '25

23 Oct '25

Hello, I have added a new "qos" with these parameters: sacctmgr add qos test-GPUs MaxJobsPerUser=6 MaxTRESPerUser=gres/gpu=1 MaxSubmitJobsPerUser=25. With it, I only allow 6 running jobs per user, a total of 25 pending+running job per user and only 1 GPU. I have applied this qos directly to a partition in slurm.conf. When a user submits to that partition requesting 2 or more GPUs, job remains "PD" (pending) and notifies "QOSMaxGRESPerUser" in NODELIST column, but I would like to know if it would be possible to direcly reject job and avoid that job remains at queue? For example, if I submit 50 jobs, after number 25 I get message "sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) sbatch: error: QOSMaxSubmitJobPerUserLimit" 25 times) Thanks.

2 1

dynamic node slurm node list order
by Assmann, Greta Marie 23 Oct '25

23 Oct '25

Hello, we would like to understand how the internal SLURM node list order works. More detailed info: Our setup: We have a Slurm cluster with N dynamic nodes (heterogeneous node types) in a partition and we observe that if all resources are free, jobs always get submitted to one specific node first. This node is also the node that turns up at the top of the list when doing an $ scontrol show nodes . As we do not have a defined node list in slurm.conf as it is done for non-dynamic nodes, we were wondering how the order is set up. Interestingly, when deleting (and unregistering) this node from the cluster and re-registering it again, the node is still at the same position when doing scontrol show nodes. Is there some internal node list order caching or similar? How is the node list order defined ? Thanks , Greta Dr. Greta Assmann Data Analysis and Research Infrastructure OBBA 230 Forschungsstrasse 111 5232 Villigen-PSI email: greta.assmann(a)psi.ch

4 5

Slurm and Slinky events this fall, update events page
by Tim Wickberg 14 Oct '25

14 Oct '25

We'll have a bit more details as conference season quickly approaches this November, but SchedMD staff are presenting at KubeCon NA on Slinky [1]. We'll be manning the Slurm Booth at SC25 [2], as well as hosting the annual Slurm Community Birds-of-a-Feather session [3]. I'll also send a link out to the survey questions for the BoF to the slurm-users list ahead of the conference, and we'll be going into a bit more depth on the answers during the BoF this year. The events page on the SchedMD website has more detail on future events as well: https://www.schedmd.com/events/ A few folks had asked, and apparently we never mentioned this more publicly, but: SchedMD does not plan to hold an in-person SLUG in 2025 or 2026. We are working to bring some of the same content to our YouTube channel [4] as a way to more broadly disseminate some of that same content, starting with the Slurm 25.11 release overview in December. - Tim [1] https://kccncna2025.sched.com/event/27FW5/ [2] The Slurm Booth is #1641. [3] https://sc25.conference-program.com/presentation/?id=bof101&sess=sess471 [4] https://www.youtube.com/SchedMDSlurm -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

Slurm version 25.05.4 is now available
by Marshall Garey 14 Oct '25

14 Oct '25

We are pleased to announce the availability of Slurm version 25.05.4. This version increases the default number of maximum connections to slurmctld from 50 to 512, fixes a regression added in 25.05.2 that broke compatibility with PMIx v2.x through v3.1.0rc1, and fixes other minor to moderate bugs. The full list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Preemption Scheduler Tuning Knobs
by Reed Dier 13 Oct '25

13 Oct '25

Hoping someone may be able to help demystify some questions around the scheduler and preemption decisions. I'm trying to help better predict the scheduler behavior as it pertains to preemption to create more predictable scheduling. Version is 24.11.5, OS U22.04 I have a fairly simple lo/hi partition which the same set of nodes are assigned > PartitionName=partition-lo Nodes=foo[00..NN] Default=YES MaxTime=INFINITE OverSubscribe=FORCE:1 State=UP > PartitionName=partition-hi Nodes=foo[00..NN] Default=YES MaxTime=INFINITE OverSubscribe=NO State=UP PreemptMode=OFF And then I have 2 QOSs per partition > Name Priority GraceTime Preempt PreemptMode UsageFactor MaxJobsPU MaxTRESPA MaxJobsPA > ---------------- ---------- ---------- ---------------------- ----------- ----------- --------- ------------- --------- > qos-stateless-lo 1 00:00:00 requeue 1.000000 > qos-stateful-lo 1 00:00:00 suspend 1.000000 NN cpu=NNN NNN > qos-stateful-hi 5 00:00:00 qos-state[ful,less]-lo cluster 1.000000 cpu=NNN > qos-stateless-hi 5 00:00:00 qos-state[ful,less]-lo cluster 1.000000 And the general way that it works out is that stateful jobs will typically spawn stateless jobs. Stateful jobs will get suspended, while stateless jobs will get requeued. And then there are some general guard rails around stateful jobs clogging the queue and preventing stateless jobs from scheduling and creating a deadlock, but thats not the specific issue here. > SchedulerType=sched/backfill > SelectType=select/cons_tres > SelectTypeParameters=CR_CPU_Memory > SchedulerParameters=max_rpc_cnt=500,\ > sched_min_interval=50000,\ > sched_max_job_start=300,\ > batch_sched_delay=6 > PriorityType=priority/multifactor > PreemptType=preempt/qos > PreemptMode=SUSPEND,GANG Workload is rather "high throughput", so a few settings were influenced by that <https://slurm.schedmd.com/high_throughput.html> guide. What I end up seeing is that it can take sometimes, but not always, take a while for lo jobs to be preempted by hi jobs, an example below, but if the mailing list eats images, a link to it here: https://imgur.com/a/7xVFC8a This is a rather low resolution view, as it is just a scraper running on 5 minute increments, but the blue filled area are "hi" jobs running, where the yellow are "lo" jobs running. Second image (dashed lines) are pending jobs for the same partitions. Oddly, this specific instance did not show any preemption events in the slurmctld logs, but users/admins were a bit perplexed as to why this drug on for so long, and without preemption kicking in. I was considering looking deeper into to try to better understand and predict preemption decisions: default_queue_depth partition_job_depth sched_interval sched_min_interval defer Hopefully someone can point me to some nuggets of information around this? Appreciate any pointers, Reed

1 0

About tpology.conf
by ysaiki18＠gmail.com 12 Oct '25

12 Oct '25

Must topology.conf, like slurm.conf, be accessible to all nodes?

3 6

Prolog error causing node to drain
by Ratnasamy, Fritz 11 Oct '25

11 Oct '25

Hi, We have a running slurm cluster and users have been submitting jobs for the past 3 months without any issues. Recently, some nodes jobs are getting drained randomly due to the reason "prolog error" Our slurm.conf has these 2 lines regarding prolog: PrologFlags=Contain,Alloc,X11 Prolog=/slurm_stuff/bin/prolog.d/prolog* Inside the prolog.d folder, there are 2 scripts which run with no errors as far as I can see but is there a way to debug why the nodes are going in draining mode once in a while because of "prolog error"? That seems to happen at random times and on random nodes. From the log file, I can see only this: Oct 06 00:57:43 pgpu008.chicagobooth.edu slurmd[3709622]: slurmd: error: prolog failed: rc:230 output:Successfully started proces> Oct 06 00:57:43 pgpu008.chicagobooth.edu slurmd[3709622]: slurmd: error: [job 20398] prolog failed status=230:0 Oct 06 00:57:43 pgpu008 slurmd[3709622]: slurmd: Job 20398 already killed, do not launch batch job Oct 06 13:06:23 pgpu008 systemd[1]: Stopping Slurm node daemon... Oct 06 13:06:23 pgpu008 slurmd[3709622]: slurmd: Caught SIGTERM. Shutting down. Oct 06 13:06:23 pgpu008 slurmd[3709622]: slurmd: Slurmd shutdown completing Currently, now the job 20398 that is getting killed in the picture above is in the state "Launch failed requeue held" after I resume the node. *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Testing Slurm on FreeBSD – Offering Feedback and Collaboration
by Rikka Göring 07 Oct '25

07 Oct '25

Hello everyone, I’d like to briefly introduce myself and share some good news. I maintain the FreeBSD port of Slurm (sysutils/slurm-wlm) and have just completed the upgrade to the latest release. Alongside this, several FreeBSD-related fixes I had been maintaining locally have now been merged upstream by SchedMD engineers, which is an encouraging step for portability. With the updated port now running reliably on FreeBSD, I can offer: - Regular build and runtime testing of Slurm on FreeBSD, - Feedback on portability patches and non-Linux code paths, - Collaboration with anyone interested in running or experimenting with Slurm on BSD systems. My goal is to help ensure that Slurm remains portable and usable on FreeBSD, and to contribute back testing and fixes where possible. If you’re working on anything that may affect portability, feel free to reach out — I’d be glad to test and provide results from the FreeBSD side. Best regards, Rikka Göring

1 0

Unexpected missing socket error
by Ozeryan, Vladimir 07 Oct '25

07 Oct '25

Hello everyone, Not sure if you guys have heard this tune already but did anyone come across a solution for "Unexpected missing socket error". There is nothing useful in the logs but the message appears on compute nodes and slurm controller node. Thank you, Vlad Ozeryan AMDS - AB1 Linux-Support Vladimir.Ozeryan(a)jhuapl.edu Ext. 23966

3 2

Setting GrpTRES for specific Account only for specific Partition(s)
by Paul Raines 06 Oct '25

06 Oct '25

I am trying to figure out how one can limit resources like with GrpTRES for specific accounts only for specific partitions in the cluster Lets say I have a cluster with 3 partitions Part1, Part2, and Part3 and 3 accounts Acct1, Acct2, and Acct3 I want Acct1 to be limited to 10 cpus on Part1, Acct2 to be limited to 5 cpus on Part1, and Acct3 to be limited to 8 cpus on Part1. No limits for any account on the other two partitions at present. I have tried # sacctmgr modify account Acct1 where partition=Part1 set GrpTRES=cpu=10 Modified account associations... Nothing modified I have tried creating a QOS to assign # sacctmgr add qos cpus10 # sacctmgr modify qos cpus10 set GrpTRES=cpu=10 # sacctmgr modify account Acct1 where partition=Part1 set qos=cpus10 Modified account associations... Nothing modified I cannot set a Partition QOS since that would apply to all accounts on the partition giving them the same limit I cannot just set a general Account limit (or QOS) since that would apply to all Partitions. I really don't want to have to have different Accounts for each partition so I can set Account wide limits. That would make users have to remember which Account to use with which Partiton. It looks like the following works but requires having to repeat for all users in every Account sacctmgr -i add user User1 account=Acct1 partition=Part1 \ fairshare=parent qos=cpus10 # limits on the Acct1 partition sacctmgr -i add user User1 account=Acct1 \ fairshare=parent # no limits on any other partition But in essense is not much better than having different Accounts for each partition as I now have to maintain multiple user entries per Account: one for partitios with no-limits and and additional one for each partition that needs limits. I guess at least users don't have to remember what account to use when submitting to a partition. Its seems there should be some simpler way to do this. A way to say "this is the limit for this Account on this Partion for all current users and any future users added to the Account" Am I missing something? The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

cloud compute limits/budget
by Tom Sparks 03 Oct '25

03 Oct '25

I am setting up a single user slurm I have limited budget for cloud gpu compute ($40) I am wondering if slurm has ability to stop me from going over my budget? I have read the slurm's powersaving / configureless nodes guides Are they other guides I should look at?

2 1

SRUN and SBATCH network issues on configless login node.
by Bruno Bruzzo 01 Oct '25

01 Oct '25

Hi all, first of all, sorry for my English, it's not my native language. We are currently experiencing an issue with srun and salloc on our login nodes, while sbatch works properly. slurm version 23.11.4. slurmctld runs on management node mmgt01. srun and salloc fail intermittently on login node, that means we can successfully use srun on login node from time to time, but it stops working for a while without us changing any configuration. login nodes reports the following to the user: $ srun -N 1 -n 64 --partition=gpunode --pty /bin/bash srun: job 4872 queued and waiting for resources srun: error: Security violation, slurm message from uid 202 srun: error: Security violation, slurm message from uid 202 srun: error: Security violation, slurm message from uid 202 srun: error: Task launch for StepId=4872.0 failed on node cn065: Invalid job credential srun: error: Application launch failed: Invalid job credential srun: Job step aborted uid 202 is the slurm user. On the server side, slurmctld logs show: sched: _slurm_rpc_allocate_resources JobId=4872 NodeList=(null) usec=228 sched: Allocate JobId=4872 NodeList=cn065 #CPUs=64 Partition=gpunode error: slurm_receive_msgs: [[snmgt01]:38727] failed: Zero Bytes were transmitted or received Killing interactive JobId=4872: Communication connection failure _job_complete: JobId=4872 WEXITSTATUS 1 _job_complete: JobId=4872 done step_partial_comp: JobId=4872 StepID=0 invalid; this step may have already completed _slurm_rpc_complete_job_allocation: JobId=4872 error Job/step already completing or completed We suspect it is a network issue regarding the Zero Bytes were transmitted or received. The configless system is working properly. Slurmd on login node can read changes made at slurm.conf after a scontrol reconfig. srun runs successfully from management nodes and from compute nodes. The issue is from the login node. scontrol ping always shows DOWN from login node, even when we can successfully run srun or salloc. $ scontrol ping Slurmctld(primary) at mmgt01 is DOWN We checked as well for munge consistency. mmgt and login nodes have the hostnames of their respective others on /etc/hosts. They can communicate. We would really appreciate some tips on what we could be missing. Best regards, Bruno Bruzzo System Administrator - Clementina XXI

4 7

Re: How to make TLS and PMIx v4 work together?
by Grigory Shamov 30 Sep '25

30 Sep '25

Forgot to add: the s2n-tls comes from EPEL and is ver 1.5.10. On 2025-09-25, 11:56 AM, "Grigory Shamov via slurm-users" <slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com>> wrote: Caution! This message was sent from outside the University of Manitoba. Hi All, We have updated SLURM to the current 25.05.x and tried to enable TLS on it. The OS is Alma 8.10, cgroups v1, and PMIx v 4. We see that srun fails for MPI jobs across the nodes, with TLS related errors when using PMIx (the default) but passes with srun --mpi=pmi2 or with mpirun . TLSType = tls/s2n TLSParameters = ca_cert_file= (has all the certs here under /etc/slurm/certs) And the errors when using PMIx are 025-09-25T11:04:43.894] error: con_close_on_poll_error: [n388:6818(fd:15)] socket error encountered while polling: Connection reset by peer [2025-09-25T11:04:50.102] [6451416.0] error: _negotiate: s2n_negotiate() failed S2N_ERR_CERT_UNTRUSTED[335544366]: Certificate is untrusted -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/tls/s2n_x509_validator.c:494 (couple of these) [2025-09-25T11:05:57.878] [6451416.0] error: tls_p_recv: s2n_recv() failed S2N_ERR_CLOSED[134217728]: connection is closed -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:37 [2025-09-25T11:05:57.883] [6451416.0] error: tls_p_send: s2n_send() failed S2N_ERR_IO[67108864]: underlying I/O operation failed, check system errno -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:28 (couple of these) [2025-09-25T11:05:59.076] error: wrap_on_data: [unix:/var/spool/slurmd/slurmd.socket(fd:17)] on_data returned rc: Unable to proxy slurmstepd message [2025-09-25T11:05:59.076] [6451416.0] error: _stepd_send_recv_msg: slurmd was unable to proxy request message to its final destination [2025-09-25T11:05:59.878] error: _slurmd_send_recv_msg: Failed to send/recv slurmstepd message MESSAGE_TASK_EXIT using proxy_type PROXY_TO_NODE_SEND_RECV 2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: pmixp_p2p_send: n388 [0]: pmixp_utils.c:469: send failed, rc=1001, exceeded the retry limit [2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: _slurm_send: n388 [0]: pmixp_server.c:1586: Cannot send message to /var/spool/slurmd/stepd.slurm.pmix.6451416.0, size = 27679, hostlist: (null) (and couple more PMIx errors). Looks like PMIx cannot talk to their peers now ? There was no specific configuration for the certgen plugin, because SLURM documentation seems to say it is optional(?). I wonder what do we miss here to have SLURM 25.05 in with TLS enabled and PMIx working? Any advice appreciated! Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com>

3 3

FUTURE nodes do not return to idle on slurmctld restart
by Steve Kirk 26 Sep '25

26 Sep '25

Hello, We've got a few nodes defined in our slurm.conf in 'FUTURE' state as it's a new hardware type we're working on brining into service. The nodes are currently all allocated to a dedicated partition. The partition is configured as 'state=UP'. As we've built the new nodes and started slurmd+munge, they've appeared in an idle state in the new partition as expected. All good so far. However if the slurmctld is restarted the nodes go back to being in 'FUTURE' state, and do not transition to idle, accept jobs etc. The slurm daemon on the new nodes can clearly still talk to the slurmctld, s* commands on the new nodes work as expected but remain in FUTURE state - until slurmd on each node is restarted. I could have misunderstood something about the FUTURE state but I was expecting them to go back to idle; I understand that slurmctld doesn't communicate out to nodes in FUTURE state but I at least expected them to be picked up when they communicate _in_ to the slurmctld. Is this expected behaviour or perhaps a bug? The reason I've defined the new nodes this way so I don't have to update slurm.conf and restart slurmctld as each is built, but can do that as a single job once everything is finished, however it seems less useful if they can 'disappear' from the cluster as far as users are concerned. Cheers, Steve

2 2

How to make TLS and PMIx v4 work together?
by Grigory Shamov 25 Sep '25

25 Sep '25

Hi All, We have updated SLURM to the current 25.05.x and tried to enable TLS on it. The OS is Alma 8.10, cgroups v1, and PMIx v 4. We see that srun fails for MPI jobs across the nodes, with TLS related errors when using PMIx (the default) but passes with srun --mpi=pmi2 or with mpirun . TLSType = tls/s2n TLSParameters = ca_cert_file= (has all the certs here under /etc/slurm/certs) And the errors when using PMIx are 025-09-25T11:04:43.894] error: con_close_on_poll_error: [n388:6818(fd:15)] socket error encountered while polling: Connection reset by peer [2025-09-25T11:04:50.102] [6451416.0] error: _negotiate: s2n_negotiate() failed S2N_ERR_CERT_UNTRUSTED[335544366]: Certificate is untrusted -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/tls/s2n_x509_validator.c:494 (couple of these) [2025-09-25T11:05:57.878] [6451416.0] error: tls_p_recv: s2n_recv() failed S2N_ERR_CLOSED[134217728]: connection is closed -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:37 [2025-09-25T11:05:57.883] [6451416.0] error: tls_p_send: s2n_send() failed S2N_ERR_IO[67108864]: underlying I/O operation failed, check system errno -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:28 (couple of these) [2025-09-25T11:05:59.076] error: wrap_on_data: [unix:/var/spool/slurmd/slurmd.socket(fd:17)] on_data returned rc: Unable to proxy slurmstepd message [2025-09-25T11:05:59.076] [6451416.0] error: _stepd_send_recv_msg: slurmd was unable to proxy request message to its final destination [2025-09-25T11:05:59.878] error: _slurmd_send_recv_msg: Failed to send/recv slurmstepd message MESSAGE_TASK_EXIT using proxy_type PROXY_TO_NODE_SEND_RECV 2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: pmixp_p2p_send: n388 [0]: pmixp_utils.c:469: send failed, rc=1001, exceeded the retry limit [2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: _slurm_send: n388 [0]: pmixp_server.c:1586: Cannot send message to /var/spool/slurmd/stepd.slurm.pmix.6451416.0, size = 27679, hostlist: (null) (and couple more PMIx errors). Looks like PMIx cannot talk to their peers now ? There was no specific configuration for the certgen plugin, because SLURM documentation seems to say it is optional(?). I wonder what do we miss here to have SLURM 25.05 in with TLS enabled and PMIx working? Any advice appreciated! Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

1 0

Node switching randomly to down state
by Julien Tailleur 24 Sep '25

24 Sep '25

Dear all, I am maintaining a small computing cluster and I have a weird behavior that I fail at debugging. My cluster comprise one master node and 16 computing servers, organized in two queues, each queue having 8 servers. All servers run up-to-date Debian bullseye. All but 3 servers work flawlessly. From the master node, I can see that 3 servers on one of the queue appear down: jtailleu@kandinsky:~$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 down* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 These servers are reachable by SSH/ping jtailleu@kandinsky:~$ ping -c 1 FX12 PING FX12 (192.168.6.22) 56(84) bytes of data. 64 bytes from FX12 (192.168.6.22): icmp_seq=1 ttl=64 time=0.070 ms --- FX12 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.070/0.070/0.070/0.000 ms ##### I can also put these nodes back into idle mode: root@kandinsky:~# scontrol update nodename=FX[12-14] state=idle root@kandinsky:~# sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 idle* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 But then, they switch back into down mode few minutes later: root@kandinsky:~# sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 down* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 root@kandinsky:~# sinfo -R REASON USER TIMESTAMP NODELIST Not responding slurm 2025-09-08T15:04:39 FX[12-14] I do not understand where the "not responding" comes from, nor how I can investigate that. Any idea what could trigger this behavior? Best wishes, Julien

4 4

No output and can't job by id
by Dhumal, Dr. Nilesh 22 Sep '25

22 Sep '25

Hello, Recently, we updated Slurm. We are running slrumcld on head node. The slurmdbd is disabled on the head node. The user submits the job and gets the job ID. The squeue is empty, and difficult to track the job progress. The job is not writing any output to a file. How can I resolve this issue? Nilesh Dhumal Associate Professor of Chemistry, http://faculty.fgcu.edu/ndhumal/ Coordinator, FGCU Computational Facility, https://www.fgcu.edu/cas/facultyresources/computationalfacility/ SH-430; Department of Chemistry and Physics Florida Gulf Coast University 10501 FGCU Boulevard South Fort Myers, FL 33965-6565 Phone: (239) 745-4394 Email: ndhumal(a)fgcu.edu

2 2

Node in drain state
by Gestió Servidors 22 Sep '25

22 Sep '25

Hello, I have got a node in "drain" state after finishing a job that was running on it. Log in node reports this information: [...] [2025-09-07T11:09:26.980] task/affinity: task_p_slurmd_batch_request: task_p_slurmd_batch_request: 59238 [2025-09-07T11:09:26.980] task/affinity: batch_bind: job 59238 CPU input mask for node: 0xFFF [2025-09-07T11:09:26.980] task/affinity: batch_bind: job 59238 CPU final HW mask for node: 0xFFF [2025-09-07T11:09:26.980] Launching batch job 59238 for UID 21310 [2025-09-07T11:09:27.006] cred/munge: init: Munge credential signature plugin loaded [2025-09-07T11:09:27.007] [59238.batch] debug: auth/munge: init: loaded [2025-09-07T11:09:27.009] [59238.batch] debug: Reading cgroup.conf file /soft/slurm-23.11.0/etc/cgroup.conf [2025-09-07T11:09:27.025] [59238.batch] debug: cgroup/v1: init: Cgroup v1 plugin loaded [2025-09-07T11:09:27.025] [59238.batch] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: core enforcement enabled [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: device enforcement enabled [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2025-09-07T11:09:27.026] [59238.batch] task/affinity: init: task affinity plugin loaded with CPU mask 0xfff [2025-09-07T11:09:27.027] [59238.batch] debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded [2025-09-07T11:09:27.027] [59238.batch] topology/default: init: topology Default plugin loaded [2025-09-07T11:09:27.030] [59238.batch] debug: gpu/generic: init: init: GPU Generic plugin loaded [2025-09-07T11:09:27.031] [59238.batch] debug: laying out the 12 tasks on 1 hosts clus09 dist 2 [2025-09-07T11:09:27.031] [59238.batch] debug: close_slurmd_conn: sending 0: No error [2025-09-07T11:09:27.031] [59238.batch] debug: Message thread started pid = 910040 [2025-09-07T11:09:27.031] [59238.batch] debug: Setting slurmstepd(910040) oom_score_adj to -1000 [2025-09-07T11:09:27.031] [59238.batch] debug: spank: opening plugin stack /soft/slurm-23.11.0/etc/plugstack.conf [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: job abstract cores are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: step abstract cores are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: job physical CPUs are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: step physical CPUs are '0-11' [2025-09-07T11:09:27.090] [59238.batch] debug levels are stderr='error', logfile='debug', syslog='fatal' [2025-09-07T11:09:27.090] [59238.batch] starting 1 tasks [2025-09-07T11:09:27.090] [59238.batch] task 0 (910044) started 2025-09-07T11:09:27 [2025-09-07T11:09:27.098] [59238.batch] debug: task/affinity: task_p_pre_launch: affinity StepId=59238.batch, task:0 bind:mask_cpu [2025-09-07T11:09:27.098] [59238.batch] _set_limit: RLIMIT_NPROC : reducing req:255366 to max:159631 [2025-09-07T11:09:27.398] [59238.batch] task 0 (910044) exited with exit code 2. [2025-09-07T11:09:27.399] [59238.batch] debug: task/affinity: task_p_post_term: affinity StepId=59238.batch, task 0 [2025-09-07T11:09:27.399] [59238.batch] debug: signaling condition [2025-09-07T11:09:27.399] [59238.batch] debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded [2025-09-07T11:09:27.400] [59238.batch] debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded [2025-09-07T11:09:27.400] [59238.batch] job 59238 completed with slurm_rc = 0, job_rc = 512 [2025-09-07T11:09:27.410] [59238.batch] debug: Message thread exited [2025-09-07T11:09:27.410] [59238.batch] stepd_cleanup: done with step (rc[0x200]:Unknown error 512, cleanup_rc[0x0]:No error) [2025-09-07T11:09:27.411] debug: _rpc_terminate_job: uid = 1000 JobId=59238 [2025-09-07T11:09:27.411] debug: credential for job 59238 revoked [...] "sinfo" shows: [root@login-node ~]# sinfo PARTITION TIMELIMIT AVAIL STATE NODELIST CPU_LOAD NODES(A/I) NODES(A/I/O/T) CPUS CPUS(A/I/O/T) REASON node.q* 4:00:00 up drained clus09 0.00 0/0 0/0/1/1 12 0/0/12/12 Kill task faile node.q* 4:00:00 up allocated clus[10-11] 13.82-15.8 2/0 2/0/0/2 12 24/0/0/24 none node.q* 4:00:00 up idle clus[01-06,12] 0.00 0/7 0/7/0/7 12 0/84/0/84 none But it seems there is no error in node... Slurmctld.log in server seems correct, too. Is there any way to reset node to "state=idle" after errors in the same way? Thanks.

3 4

Compute node not responding
by Dhumal, Dr. Nilesh 21 Sep '25

21 Sep '25

Hello, Recently, we installed slum 25 on our cluster. We are not monitoring the user's account. We didn't configure the sql database on the head node. We are running slurmcld on head node and slumd on the compute node. We are getting the following error Head node: compute node not responding. Compute node: 2025-09-19T15:30:23.461] error: Unable to register: Unable to contact slurm controller (connect failure) Do we need to run slumdbd on the head node? I checked the network connection by pinging the compute node from the head node. Do you have any suggestions to resolve this issue? Thanks Nilesh Get Outlook for Android<https://aka.ms/AAb9ysg>

4 5

Re: Node in drain state
by Gestió Servidors 19 Sep '25

19 Sep '25

Hi, After reading answer from Ole Holm Nielsen, I have increased "MessageTimeout" to 20s (by default is 5s) and "UnkillableStepTimeout" to 150s (by default is 60s and, always 5 times larger than "MessageTimeout"). However, I have also read that UnkillableStepProgram indicates the program to use in that cases... but, by default there is no program assigned to that parameter (no program to run). So my question is if someone uses a customized "UnkillableStepProgram" and if he/she could explain that. Thanks a lot!

3 2

seff for GPU
by Josu Lazkano Lete 18 Sep '25

18 Sep '25

Hello, We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff? It will be great to know how much GPU resources the users request and compare with how much GPU resources they use. Kind regards. [image: Vicomtech] <https://www.vicomtech.org> Josu Lazkano Lete Systems Manager Infrastructures and General Services jlazkano(a)vicomtech.org +(34) 943 30 92 30 The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it. Avoid printing this message if it is not strictly necessary.

7 9

Scheduling issues with multiple different types of GPU in one partition
by Kevin M. Hildebrand 17 Sep '25

17 Sep '25

We have several different types of GPUs in the same 'gpu' partition. The problem we're having occurs when one of those types of GPUs is fully occupied and there are a bunch of queued jobs waiting for those GPUs. If someone requests idle GPUs of a different type, those jobs end up getting stalled, even though there are plenty of GPUs available. For example, say we have 10 A100 GPUs and 10 H100 GPUs. If there are 10 H100 GPU jobs running and more in queue waiting for them, subsequently submitted A100 jobs will sit in queue even if there are plenty of idle A100 GPUs. The only way we can get the A100 jobs to run is by manually bumping their priority higher than the pending H100 jobs. Has anyone else encountered this issue? The only way we can think of to potentially solve it is to have separate partitions for each GPU type, but that seems unwieldy. We are currently running Slurm 24.05.8. Thanks, Kevin -- Kevin Hildebrand Director of Research Technology and HPC Services Division of IT

7 7

New "NOT-state" selection of the sinfo command in Slurm 25.05
by Ole Holm Nielsen 10 Sep '25

10 Sep '25

We just upgraded Slurm to 25.05.3, and I would like to highlight a new functionality of the "sinfo -t <state>" command in 25.05: > The state can be prefixed with '~' which will invert the result of match. We find this "NOT-state" selection useful together with Slurm power saving [1] where any idle nodes may be powered off to save electrical power or cloud costs. For example, the ClusterShell [2] clush command can now skip connections to the powered down nodes like in this example: $ clush -bw@slurmstate:~POWERED_DOWN uname -r You can even configure ClusterShell so that "clush -a" includes all nodes which are *not* powered down. The details are described in the Wiki page [3]. Best regards, Ole [1] https://slurm.schedmd.com/power_save.html [2] https://clustershell.readthedocs.io/en/latest/intro.html [3] https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_operations/#skipping-powere… -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

1 0

Development RPMs for cgroups v2
by John Hearns 10 Sep '25

10 Sep '25

I am building a version of Slurm on RHEL 9.4 When I run an rpmbuild, he slurm rpm does not contain /usr/lib64/slurm/cgroup_v2.so I have tried to look in the build logs. I suspect I am lacking some development RPMs - but which ones? All hints gratefully received John H

4 5

NHC skips nodes in "mix-" state
by blines＠tgen.org 08 Sep '25

08 Sep '25

The node-mark-offline script skips nodes in the "mix-" state with the error: State "mix-" not yet handled; ignoring. "mix-" seems to be a valid state indicating that a node is mixed and is "planned by the backfill scheduler for a higher priority job". Anyone know the reason that nodes in this state are being skipped? The responsible code appears to be: case "$STATUS" in *'@'*|*'#'*|boot*|*-*|plnd*) # These states aren't handled yet. echo "$0: State \"$STATUS\" not yet handled; ignoring." exit 0 As far as I can tell all the states being skipped are related to nodes being configured or rebooted or powering on/off except *-*. There are other states related to node power that include underscore in the state name; power_down, power_up, pow_dn, pow_up. Should the case statement be *_* instead of *-*?

1 0

Creating /run/user/$UID - for Podman runtime
by John Snowdon 08 Sep '25

08 Sep '25

We are in the middle of implementing an extensive range of container support on our new HPC platform and have decided to offer our users a wide suite of technologies to better support their workloads: * Apptainer * Podman (rootless) * Docker (rootless) We've already got a solution for automated entries in /etc/subuid and /etc/subgid on the head nodes (available here under GPL: https://github.com/megatron-uk/pam_subid), which is where we intend users to build their container images, and building and running containers using Apptainer and Podman in those environments works really well - we're happy that it should take care of 95% of our users needs (Docker is the last few percent....) and not involve giving them any special permissions. If I ssh directly to a compute node, then Podman also works there to run an existing image (podman container run ...). What I'm struggling with now is running Podman under Slurm itself on our compute nodes. It appears as though Podman (in rootless mode) wants to put the majority of its run time / state information under /run/user/$UID ... this is fine on the head nodes which have interactive logins hitting PAM modules which instantiate the /run/user/$UID directories, but not under sbatch/srun which doesn't create them by default. I've not been able to find a single, magical setting which will move all of the Podman state information out from /run/user to another location - there are 3 or 4 settings involved, and even then I still find various bits of Podman want to create stuff under there. Rather than hacking away at getting Podman changed to move all settings and state information elsewhere, it seems like the cleanest solution would just be to put the regular /run/user/$UID directory in place at the point Slurm starts the job instead. What's the best way to get Slurm to create this and clean-up afterwards? Should this be in a prolog/epilog wrapper (e.g. directly calling loginctl) or is it cleaner to get Slurm to trigger the usual PAM session machinery in some manner? John Snowdon Senior Research Infrastructure Engineer (HPC) Research Software Engineering Catalyst Building, Room 2.01 Newcastle University 3 Science Square Newcastle Helix Newcastle upon Tyne NE4 5TG https://hpc.researchcomputing.ncl.ac.uk

5 7

Slurm version 25.05.3 is now available
by Marshall Garey 04 Sep '25

04 Sep '25

We are pleased to announce the availability of Slurm version 25.05.3. This version fixes an issue that prevented deleting a QOS when running with MySQL servers (MariaDB is was unaffected). Please note that the slurmdbd will require MySQL 8.0.4+ or MariaDB 10.0.5+ to function correctly. This version also fixes heterogeneous jobs when TLS is enabled, a logging issue with syslog, and various mild to moderate stability fixes. The full list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Discussion about the --cpus-per-task flag
by Michele Esposito Marzino 04 Sep '25

04 Sep '25

Hi everyone, I would like to ask about the rationale behind the --cpus-per-task flag. While I do understand the meaning and its intended goal, I was wondering how it is managed in practice. I tried inferring this from the source code, but it is quite intricate, and not really straightforward by simply searching for the string match. Does it affect the CPUSET of each task? Who is responsible for setting this? The slurmcltd or slurmd? And how? Or at least, point me where in the code it is handled, so that I can reconstruct the whole process myself. Thanks

1 0

sacctmgr output with and without associations
by Rémy Dernat 04 Sep '25

04 Sep '25

Hi, I have a stupid issue about sacctmgr output and associations. I would like to retrieve my accounts with all the following data : "Account","Clusters","DefaultQOS","Description","Fairshare","Flags","GrpTRESMins","GrpTRES","GrpJobs","GrpSubmitJob",GrpWall", "MaxTRESMins","MaxTRES","MaxJobs","MaxNodes","MaxSubmitJobs","MaxWall","Organization","ParentName","Priority","QosLevel" So my command is the following : sacctmgr list accounts --assoc format="Account","Clusters","DefaultQOS","Description","Fairshare","Flags","GrpTRESMins","GrpTRES","GrpJobs","GrpSubmitJob","GrpWall","MaxTRESMins","MaxTRES","MaxJobs","MaxNodes","MaxSubmitJobs","MaxWall","Organization","ParentName","Priority","QosLevel" -n -P However, I have a lot of outputs corresponding to each User association entry. That would be fine if a basic `|uniq` would remove everything I don't need, but I am stuck with grptres, which can be seen in the main account, but not in the children associations with the Users. On the opposite, if I remove `--assoc`, I don't have anymore grptres, ParentName, or related Qos values. I had to add `where User=""` to my first command to just retrieve my main accounts, without User associations. I dont' know if that is clear enough; is there something I missed with the sacctmgr output ? Thanks, Best regards, Rémy

2 1

Can't remove account
by Matthias Leopold 01 Sep '25

01 Sep '25

Hi, I have a cluster with account foo. I want to delete this account. For every user that had associations in this account I created new associations in a different account and changed the DefaultAccount of the user. Then I deleted all associations in account foo. Now I want to delete account foo: sacctmgr delete account name=foo This gives: Error with request: You can not remove the default account of a user ... Please either remove the accounts listed above from list and resubmit, or change these users' default accounts to remove the account(s). Changes Discarded It names all associations for users that FORMERLY had DefaultAccount foo?? NO user has DefaultAccount foo anymore... I have to mention that I also have partition foo and affected users have an association in their new account with partition foo. But that shouldn't matter IMHO Has anybody seen this? What can I do? Am I getting something wrong? Slurm version is 24.05.7 thanks Matthias

1 0

slurmrestd unable to find plugin rest_auth/jwt
by Jon Marshall 28 Aug '25

28 Aug '25

Hi, I feel like I must have missed something obvious here, so hopefully someone can spot it! We recently updated to Slurm 24.11.6 on our cluster and I made sure to compile --with slurmrestd and --with jwt, as there are a few things I've been meaning to try out with the REST API. From the rpmbuild config.log I get: $ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc/slurm --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-systemdsystemunitdir=/usr/lib/systemd/system --enable-pkgconfig --with-jwt -- configure:21020: checking for jwt.h configure:21020: gcc -c -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer conftest.c >&5 configure:21020: $? = 0 configure:21020: result: yes configure:21028: checking for jwt_add_header in -ljwt configure:21051: gcc -o conftest -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -Wl,-z,relro -Wl,-z,lazy conftest.c -ljwt -ldl -lrt -lm -lresolv >&5 configure:21051: $? = 0 configure:21061: result: yes -- config.status:1746: creating src/plugins/auth/jwt/Makefile -- config.status:1746: creating src/slurmrestd/plugins/auth/jwt/Makefile -- ac_cv_header_jwt_h=yes -- ac_cv_lib_jwt_jwt_add_header=yes -- JWT_LDFLAGS='-ljwt' And: configure:26873: checking whether to compile slurmrestd configure:26904: result: yes -- configure:26915: checking for slurmrestd default port configure:26933: result: 6820 And indeed I get a slurmrestd package out of it, which I've successfully installed with no errors. I've created a JWT key and rolled it out to the controller node, but when I go to check auth types I get: slurmrestd -a rest_auth/jwt slurmrestd: fatal: Unable to find plugin: rest_auth/jwt Running an strace on this process shows that it does in fact stat the .so that does in fact exist: stat("/usr/lib64/slurm/auth_jwt.so", {st_mode=S_IFREG|0755, st_size=283728, ...}) = 0 And I'm kind of stumped now. I've checked through all the resolved bug tickets to see if I can spot what is wrong but I'm getting nothing, and I can't find anything in the mailing lists either, other than things I've checked. Has anyone else had similar issues, or can someone see the blindingly obvious thing I'm missing? Cheers Jon Jon Marshall High Performance Computing Specialist IT and Scientific Computing Team Cancer Research UK Cambridge Institute Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE Web<http://www.cruk.cam.ac.uk/> | Facebook<http://www.facebook.com/cancerresearchuk> | Twitter<http://twitter.com/CR_UK> [Description: CRI Logo]<http://www.cruk.cam.ac.uk/>

2 4

Assistance with Node Restrictions and Priority for Users in Floating Partition
by Manisha Yadav 28 Aug '25

28 Aug '25

Dear Team, I have a scenario where I need to provide priority access to multiple users from different projects for only 3 nodes. This means that, at any given time, only 3 nodes can be used in that partition, and if one user is utilizing all 3 nodes, no other user should be able to submit jobs to that partition, or their jobs should remain in the queue. To achieve this, I attempted to use QoS by creating a floating partition with some of the nodes and configuring a QoS with priority. I also set a limit with GrpTRES=gres/gpu=24, given that each node has 8 GPUs, and there are 3 nodes in total. I then attached the QoS to the partition and assigned it to the users who need access. I Also tried MaxTRES=gres/gpu=24 While this setup works as expected in the testing environment for CPUs, it is not functioning as intended in production, and it is not effectively restricting node usage in the partition. Could anyone provide suggestions or guidance on how to properly implement node restrictions along with priority? Thank you for your assistance. Best regards, Manisha Yadav ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

4 8

Node Health Check Program
by Paul Edmon 27 Aug '25

27 Aug '25

We've been using NHC (https://github.com/mej/nhc) for years with much success. However that project hasn't had a release in 2 years and the various Issues filed indicate that there might be problems with Rocky 9 (which we are looking to upgrade to). Do people that are at EL9 use NHC? Is there a fork? Is there a different code that people use for doing node health checks? -Paul Edmon-

8 16

Slurm 25.05: Retrieving jobs GPU Indices on Heterogeneous Cluster
by David Gauchard 27 Aug '25

27 Aug '25

Hello, I'm running Slurm 25.05 on a heterogeneous cluster (several kind of GPU in the same node) with AutoDetect=nvml and shared mode. When submitting a job with `#SBATCH --gres=gpu:1`, CUDA_VISIBLE_DEVICES is correctly set to a single and valid free GPU index but `scontrol show jobid` does not report any detail about GPU allocation. How can I retrieve the GPU indices assigned to running jobs (reflecting CUDA_VISIBLE_DEVICES) in shared mode? Is there a Slurm command or configuration to enable tracking of these indices on a heterogeneous cluster? The goal is to help automatic choice of the free and appropriate gpu according to the job's needs in order to save bigger gpus for bigger jobs at the time of submitting. Thanks

2 2

Issues with dynamic configless nodes and pam_slurm_adopt
by Leonardo Sala 26 Aug '25

26 Aug '25

Hallo everyone we have recently noticed that when running nodes in configless and dynamic mode, pam_slurm_adopt does not work properly, denying ssh access when the user has a running job. We traced down the issue to the fact that the dynamic nodes are registered with FQDN (e.g. c01.psi.ch), while pam_slurm_adopt uses this routine (https://github.com/SchedMD/slurm/blob/master/src/common/stepd_api.c#L181) to guess the stepd hostname, using the host short name (so, c01). If we use gethostname instead of gethostname_short then it works again. Has anybody experience with this, and is there a way to have dynamic configless nodes registering with the short hostname? Thanks! cheers leo -- Paul Scherrer Institut Dr. Leonardo Sala Group Leader Data Analysis and Research Infrastructure Group Leader Data Curation a.i. Deputy Department Head Science IT Infrastructure and Services department Science IT Infrastructure and Services department (AWI) OBBA/230 Forschungstrasse 111 5232 Villigen PSI Switzerland Phone: +41 56 310 3369 leonardo.sala(a)psi.ch www.psi.ch

1 0

Tips or experiences with Burst Buffers?
by Bjørn-Helge Mevik 21 Aug '25

21 Aug '25

Dear all! We are currently considering setting up the (Lua) Burst Buffer plugin to stage in and out files. Those of you who have done this (or at least tried): do you have any experiences or tips to share? -- Regards, Bjørn-Helge Mevik

2 1

GPFS nvme limit storage per job
by Ratnasamy, Fritz 18 Aug '25

18 Aug '25

We have our new dedicated GPFS storage and were wondering how to accommodate all the users on the local nvme. Is there a way for slurm to enforce a per-job quota for every user so we make sure they don't take all the space? Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

1 0

Nodes Become Invalid Due to Less Total RAM Than Expected
by Xaver Stiensmeier 18 Aug '25

18 Aug '25

Dear slurm-user list, in the past we had a bigger buffer between RealMemory <https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory> and the instance memory. We then discovered that the right way is to activating the *memory option* (SelectTypeParameters=CR_Core_Memory) and setting MemSpecLimit <https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit> to secure RAM for system processes. However, now we run into the problem that due to *on demand scheduling*, we have to setup the slurm.conf in advance by using the RAM values from our flavors as reported by our cloud provider (OpenStack). These RAM values are higher than the RAM values the machines actually have later on: ram_in_mib by openstack total_ram_in_mib by top/slurm 2048 1968 16384 15991 32768 32093 65536 64297 122880 120749 245760 241608 491520 483528 Given that we have to define the slurm.conf in advance, we kinda have to predict how much total ram the instances have once created. Of course I used linear regression to approximate the total ram and then lowered it a bit to have some cushion, but this feels unsafe given that future flavors could differ from that. From the kernel documentation <https://www.kernel.org/doc/Documentation/filesystems/proc.txt> I know that MemTotal is MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code) but given that the concrete reserved bits are quite complex <https://witekio.com/blog/cat-proc-meminfo-memtotal/>, I am wondering whether I am doing something wrong as this issue doesn't feel niche enough to be that complicated. --- Anyway, setting the RAM value in the slurm.conf above total ram by predicting too much, leads to errors and nodes being marked as invalid: [2025-08-11T08:19:04.736] debug: Node NODE_NAME has low real_memory size (241607 / 245760) < 100.00% [2025-08-11T08:19:04.736] error: _slurm_rpc_node_registration node=NODE_NAME: Invalid argument or |[2025-07-03T12:57:18.486] error: Setting node NODE_NAME state to INVAL with reason:Low RealMemory (reported:64295 < 100.00% of configured:68719)| |Any hint on how to solve this is much appreciated! | Best regards, Xaver

3 3

Run a healthcheck job on all nodes
by John Hearns 18 Aug '25

18 Aug '25

I may have asked this already. I want to run a healtcheck job on all nodes. I can select the nodes in a partition by hand, the write a bash cript to get a list of nodes using nodeset -e Then submit to each node in the list using sbatch -w Is there a cleaner way of doing this? John Hearns

4 5

2026

2025

2024