- slurm-users - lists.schedmd.com

Slurm and Slinky events this fall, update events page
by Tim Wickberg 14 Oct '25

14 Oct '25

We'll have a bit more details as conference season quickly approaches this November, but SchedMD staff are presenting at KubeCon NA on Slinky [1]. We'll be manning the Slurm Booth at SC25 [2], as well as hosting the annual Slurm Community Birds-of-a-Feather session [3]. I'll also send a link out to the survey questions for the BoF to the slurm-users list ahead of the conference, and we'll be going into a bit more depth on the answers during the BoF this year. The events page on the SchedMD website has more detail on future events as well: https://www.schedmd.com/events/ A few folks had asked, and apparently we never mentioned this more publicly, but: SchedMD does not plan to hold an in-person SLUG in 2025 or 2026. We are working to bring some of the same content to our YouTube channel [4] as a way to more broadly disseminate some of that same content, starting with the Slurm 25.11 release overview in December. - Tim [1] https://kccncna2025.sched.com/event/27FW5/ [2] The Slurm Booth is #1641. [3] https://sc25.conference-program.com/presentation/?id=bof101&sess=sess471 [4] https://www.youtube.com/SchedMDSlurm -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

Slurm version 25.05.4 is now available
by Marshall Garey 14 Oct '25

14 Oct '25

We are pleased to announce the availability of Slurm version 25.05.4. This version increases the default number of maximum connections to slurmctld from 50 to 512, fixes a regression added in 25.05.2 that broke compatibility with PMIx v2.x through v3.1.0rc1, and fixes other minor to moderate bugs. The full list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Preemption Scheduler Tuning Knobs
by Reed Dier 13 Oct '25

13 Oct '25

Hoping someone may be able to help demystify some questions around the scheduler and preemption decisions. I'm trying to help better predict the scheduler behavior as it pertains to preemption to create more predictable scheduling. Version is 24.11.5, OS U22.04 I have a fairly simple lo/hi partition which the same set of nodes are assigned > PartitionName=partition-lo Nodes=foo[00..NN] Default=YES MaxTime=INFINITE OverSubscribe=FORCE:1 State=UP > PartitionName=partition-hi Nodes=foo[00..NN] Default=YES MaxTime=INFINITE OverSubscribe=NO State=UP PreemptMode=OFF And then I have 2 QOSs per partition > Name Priority GraceTime Preempt PreemptMode UsageFactor MaxJobsPU MaxTRESPA MaxJobsPA > ---------------- ---------- ---------- ---------------------- ----------- ----------- --------- ------------- --------- > qos-stateless-lo 1 00:00:00 requeue 1.000000 > qos-stateful-lo 1 00:00:00 suspend 1.000000 NN cpu=NNN NNN > qos-stateful-hi 5 00:00:00 qos-state[ful,less]-lo cluster 1.000000 cpu=NNN > qos-stateless-hi 5 00:00:00 qos-state[ful,less]-lo cluster 1.000000 And the general way that it works out is that stateful jobs will typically spawn stateless jobs. Stateful jobs will get suspended, while stateless jobs will get requeued. And then there are some general guard rails around stateful jobs clogging the queue and preventing stateless jobs from scheduling and creating a deadlock, but thats not the specific issue here. > SchedulerType=sched/backfill > SelectType=select/cons_tres > SelectTypeParameters=CR_CPU_Memory > SchedulerParameters=max_rpc_cnt=500,\ > sched_min_interval=50000,\ > sched_max_job_start=300,\ > batch_sched_delay=6 > PriorityType=priority/multifactor > PreemptType=preempt/qos > PreemptMode=SUSPEND,GANG Workload is rather "high throughput", so a few settings were influenced by that <https://slurm.schedmd.com/high_throughput.html> guide. What I end up seeing is that it can take sometimes, but not always, take a while for lo jobs to be preempted by hi jobs, an example below, but if the mailing list eats images, a link to it here: https://imgur.com/a/7xVFC8a This is a rather low resolution view, as it is just a scraper running on 5 minute increments, but the blue filled area are "hi" jobs running, where the yellow are "lo" jobs running. Second image (dashed lines) are pending jobs for the same partitions. Oddly, this specific instance did not show any preemption events in the slurmctld logs, but users/admins were a bit perplexed as to why this drug on for so long, and without preemption kicking in. I was considering looking deeper into to try to better understand and predict preemption decisions: default_queue_depth partition_job_depth sched_interval sched_min_interval defer Hopefully someone can point me to some nuggets of information around this? Appreciate any pointers, Reed

1 0

About tpology.conf
by ysaiki18＠gmail.com 12 Oct '25

12 Oct '25

Must topology.conf, like slurm.conf, be accessible to all nodes?

3 6

Prolog error causing node to drain
by Ratnasamy, Fritz 11 Oct '25

11 Oct '25

Hi, We have a running slurm cluster and users have been submitting jobs for the past 3 months without any issues. Recently, some nodes jobs are getting drained randomly due to the reason "prolog error" Our slurm.conf has these 2 lines regarding prolog: PrologFlags=Contain,Alloc,X11 Prolog=/slurm_stuff/bin/prolog.d/prolog* Inside the prolog.d folder, there are 2 scripts which run with no errors as far as I can see but is there a way to debug why the nodes are going in draining mode once in a while because of "prolog error"? That seems to happen at random times and on random nodes. From the log file, I can see only this: Oct 06 00:57:43 pgpu008.chicagobooth.edu slurmd[3709622]: slurmd: error: prolog failed: rc:230 output:Successfully started proces> Oct 06 00:57:43 pgpu008.chicagobooth.edu slurmd[3709622]: slurmd: error: [job 20398] prolog failed status=230:0 Oct 06 00:57:43 pgpu008 slurmd[3709622]: slurmd: Job 20398 already killed, do not launch batch job Oct 06 13:06:23 pgpu008 systemd[1]: Stopping Slurm node daemon... Oct 06 13:06:23 pgpu008 slurmd[3709622]: slurmd: Caught SIGTERM. Shutting down. Oct 06 13:06:23 pgpu008 slurmd[3709622]: slurmd: Slurmd shutdown completing Currently, now the job 20398 that is getting killed in the picture above is in the state "Launch failed requeue held" after I resume the node. *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Testing Slurm on FreeBSD – Offering Feedback and Collaboration
by Rikka Göring 07 Oct '25

07 Oct '25

Hello everyone, I’d like to briefly introduce myself and share some good news. I maintain the FreeBSD port of Slurm (sysutils/slurm-wlm) and have just completed the upgrade to the latest release. Alongside this, several FreeBSD-related fixes I had been maintaining locally have now been merged upstream by SchedMD engineers, which is an encouraging step for portability. With the updated port now running reliably on FreeBSD, I can offer: - Regular build and runtime testing of Slurm on FreeBSD, - Feedback on portability patches and non-Linux code paths, - Collaboration with anyone interested in running or experimenting with Slurm on BSD systems. My goal is to help ensure that Slurm remains portable and usable on FreeBSD, and to contribute back testing and fixes where possible. If you’re working on anything that may affect portability, feel free to reach out — I’d be glad to test and provide results from the FreeBSD side. Best regards, Rikka Göring

1 0

Unexpected missing socket error
by Ozeryan, Vladimir 07 Oct '25

07 Oct '25

Hello everyone, Not sure if you guys have heard this tune already but did anyone come across a solution for "Unexpected missing socket error". There is nothing useful in the logs but the message appears on compute nodes and slurm controller node. Thank you, Vlad Ozeryan AMDS - AB1 Linux-Support Vladimir.Ozeryan(a)jhuapl.edu Ext. 23966

3 2

Setting GrpTRES for specific Account only for specific Partition(s)
by Paul Raines 06 Oct '25

06 Oct '25

I am trying to figure out how one can limit resources like with GrpTRES for specific accounts only for specific partitions in the cluster Lets say I have a cluster with 3 partitions Part1, Part2, and Part3 and 3 accounts Acct1, Acct2, and Acct3 I want Acct1 to be limited to 10 cpus on Part1, Acct2 to be limited to 5 cpus on Part1, and Acct3 to be limited to 8 cpus on Part1. No limits for any account on the other two partitions at present. I have tried # sacctmgr modify account Acct1 where partition=Part1 set GrpTRES=cpu=10 Modified account associations... Nothing modified I have tried creating a QOS to assign # sacctmgr add qos cpus10 # sacctmgr modify qos cpus10 set GrpTRES=cpu=10 # sacctmgr modify account Acct1 where partition=Part1 set qos=cpus10 Modified account associations... Nothing modified I cannot set a Partition QOS since that would apply to all accounts on the partition giving them the same limit I cannot just set a general Account limit (or QOS) since that would apply to all Partitions. I really don't want to have to have different Accounts for each partition so I can set Account wide limits. That would make users have to remember which Account to use with which Partiton. It looks like the following works but requires having to repeat for all users in every Account sacctmgr -i add user User1 account=Acct1 partition=Part1 \ fairshare=parent qos=cpus10 # limits on the Acct1 partition sacctmgr -i add user User1 account=Acct1 \ fairshare=parent # no limits on any other partition But in essense is not much better than having different Accounts for each partition as I now have to maintain multiple user entries per Account: one for partitios with no-limits and and additional one for each partition that needs limits. I guess at least users don't have to remember what account to use when submitting to a partition. Its seems there should be some simpler way to do this. A way to say "this is the limit for this Account on this Partion for all current users and any future users added to the Account" Am I missing something? The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

cloud compute limits/budget
by Tom Sparks 03 Oct '25

03 Oct '25

I am setting up a single user slurm I have limited budget for cloud gpu compute ($40) I am wondering if slurm has ability to stop me from going over my budget? I have read the slurm's powersaving / configureless nodes guides Are they other guides I should look at?

2 1

SRUN and SBATCH network issues on configless login node.
by Bruno Bruzzo 01 Oct '25

01 Oct '25

Hi all, first of all, sorry for my English, it's not my native language. We are currently experiencing an issue with srun and salloc on our login nodes, while sbatch works properly. slurm version 23.11.4. slurmctld runs on management node mmgt01. srun and salloc fail intermittently on login node, that means we can successfully use srun on login node from time to time, but it stops working for a while without us changing any configuration. login nodes reports the following to the user: $ srun -N 1 -n 64 --partition=gpunode --pty /bin/bash srun: job 4872 queued and waiting for resources srun: error: Security violation, slurm message from uid 202 srun: error: Security violation, slurm message from uid 202 srun: error: Security violation, slurm message from uid 202 srun: error: Task launch for StepId=4872.0 failed on node cn065: Invalid job credential srun: error: Application launch failed: Invalid job credential srun: Job step aborted uid 202 is the slurm user. On the server side, slurmctld logs show: sched: _slurm_rpc_allocate_resources JobId=4872 NodeList=(null) usec=228 sched: Allocate JobId=4872 NodeList=cn065 #CPUs=64 Partition=gpunode error: slurm_receive_msgs: [[snmgt01]:38727] failed: Zero Bytes were transmitted or received Killing interactive JobId=4872: Communication connection failure _job_complete: JobId=4872 WEXITSTATUS 1 _job_complete: JobId=4872 done step_partial_comp: JobId=4872 StepID=0 invalid; this step may have already completed _slurm_rpc_complete_job_allocation: JobId=4872 error Job/step already completing or completed We suspect it is a network issue regarding the Zero Bytes were transmitted or received. The configless system is working properly. Slurmd on login node can read changes made at slurm.conf after a scontrol reconfig. srun runs successfully from management nodes and from compute nodes. The issue is from the login node. scontrol ping always shows DOWN from login node, even when we can successfully run srun or salloc. $ scontrol ping Slurmctld(primary) at mmgt01 is DOWN We checked as well for munge consistency. mmgt and login nodes have the hostnames of their respective others on /etc/hosts. They can communicate. We would really appreciate some tips on what we could be missing. Best regards, Bruno Bruzzo System Administrator - Clementina XXI

4 7

Re: How to make TLS and PMIx v4 work together?
by Grigory Shamov 30 Sep '25

30 Sep '25

Forgot to add: the s2n-tls comes from EPEL and is ver 1.5.10. On 2025-09-25, 11:56 AM, "Grigory Shamov via slurm-users" <slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com>> wrote: Caution! This message was sent from outside the University of Manitoba. Hi All, We have updated SLURM to the current 25.05.x and tried to enable TLS on it. The OS is Alma 8.10, cgroups v1, and PMIx v 4. We see that srun fails for MPI jobs across the nodes, with TLS related errors when using PMIx (the default) but passes with srun --mpi=pmi2 or with mpirun . TLSType = tls/s2n TLSParameters = ca_cert_file= (has all the certs here under /etc/slurm/certs) And the errors when using PMIx are 025-09-25T11:04:43.894] error: con_close_on_poll_error: [n388:6818(fd:15)] socket error encountered while polling: Connection reset by peer [2025-09-25T11:04:50.102] [6451416.0] error: _negotiate: s2n_negotiate() failed S2N_ERR_CERT_UNTRUSTED[335544366]: Certificate is untrusted -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/tls/s2n_x509_validator.c:494 (couple of these) [2025-09-25T11:05:57.878] [6451416.0] error: tls_p_recv: s2n_recv() failed S2N_ERR_CLOSED[134217728]: connection is closed -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:37 [2025-09-25T11:05:57.883] [6451416.0] error: tls_p_send: s2n_send() failed S2N_ERR_IO[67108864]: underlying I/O operation failed, check system errno -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:28 (couple of these) [2025-09-25T11:05:59.076] error: wrap_on_data: [unix:/var/spool/slurmd/slurmd.socket(fd:17)] on_data returned rc: Unable to proxy slurmstepd message [2025-09-25T11:05:59.076] [6451416.0] error: _stepd_send_recv_msg: slurmd was unable to proxy request message to its final destination [2025-09-25T11:05:59.878] error: _slurmd_send_recv_msg: Failed to send/recv slurmstepd message MESSAGE_TASK_EXIT using proxy_type PROXY_TO_NODE_SEND_RECV 2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: pmixp_p2p_send: n388 [0]: pmixp_utils.c:469: send failed, rc=1001, exceeded the retry limit [2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: _slurm_send: n388 [0]: pmixp_server.c:1586: Cannot send message to /var/spool/slurmd/stepd.slurm.pmix.6451416.0, size = 27679, hostlist: (null) (and couple more PMIx errors). Looks like PMIx cannot talk to their peers now ? There was no specific configuration for the certgen plugin, because SLURM documentation seems to say it is optional(?). I wonder what do we miss here to have SLURM 25.05 in with TLS enabled and PMIx working? Any advice appreciated! Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com>

3 3

FUTURE nodes do not return to idle on slurmctld restart
by Steve Kirk 26 Sep '25

26 Sep '25

Hello, We've got a few nodes defined in our slurm.conf in 'FUTURE' state as it's a new hardware type we're working on brining into service. The nodes are currently all allocated to a dedicated partition. The partition is configured as 'state=UP'. As we've built the new nodes and started slurmd+munge, they've appeared in an idle state in the new partition as expected. All good so far. However if the slurmctld is restarted the nodes go back to being in 'FUTURE' state, and do not transition to idle, accept jobs etc. The slurm daemon on the new nodes can clearly still talk to the slurmctld, s* commands on the new nodes work as expected but remain in FUTURE state - until slurmd on each node is restarted. I could have misunderstood something about the FUTURE state but I was expecting them to go back to idle; I understand that slurmctld doesn't communicate out to nodes in FUTURE state but I at least expected them to be picked up when they communicate _in_ to the slurmctld. Is this expected behaviour or perhaps a bug? The reason I've defined the new nodes this way so I don't have to update slurm.conf and restart slurmctld as each is built, but can do that as a single job once everything is finished, however it seems less useful if they can 'disappear' from the cluster as far as users are concerned. Cheers, Steve

2 2

How to make TLS and PMIx v4 work together?
by Grigory Shamov 25 Sep '25

25 Sep '25

Hi All, We have updated SLURM to the current 25.05.x and tried to enable TLS on it. The OS is Alma 8.10, cgroups v1, and PMIx v 4. We see that srun fails for MPI jobs across the nodes, with TLS related errors when using PMIx (the default) but passes with srun --mpi=pmi2 or with mpirun . TLSType = tls/s2n TLSParameters = ca_cert_file= (has all the certs here under /etc/slurm/certs) And the errors when using PMIx are 025-09-25T11:04:43.894] error: con_close_on_poll_error: [n388:6818(fd:15)] socket error encountered while polling: Connection reset by peer [2025-09-25T11:04:50.102] [6451416.0] error: _negotiate: s2n_negotiate() failed S2N_ERR_CERT_UNTRUSTED[335544366]: Certificate is untrusted -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/tls/s2n_x509_validator.c:494 (couple of these) [2025-09-25T11:05:57.878] [6451416.0] error: tls_p_recv: s2n_recv() failed S2N_ERR_CLOSED[134217728]: connection is closed -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:37 [2025-09-25T11:05:57.883] [6451416.0] error: tls_p_send: s2n_send() failed S2N_ERR_IO[67108864]: underlying I/O operation failed, check system errno -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:28 (couple of these) [2025-09-25T11:05:59.076] error: wrap_on_data: [unix:/var/spool/slurmd/slurmd.socket(fd:17)] on_data returned rc: Unable to proxy slurmstepd message [2025-09-25T11:05:59.076] [6451416.0] error: _stepd_send_recv_msg: slurmd was unable to proxy request message to its final destination [2025-09-25T11:05:59.878] error: _slurmd_send_recv_msg: Failed to send/recv slurmstepd message MESSAGE_TASK_EXIT using proxy_type PROXY_TO_NODE_SEND_RECV 2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: pmixp_p2p_send: n388 [0]: pmixp_utils.c:469: send failed, rc=1001, exceeded the retry limit [2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: _slurm_send: n388 [0]: pmixp_server.c:1586: Cannot send message to /var/spool/slurmd/stepd.slurm.pmix.6451416.0, size = 27679, hostlist: (null) (and couple more PMIx errors). Looks like PMIx cannot talk to their peers now ? There was no specific configuration for the certgen plugin, because SLURM documentation seems to say it is optional(?). I wonder what do we miss here to have SLURM 25.05 in with TLS enabled and PMIx working? Any advice appreciated! Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

1 0

Node switching randomly to down state
by Julien Tailleur 24 Sep '25

24 Sep '25

Dear all, I am maintaining a small computing cluster and I have a weird behavior that I fail at debugging. My cluster comprise one master node and 16 computing servers, organized in two queues, each queue having 8 servers. All servers run up-to-date Debian bullseye. All but 3 servers work flawlessly. From the master node, I can see that 3 servers on one of the queue appear down: jtailleu@kandinsky:~$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 down* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 These servers are reachable by SSH/ping jtailleu@kandinsky:~$ ping -c 1 FX12 PING FX12 (192.168.6.22) 56(84) bytes of data. 64 bytes from FX12 (192.168.6.22): icmp_seq=1 ttl=64 time=0.070 ms --- FX12 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.070/0.070/0.070/0.000 ms ##### I can also put these nodes back into idle mode: root@kandinsky:~# scontrol update nodename=FX[12-14] state=idle root@kandinsky:~# sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 idle* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 But then, they switch back into down mode few minutes later: root@kandinsky:~# sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 down* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 root@kandinsky:~# sinfo -R REASON USER TIMESTAMP NODELIST Not responding slurm 2025-09-08T15:04:39 FX[12-14] I do not understand where the "not responding" comes from, nor how I can investigate that. Any idea what could trigger this behavior? Best wishes, Julien

4 4

No output and can't job by id
by Dhumal, Dr. Nilesh 22 Sep '25

22 Sep '25

Hello, Recently, we updated Slurm. We are running slrumcld on head node. The slurmdbd is disabled on the head node. The user submits the job and gets the job ID. The squeue is empty, and difficult to track the job progress. The job is not writing any output to a file. How can I resolve this issue? Nilesh Dhumal Associate Professor of Chemistry, http://faculty.fgcu.edu/ndhumal/ Coordinator, FGCU Computational Facility, https://www.fgcu.edu/cas/facultyresources/computationalfacility/ SH-430; Department of Chemistry and Physics Florida Gulf Coast University 10501 FGCU Boulevard South Fort Myers, FL 33965-6565 Phone: (239) 745-4394 Email: ndhumal(a)fgcu.edu

2 2

Node in drain state
by Gestió Servidors 22 Sep '25

22 Sep '25

Hello, I have got a node in "drain" state after finishing a job that was running on it. Log in node reports this information: [...] [2025-09-07T11:09:26.980] task/affinity: task_p_slurmd_batch_request: task_p_slurmd_batch_request: 59238 [2025-09-07T11:09:26.980] task/affinity: batch_bind: job 59238 CPU input mask for node: 0xFFF [2025-09-07T11:09:26.980] task/affinity: batch_bind: job 59238 CPU final HW mask for node: 0xFFF [2025-09-07T11:09:26.980] Launching batch job 59238 for UID 21310 [2025-09-07T11:09:27.006] cred/munge: init: Munge credential signature plugin loaded [2025-09-07T11:09:27.007] [59238.batch] debug: auth/munge: init: loaded [2025-09-07T11:09:27.009] [59238.batch] debug: Reading cgroup.conf file /soft/slurm-23.11.0/etc/cgroup.conf [2025-09-07T11:09:27.025] [59238.batch] debug: cgroup/v1: init: Cgroup v1 plugin loaded [2025-09-07T11:09:27.025] [59238.batch] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: core enforcement enabled [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: device enforcement enabled [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2025-09-07T11:09:27.026] [59238.batch] task/affinity: init: task affinity plugin loaded with CPU mask 0xfff [2025-09-07T11:09:27.027] [59238.batch] debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded [2025-09-07T11:09:27.027] [59238.batch] topology/default: init: topology Default plugin loaded [2025-09-07T11:09:27.030] [59238.batch] debug: gpu/generic: init: init: GPU Generic plugin loaded [2025-09-07T11:09:27.031] [59238.batch] debug: laying out the 12 tasks on 1 hosts clus09 dist 2 [2025-09-07T11:09:27.031] [59238.batch] debug: close_slurmd_conn: sending 0: No error [2025-09-07T11:09:27.031] [59238.batch] debug: Message thread started pid = 910040 [2025-09-07T11:09:27.031] [59238.batch] debug: Setting slurmstepd(910040) oom_score_adj to -1000 [2025-09-07T11:09:27.031] [59238.batch] debug: spank: opening plugin stack /soft/slurm-23.11.0/etc/plugstack.conf [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: job abstract cores are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: step abstract cores are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: job physical CPUs are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: step physical CPUs are '0-11' [2025-09-07T11:09:27.090] [59238.batch] debug levels are stderr='error', logfile='debug', syslog='fatal' [2025-09-07T11:09:27.090] [59238.batch] starting 1 tasks [2025-09-07T11:09:27.090] [59238.batch] task 0 (910044) started 2025-09-07T11:09:27 [2025-09-07T11:09:27.098] [59238.batch] debug: task/affinity: task_p_pre_launch: affinity StepId=59238.batch, task:0 bind:mask_cpu [2025-09-07T11:09:27.098] [59238.batch] _set_limit: RLIMIT_NPROC : reducing req:255366 to max:159631 [2025-09-07T11:09:27.398] [59238.batch] task 0 (910044) exited with exit code 2. [2025-09-07T11:09:27.399] [59238.batch] debug: task/affinity: task_p_post_term: affinity StepId=59238.batch, task 0 [2025-09-07T11:09:27.399] [59238.batch] debug: signaling condition [2025-09-07T11:09:27.399] [59238.batch] debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded [2025-09-07T11:09:27.400] [59238.batch] debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded [2025-09-07T11:09:27.400] [59238.batch] job 59238 completed with slurm_rc = 0, job_rc = 512 [2025-09-07T11:09:27.410] [59238.batch] debug: Message thread exited [2025-09-07T11:09:27.410] [59238.batch] stepd_cleanup: done with step (rc[0x200]:Unknown error 512, cleanup_rc[0x0]:No error) [2025-09-07T11:09:27.411] debug: _rpc_terminate_job: uid = 1000 JobId=59238 [2025-09-07T11:09:27.411] debug: credential for job 59238 revoked [...] "sinfo" shows: [root@login-node ~]# sinfo PARTITION TIMELIMIT AVAIL STATE NODELIST CPU_LOAD NODES(A/I) NODES(A/I/O/T) CPUS CPUS(A/I/O/T) REASON node.q* 4:00:00 up drained clus09 0.00 0/0 0/0/1/1 12 0/0/12/12 Kill task faile node.q* 4:00:00 up allocated clus[10-11] 13.82-15.8 2/0 2/0/0/2 12 24/0/0/24 none node.q* 4:00:00 up idle clus[01-06,12] 0.00 0/7 0/7/0/7 12 0/84/0/84 none But it seems there is no error in node... Slurmctld.log in server seems correct, too. Is there any way to reset node to "state=idle" after errors in the same way? Thanks.

3 4

Compute node not responding
by Dhumal, Dr. Nilesh 21 Sep '25

21 Sep '25

Hello, Recently, we installed slum 25 on our cluster. We are not monitoring the user's account. We didn't configure the sql database on the head node. We are running slurmcld on head node and slumd on the compute node. We are getting the following error Head node: compute node not responding. Compute node: 2025-09-19T15:30:23.461] error: Unable to register: Unable to contact slurm controller (connect failure) Do we need to run slumdbd on the head node? I checked the network connection by pinging the compute node from the head node. Do you have any suggestions to resolve this issue? Thanks Nilesh Get Outlook for Android<https://aka.ms/AAb9ysg>

4 5

Re: Node in drain state
by Gestió Servidors 19 Sep '25

19 Sep '25

Hi, After reading answer from Ole Holm Nielsen, I have increased "MessageTimeout" to 20s (by default is 5s) and "UnkillableStepTimeout" to 150s (by default is 60s and, always 5 times larger than "MessageTimeout"). However, I have also read that UnkillableStepProgram indicates the program to use in that cases... but, by default there is no program assigned to that parameter (no program to run). So my question is if someone uses a customized "UnkillableStepProgram" and if he/she could explain that. Thanks a lot!

3 2

seff for GPU
by Josu Lazkano Lete 18 Sep '25

18 Sep '25

Hello, We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff? It will be great to know how much GPU resources the users request and compare with how much GPU resources they use. Kind regards. [image: Vicomtech] <https://www.vicomtech.org> Josu Lazkano Lete Systems Manager Infrastructures and General Services jlazkano(a)vicomtech.org +(34) 943 30 92 30 The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it. Avoid printing this message if it is not strictly necessary.

7 9

Scheduling issues with multiple different types of GPU in one partition
by Kevin M. Hildebrand 17 Sep '25

17 Sep '25

We have several different types of GPUs in the same 'gpu' partition. The problem we're having occurs when one of those types of GPUs is fully occupied and there are a bunch of queued jobs waiting for those GPUs. If someone requests idle GPUs of a different type, those jobs end up getting stalled, even though there are plenty of GPUs available. For example, say we have 10 A100 GPUs and 10 H100 GPUs. If there are 10 H100 GPU jobs running and more in queue waiting for them, subsequently submitted A100 jobs will sit in queue even if there are plenty of idle A100 GPUs. The only way we can get the A100 jobs to run is by manually bumping their priority higher than the pending H100 jobs. Has anyone else encountered this issue? The only way we can think of to potentially solve it is to have separate partitions for each GPU type, but that seems unwieldy. We are currently running Slurm 24.05.8. Thanks, Kevin -- Kevin Hildebrand Director of Research Technology and HPC Services Division of IT

7 7

New "NOT-state" selection of the sinfo command in Slurm 25.05
by Ole Holm Nielsen 10 Sep '25

10 Sep '25

We just upgraded Slurm to 25.05.3, and I would like to highlight a new functionality of the "sinfo -t <state>" command in 25.05: > The state can be prefixed with '~' which will invert the result of match. We find this "NOT-state" selection useful together with Slurm power saving [1] where any idle nodes may be powered off to save electrical power or cloud costs. For example, the ClusterShell [2] clush command can now skip connections to the powered down nodes like in this example: $ clush -bw@slurmstate:~POWERED_DOWN uname -r You can even configure ClusterShell so that "clush -a" includes all nodes which are *not* powered down. The details are described in the Wiki page [3]. Best regards, Ole [1] https://slurm.schedmd.com/power_save.html [2] https://clustershell.readthedocs.io/en/latest/intro.html [3] https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_operations/#skipping-powere… -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

1 0

Development RPMs for cgroups v2
by John Hearns 10 Sep '25

10 Sep '25

I am building a version of Slurm on RHEL 9.4 When I run an rpmbuild, he slurm rpm does not contain /usr/lib64/slurm/cgroup_v2.so I have tried to look in the build logs. I suspect I am lacking some development RPMs - but which ones? All hints gratefully received John H

4 5

NHC skips nodes in "mix-" state
by blines＠tgen.org 08 Sep '25

08 Sep '25

The node-mark-offline script skips nodes in the "mix-" state with the error: State "mix-" not yet handled; ignoring. "mix-" seems to be a valid state indicating that a node is mixed and is "planned by the backfill scheduler for a higher priority job". Anyone know the reason that nodes in this state are being skipped? The responsible code appears to be: case "$STATUS" in *'@'*|*'#'*|boot*|*-*|plnd*) # These states aren't handled yet. echo "$0: State \"$STATUS\" not yet handled; ignoring." exit 0 As far as I can tell all the states being skipped are related to nodes being configured or rebooted or powering on/off except *-*. There are other states related to node power that include underscore in the state name; power_down, power_up, pow_dn, pow_up. Should the case statement be *_* instead of *-*?

1 0

Creating /run/user/$UID - for Podman runtime
by John Snowdon 08 Sep '25

08 Sep '25

We are in the middle of implementing an extensive range of container support on our new HPC platform and have decided to offer our users a wide suite of technologies to better support their workloads: * Apptainer * Podman (rootless) * Docker (rootless) We've already got a solution for automated entries in /etc/subuid and /etc/subgid on the head nodes (available here under GPL: https://github.com/megatron-uk/pam_subid) which is where we intend users to build their container images, and building and running containers using Apptainer and Podman in those environments works really well - we're happy that it should take care of 95% of our users needs (Docker is the last few percent....) and not involve giving them any special permissions. If I ssh directly to a compute node, then Podman also works there to run an existing image (podman container run ...). What I'm struggling with now is running Podman under Slurm itself on our compute nodes. It appears as though Podman (in rootless mode) wants to put the majority of its run time / state information under /run/user/$UID ... this is fine on the head nodes which have interactive logins hitting PAM modules which instantiate the /run/user/$UID directories, but not under sbatch/srun which doesn't create them by default. I've not been able to find a single, magical setting which will move all of the Podman state information out from /run/user to another location - there are 3 or 4 settings involved, and even then I still find various bits of Podman want to create stuff under there. Rather than hacking away at getting Podman changed to move all settings and state information elsewhere, it seems like the cleanest solution would just be to put the regular /run/user/$UID directory in place at the point Slurm starts the job instead. What's the best way to get Slurm to create this and clean-up afterwards? Should this be in a prolog/epilog wrapper (e.g. directly calling loginctl) or is it cleaner to get Slurm to trigger the usual PAM session machinery in some manner? John Snowdon Senior Research Infrastructure Engineer (HPC) Research Software Engineering Catalyst Building, Room 2.01 Newcastle University 3 Science Square Newcastle Helix Newcastle upon Tyne NE4 5TG https://hpc.researchcomputing.ncl.ac.uk

5 7

Slurm version 25.05.3 is now available
by Marshall Garey 04 Sep '25

04 Sep '25

We are pleased to announce the availability of Slurm version 25.05.3. This version fixes an issue that prevented deleting a QOS when running with MySQL servers (MariaDB is was unaffected). Please note that the slurmdbd will require MySQL 8.0.4+ or MariaDB 10.0.5+ to function correctly. This version also fixes heterogeneous jobs when TLS is enabled, a logging issue with syslog, and various mild to moderate stability fixes. The full list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Discussion about the --cpus-per-task flag
by Michele Esposito Marzino 04 Sep '25

04 Sep '25

Hi everyone, I would like to ask about the rationale behind the --cpus-per-task flag. While I do understand the meaning and its intended goal, I was wondering how it is managed in practice. I tried inferring this from the source code, but it is quite intricate, and not really straightforward by simply searching for the string match. Does it affect the CPUSET of each task? Who is responsible for setting this? The slurmcltd or slurmd? And how? Or at least, point me where in the code it is handled, so that I can reconstruct the whole process myself. Thanks

1 0

sacctmgr output with and without associations
by Rémy Dernat 04 Sep '25

04 Sep '25

Hi, I have a stupid issue about sacctmgr output and associations. I would like to retrieve my accounts with all the following data : "Account","Clusters","DefaultQOS","Description","Fairshare","Flags","GrpTRESMins","GrpTRES","GrpJobs","GrpSubmitJob",GrpWall", "MaxTRESMins","MaxTRES","MaxJobs","MaxNodes","MaxSubmitJobs","MaxWall","Organization","ParentName","Priority","QosLevel" So my command is the following : sacctmgr list accounts --assoc format="Account","Clusters","DefaultQOS","Description","Fairshare","Flags","GrpTRESMins","GrpTRES","GrpJobs","GrpSubmitJob","GrpWall","MaxTRESMins","MaxTRES","MaxJobs","MaxNodes","MaxSubmitJobs","MaxWall","Organization","ParentName","Priority","QosLevel" -n -P However, I have a lot of outputs corresponding to each User association entry. That would be fine if a basic `|uniq` would remove everything I don't need, but I am stuck with grptres, which can be seen in the main account, but not in the children associations with the Users. On the opposite, if I remove `--assoc`, I don't have anymore grptres, ParentName, or related Qos values. I had to add `where User=""` to my first command to just retrieve my main accounts, without User associations. I dont' know if that is clear enough; is there something I missed with the sacctmgr output ? Thanks, Best regards, Rémy

2 1

Can't remove account
by Matthias Leopold 01 Sep '25

01 Sep '25

Hi, I have a cluster with account foo. I want to delete this account. For every user that had associations in this account I created new associations in a different account and changed the DefaultAccount of the user. Then I deleted all associations in account foo. Now I want to delete account foo: sacctmgr delete account name=foo This gives: Error with request: You can not remove the default account of a user ... Please either remove the accounts listed above from list and resubmit, or change these users' default accounts to remove the account(s). Changes Discarded It names all associations for users that FORMERLY had DefaultAccount foo?? NO user has DefaultAccount foo anymore... I have to mention that I also have partition foo and affected users have an association in their new account with partition foo. But that shouldn't matter IMHO Has anybody seen this? What can I do? Am I getting something wrong? Slurm version is 24.05.7 thanks Matthias

1 0

slurmrestd unable to find plugin rest_auth/jwt
by Jon Marshall 28 Aug '25

28 Aug '25

Hi, I feel like I must have missed something obvious here, so hopefully someone can spot it! We recently updated to Slurm 24.11.6 on our cluster and I made sure to compile --with slurmrestd and --with jwt, as there are a few things I've been meaning to try out with the REST API. From the rpmbuild config.log I get: $ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc/slurm --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-systemdsystemunitdir=/usr/lib/systemd/system --enable-pkgconfig --with-jwt -- configure:21020: checking for jwt.h configure:21020: gcc -c -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer conftest.c >&5 configure:21020: $? = 0 configure:21020: result: yes configure:21028: checking for jwt_add_header in -ljwt configure:21051: gcc -o conftest -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -Wl,-z,relro -Wl,-z,lazy conftest.c -ljwt -ldl -lrt -lm -lresolv >&5 configure:21051: $? = 0 configure:21061: result: yes -- config.status:1746: creating src/plugins/auth/jwt/Makefile -- config.status:1746: creating src/slurmrestd/plugins/auth/jwt/Makefile -- ac_cv_header_jwt_h=yes -- ac_cv_lib_jwt_jwt_add_header=yes -- JWT_LDFLAGS='-ljwt' And: configure:26873: checking whether to compile slurmrestd configure:26904: result: yes -- configure:26915: checking for slurmrestd default port configure:26933: result: 6820 And indeed I get a slurmrestd package out of it, which I've successfully installed with no errors. I've created a JWT key and rolled it out to the controller node, but when I go to check auth types I get: slurmrestd -a rest_auth/jwt slurmrestd: fatal: Unable to find plugin: rest_auth/jwt Running an strace on this process shows that it does in fact stat the .so that does in fact exist: stat("/usr/lib64/slurm/auth_jwt.so", {st_mode=S_IFREG|0755, st_size=283728, ...}) = 0 And I'm kind of stumped now. I've checked through all the resolved bug tickets to see if I can spot what is wrong but I'm getting nothing, and I can't find anything in the mailing lists either, other than things I've checked. Has anyone else had similar issues, or can someone see the blindingly obvious thing I'm missing? Cheers Jon Jon Marshall High Performance Computing Specialist IT and Scientific Computing Team Cancer Research UK Cambridge Institute Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE Web<http://www.cruk.cam.ac.uk/> | Facebook<http://www.facebook.com/cancerresearchuk> | Twitter<http://twitter.com/CR_UK> [Description: CRI Logo]<http://www.cruk.cam.ac.uk/>

2 4

Assistance with Node Restrictions and Priority for Users in Floating Partition
by Manisha Yadav 28 Aug '25

28 Aug '25

Dear Team, I have a scenario where I need to provide priority access to multiple users from different projects for only 3 nodes. This means that, at any given time, only 3 nodes can be used in that partition, and if one user is utilizing all 3 nodes, no other user should be able to submit jobs to that partition, or their jobs should remain in the queue. To achieve this, I attempted to use QoS by creating a floating partition with some of the nodes and configuring a QoS with priority. I also set a limit with GrpTRES=gres/gpu=24, given that each node has 8 GPUs, and there are 3 nodes in total. I then attached the QoS to the partition and assigned it to the users who need access. I Also tried MaxTRES=gres/gpu=24 While this setup works as expected in the testing environment for CPUs, it is not functioning as intended in production, and it is not effectively restricting node usage in the partition. Could anyone provide suggestions or guidance on how to properly implement node restrictions along with priority? Thank you for your assistance. Best regards, Manisha Yadav ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

4 8

Node Health Check Program
by Paul Edmon 27 Aug '25

27 Aug '25

We've been using NHC (https://github.com/mej/nhc) for years with much success. However that project hasn't had a release in 2 years and the various Issues filed indicate that there might be problems with Rocky 9 (which we are looking to upgrade to). Do people that are at EL9 use NHC? Is there a fork? Is there a different code that people use for doing node health checks? -Paul Edmon-

8 16

Slurm 25.05: Retrieving jobs GPU Indices on Heterogeneous Cluster
by David Gauchard 27 Aug '25

27 Aug '25

Hello, I'm running Slurm 25.05 on a heterogeneous cluster (several kind of GPU in the same node) with AutoDetect=nvml and shared mode. When submitting a job with `#SBATCH --gres=gpu:1`, CUDA_VISIBLE_DEVICES is correctly set to a single and valid free GPU index but `scontrol show jobid` does not report any detail about GPU allocation. How can I retrieve the GPU indices assigned to running jobs (reflecting CUDA_VISIBLE_DEVICES) in shared mode? Is there a Slurm command or configuration to enable tracking of these indices on a heterogeneous cluster? The goal is to help automatic choice of the free and appropriate gpu according to the job's needs in order to save bigger gpus for bigger jobs at the time of submitting. Thanks

2 2

Issues with dynamic configless nodes and pam_slurm_adopt
by Leonardo Sala 26 Aug '25

26 Aug '25

Hallo everyone we have recently noticed that when running nodes in configless and dynamic mode, pam_slurm_adopt does not work properly, denying ssh access when the user has a running job. We traced down the issue to the fact that the dynamic nodes are registered with FQDN (e.g. c01.psi.ch), while pam_slurm_adopt uses this routine (https://github.com/SchedMD/slurm/blob/master/src/common/stepd_api.c#L181) to guess the stepd hostname, using the host short name (so, c01). If we use gethostname instead of gethostname_short then it works again. Has anybody experience with this, and is there a way to have dynamic configless nodes registering with the short hostname? Thanks! cheers leo -- Paul Scherrer Institut Dr. Leonardo Sala Group Leader Data Analysis and Research Infrastructure Group Leader Data Curation a.i. Deputy Department Head Science IT Infrastructure and Services department Science IT Infrastructure and Services department (AWI) OBBA/230 Forschungstrasse 111 5232 Villigen PSI Switzerland Phone: +41 56 310 3369 leonardo.sala(a)psi.ch www.psi.ch

1 0

Tips or experiences with Burst Buffers?
by Bjørn-Helge Mevik 21 Aug '25

21 Aug '25

Dear all! We are currently considering setting up the (Lua) Burst Buffer plugin to stage in and out files. Those of you who have done this (or at least tried): do you have any experiences or tips to share? -- Regards, Bjørn-Helge Mevik

2 1

GPFS nvme limit storage per job
by Ratnasamy, Fritz 18 Aug '25

18 Aug '25

We have our new dedicated GPFS storage and were wondering how to accommodate all the users on the local nvme. Is there a way for slurm to enforce a per-job quota for every user so we make sure they don't take all the space? Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

1 0

Nodes Become Invalid Due to Less Total RAM Than Expected
by Xaver Stiensmeier 18 Aug '25

18 Aug '25

Dear slurm-user list, in the past we had a bigger buffer between RealMemory <https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory> and the instance memory. We then discovered that the right way is to activating the *memory option* (SelectTypeParameters=CR_Core_Memory) and setting MemSpecLimit <https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit> to secure RAM for system processes. However, now we run into the problem that due to *on demand scheduling*, we have to setup the slurm.conf in advance by using the RAM values from our flavors as reported by our cloud provider (OpenStack). These RAM values are higher than the RAM values the machines actually have later on: ram_in_mib by openstack total_ram_in_mib by top/slurm 2048 1968 16384 15991 32768 32093 65536 64297 122880 120749 245760 241608 491520 483528 Given that we have to define the slurm.conf in advance, we kinda have to predict how much total ram the instances have once created. Of course I used linear regression to approximate the total ram and then lowered it a bit to have some cushion, but this feels unsafe given that future flavors could differ from that. From the kernel documentation <https://www.kernel.org/doc/Documentation/filesystems/proc.txt> I know that MemTotal is MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code) but given that the concrete reserved bits are quite complex <https://witekio.com/blog/cat-proc-meminfo-memtotal/>, I am wondering whether I am doing something wrong as this issue doesn't feel niche enough to be that complicated. --- Anyway, setting the RAM value in the slurm.conf above total ram by predicting too much, leads to errors and nodes being marked as invalid: [2025-08-11T08:19:04.736] debug: Node NODE_NAME has low real_memory size (241607 / 245760) < 100.00% [2025-08-11T08:19:04.736] error: _slurm_rpc_node_registration node=NODE_NAME: Invalid argument or |[2025-07-03T12:57:18.486] error: Setting node NODE_NAME state to INVAL with reason:Low RealMemory (reported:64295 < 100.00% of configured:68719)| |Any hint on how to solve this is much appreciated! | Best regards, Xaver

3 3

Run a healthcheck job on all nodes
by John Hearns 18 Aug '25

18 Aug '25

I may have asked this already. I want to run a healtcheck job on all nodes. I can select the nodes in a partition by hand, the write a bash cript to get a list of nodes using nodeset -e Then submit to each node in the list using sbatch -w Is there a cleaner way of doing this? John Hearns

4 5

SLUG 25?
by Dave Lloyd 11 Aug '25

11 Aug '25

Are there plans for a SLUG this year? I didn’t see any announcements on SchedMD’s website nor anything in the list. Thanks, -- Dave Lloyd This e-mail and any attachments may contain information that is confidential and proprietary and otherwise protected from disclosure. If you are not the intended recipient of this e-mail, do not read, duplicate or redistribute it by any means. Please immediately delete it and any attachments and notify the sender that you have received it by mistake. Unintended recipients are prohibited from taking action on the basis of information in this e-mail or any attachments. The DRW Companies make no representations that this e-mail or any attachments are free of computer viruses or other defects.

1 0

sinfo history
by Steve Kirk 11 Aug '25

11 Aug '25

Hi, Am I correct in thinking that the history of a *node* as shown by sinfo isn't stored anywhere by Slurm? Interested to know if slurm can tell me historically when a node was draining,drained etc. Regards, Steve

5 6

Slurm version 25.05.2 is now available
by Marshall Garey 07 Aug '25

07 Aug '25

We are pleased to announce the availability of Slurm version 25.05.2. This version fixes a few regressions with x11 forwarding in 25.05 that may prevent applications from launching, adds support for PMIx v6.x, fixes a variety of stability issues, fixes a regression where --tres-per-task was ignored, and fixes additional minor to moderate severity issues. The full list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Building Debian packages - problem
by John Hearns 05 Aug '25

05 Aug '25

I am trying to build Slurm version 24.11.6 on an Ubuntu 22.04 system. I download and unpack the source mk-build-deps -i debian/control (not run as root) debuild -b -uc -us I get this error repeated many times: dpkg-shlibdeps: warning: can't extract name and version from library name 'libslurmfull.so' Has anyone seen similar issues? For information, there is an existing SLurm installation on this syste, with version 21 dpkgs from the Ubuntu repository.

4 4

slurmctld failed to start
by Dhumal, Dr. Nilesh 01 Aug '25

01 Aug '25

Hello, We recently installed slurm-25 on Redhat linux. We failed to start the slurmctld service. sudo systemctl start slurmctld Job for slurmctld.service failed because the control process exited with error code. See "systemctl status slurmctld.service" and "journalctl -xeu slurmctld.service" for details. sudo systemctl status slurmctld × slurmctld.service - Slurm controller daemon Loaded: loaded (/usr/local/lib/systemd/system/slurmctld.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Thu 2025-07-31 22:23:18 EDT; 1min 3s ago Process: 44317 ExecStart=/usr/local/sbin/slurmctld --systemd $SLURMCTLD_OPTIONS (code=exited, status=1/FAILURE) Main PID: 44317 (code=exited, status=1/FAILURE) CPU: 35ms Jul 31 22:22:29 fgcu-compute01 systemd[1]: Starting Slurm controller daemon... Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: If munged is up, restart with --num-threads=10 Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: Munge encode failed: Failed to connect to "/run/munge/mung> Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: Failed to create MUNGE Credential Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: Couldn't load specified plugin name for auth/munge: Plugin> Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] error: cannot create auth context for auth/munge Jul 31 22:23:18 fgcu-compute01 slurmctld[44317]: [2025-07-31T22:23:18.679] fatal: failed to initialize auth plugin Jul 31 22:23:18 fgcu-compute01 systemd[1]: slurmctld.service: Main process exited, code=exited, status=1/FAILURE Jul 31 22:23:18 fgcu-compute01 systemd[1]: slurmctld.service: Failed with result 'exit-code'. Jul 31 22:23:18 fgcu-compute01 systemd[1]: Failed to start Slurm controller daemon. Here is munge service status. munge.service - MUNGE authentication service Loaded: loaded (/usr/local/lib/systemd/system/munge.service; enabled; preset: disabled) Active: active (running) since Thu 2025-07-31 22:06:14 EDT; 19min ago Docs: man:munged(8) Main PID: 44039 (munged) Tasks: 4 (limit: 606218) Memory: 1.4M CPU: 18ms CGroup: /system.slice/munge.service └─44039 /usr/local/sbin/munged Jul 31 22:06:14 fgcu-compute01 systemd[1]: Starting MUNGE authentication service... Jul 31 22:06:14 fgcu-compute01 systemd[1]: Started MUNGE authentication service. Any suggestion is apprecieted to resolve this issue. Thanks, Nilesh Dhumal Associate Professor of Chemistry, http://faculty.fgcu.edu/ndhumal/ Coordinator, FGCU Computational Facility, https://www.fgcu.edu/cas/facultyresources/computationalfacility/ SH-430; Department of Chemistry and Physics Florida Gulf Coast University 10501 FGCU Boulevard South Fort Myers, FL 33965-6565 Phone: (239) 745-4394 Email: ndhumal(a)fgcu.edu

2 3

X11 forwarding broken in 25.05.1
by Patryk Bełzak 24 Jul '25

24 Jul '25

Hi, we've recentry upgraded our slurm from 24.11.3 to 25.05.1 and it seems that since the upgrade the ssh X11 forwaring is broken. Quick recap - * on Monday 14'th I performed slurdbd and slurmctld upgrades - X forwarding was still working * on Tuesday 15'th I performed slurmd upgrades - X forwarding stopped working The issue is very hard to determine and it looks like it sits somhere in slurm code. You can submit a job with --x11 and it starts corretly. Xauthority is created, you have all the magic cookies needed, but when you try to start any application, you get error related to permissions I guess, see for yourself: ``` me@sand ~ ssh -X -Y ui [wcss] me@ui.wcss.pl:~ > srun -p lem-cpu-short -A kdm-staff --gres=storage:local:50G -c 12 --mem 12G -t 1:0:0 --x11 --pty /bin/bash [wcss] me@r17ch05b01 ~ > xauth list r17ch05b01.lem.kdm.wcss.pl/unix:91 MIT-MAGIC-COOKIE-1 d82a2efd [wcss] me@r17ch05b01 ~ > xterm xterm: Xt error: Can't open display: localhost:91.0 [wcss] me@r17ch05b01 ~ > date && telnet -4 localhost 6091 || date Wed Jul 23 12:02:39 CEST 2025 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Connection closed by foreign host. Wed Jul 23 12:02:41 CEST 2025 ``` As you can see the connection to port is being dropped/killed after a second or two. Now, it doesn't really matter which flags for ssh you pick (-X or -Y or both). X forwarding is working when you log in as a regular user outside of slurm job. Also if I do ssh localhost inside a job, then I can perform connection to port assigned to $DISPLAY and it isn't dropped - but it doesn't work since $DISPLAY and cookies are being messed up when you perform triple jump and one within same host. Our worker nodes are mostly on el9.5 AlmaLinux. Some are on el8.10 - and there acutally you can do some X forwarding but you must use both -X and -Y (which wasn't the case before slurm upgrade). TLS is disabled in slurm.conf. I am 100% sure that both SSHD and Xorg are properly configured. Has anyone encountered similiar issue? Or any comment from slurm dev team? Best regards Patryk -- Wroclaw Centre for Networking and Supercomputing

3 2

New SchedMD Bike Kits Are Here - Preorder Now Open!
by Victoria Hobson 23 Jul '25

23 Jul '25

We’re excited to announce that SchedMD-branded bike kits are officially available for preorder! Each kit features both the SchedMD and Slurm logos—designed for comfort, performance, and pride.Pick between thoughtfully styled designs for mountain and road biking in each of the following categories: • Jerseys (zip-up or v-neck short sleeve) • Bibs (gravel bib with side and back pockets or switch bib) • Gloves (tech or summer) • Cycling socks with the SchedMD or Slurm logo (the right answer here is to just order both!) Preorders are open now through July 30, 2025. After the store closes, kits will be handcrafted and shipped directly to your door. Please note: delivery will take 8–10 weeks after preorder ends. Want 20% off your order? Reach out to Jess directly (jess(a)schedmd.com) to get an exclusive promo code! Ready to ride in style? Check out the SchedMD bike kits here: https://store.dnacycling.com/schedmd — Thanks for being part of the SchedMD community!

1 0

strange GPU "no resources selected" error
by Paul Raines 23 Jul '25

23 Jul '25

Recently (past 3 days) on our NVIDIA DGX A100 systems running Ubuntu 22.04.5 and slurm 24.11.5 we have had jobs that ask for a gpu, get started by Slurm, but fail to be given a GPU and then fail. In the slurmctld log we see a line like: [2025-07-22T02:46:29.697] error: gres/gpu: job 6919154 node A100-04 no resources selected on the slurmd log I see no errors for the job but there is a line like [2025-07-22T02:46:29.757] [6919154.extern] task/cgroup: _handle_device_access: GRES: job devices.deny: adding c 195:0 rwm(/dev/nvidia0) for all 8 of the GPUs on the node. Other jobs still seem to start up and get a GPU fine. If you look at the job stats one sees: ReqTRES : billing=7,cpu=1,gres/gpu=1,mem=96G,node=1 AllocTRES : billing=3,cpu=1,mem=96G,node=1 showing that even though the gpu was requested, it was not allocated. Occasionly on these boxes (and only these -- my Dell Rocky 8 boxes with GPUS have no problem) we see the nodes go into drain mode with the "res/gpu GRES core specification ... doesn't match socket boundaries." message as per https://support.schedmd.com/show_bug.cgi?id=22498 It does seem to happen after slurmctld restart. I then restart slurmd on the nodes and can resume SLURM on the nodes whenever that happens. Otherwise nothing has changed on these nodes with SLURM config or the OS in over a month. Definition of the nodes are NodeName=A100-[01-04] \ CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 \ ThreadsPerCore=1 RealMemory=1031000 MemSpecLimit=2048 \ TmpDisk=1400000 Feature=amd,epyc,a100 \ Gres=gpu:a100-sxm4-40gb:8 and gres.conf on the nodes is simply AutoDetect=nvml --------------------------------------------------------------- Paul Raines http://help.nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

slurm 25.05.0 : allocates less GPUs than requested by --gres=gpu:count
by Jan Gmys 23 Jul '25

23 Jul '25

Since upgrading slurm to 25.05.0 (22.05.9 -> 23.11.11 -> 25.05.0) some jobs requesting --gres=gpu:reqcount GPUs are allocated less than reqcount GPUs if some of the node's GPUs are already in use by other jobs. We have a node - let's call it ares-c02-06 - with 2 GPUs. Consider the following test script: > #!/bin/bash > #SBATCH --nodes=1 > #SBATCH --ntasks-per-node=1 > #SBATCH --time=1-00:00:00 > > echo "CUDA_VISIBLE_DEVICES: " $CUDA_VISIBLE_DEVICES > echo "SLURM_JOB_GPUS: " $SLURM_JOB_GPUS > echo "SLURM_GPUS_ON_NODE: " $SLURM_GPUS_ON_NODE > sleep 10d Submit a job to the node: > > sbatch *--gres=gpu:1* --nodelist=ares-c02-06 job.sh > > Submitted batch job 1950559 > The job starts. Now submit the script again, asking for 2 GPUs > sbatch*--gres=gpu:2* --nodelist=ares-c02-06 job.sh > > Submitted batch job 1950567 This second job should not start as the ressources are not available. Surprisingly, _both jobs are running_ > $ squeue -w ares-c02-06 > JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) > 1950567 gpuai job.sh jan.gmys R 1:55 1 ares-c02-06 > 1950559 gpuai job.sh jan.gmys R 2:58 1 ares-c02-06 In the second job - 1950567 - AllocTRES shows gres/gpu=1, instead of the requested gres/gpu=2. > # sacct -j 1950567,1950559 -X -o jobid%10,reqtres%45,alloctres%60 > JobID ReqTRES AllocTRES > ---------- --------------------------------------------- ------------------------------------------------------------ > 1950559 billing=1,cpu=1,gres/gpu=1,mem=4G,node=1 billing=1,cpu=1,gres/gpu:l40s=1,gres/gpu=1,mem=4G,node=1 > 1950567 billing=1,cpu=1,*gres/gpu=2*,mem=4G,node=1 billing=1,cpu=1,gres/gpu:l40s=1,*gres/gpu=1*,mem=4G,node=1 The output of both jobs > $ cat slurm-1950559.out > CUDA_VISIBLE_DEVICES: 0 > SLURM_JOB_GPUS: 0 > SLURM_GPUS_ON_NODE: 1 > > $ cat slurm-1950567.out > CUDA_VISIBLE_DEVICES: 0 > SLURM_JOB_GPUS: 1 > SLURM_GPUS_ON_NODE: 1 CUDA_VISIBLE_DEVICES is set to 0 for both jobs. SLURM_JOB_GPUS is 0 resp. 1. *Environment: * - RHEL 9.1 - slurm 25.05.0 - The GRES configuration seems fine, AutoDetect is off : > # /usr/sbin/slurmd -G --conf-server hpc-slurm.cluster.hpc -v > [2025-07-22T16:44:05.548] GRES: Global*AutoDetect=off*(4) > [2025-07-22T16:44:05.548] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, > minor 0 > [2025-07-22T16:44:05.548] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, > minor 1 > [2025-07-22T16:44:05.548] GRES: gpu device number 0(/dev/nvidia0):c 195:0 rwm > [2025-07-22T16:44:05.548] GRES: gpu device number 1(/dev/nvidia1):c 195:1 rwm > [2025-07-22T16:44:05.548] Gres Name=gpu Type=L40S Count=2 Index=0 ID=7696487 > File=/dev/nvidia[0-1] Links=(null) > Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT and 'scontrol show node' > NodeName=ares-c02-06 Arch=x86_64 CoresPerSocket=24 > CPUAlloc=0 CPUEfctv=48 CPUTot=48 CPULoad=0.00 > AvailableFeatures=(null) > ActiveFeatures=(null) > Gres=gpu:L40S:2 > NodeAddr=ares-c02-06 NodeHostName=ares-c02-06 Version=25.05.0 > OS=Linux 5.14.0-162.6.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 30 > 07:36:03 EDT 2022 > RealMemory=386000 AllocMem=0 FreeMem=363069 Sockets=2 Boards=1 > State=IDLE+RESERVED+PLANNED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A > MCS_label=N/A > Partitions=gpuai > BootTime=2025-07-04T10:26:27 SlurmdStartTime=2025-07-22T14:48:54 > LastBusyTime=2025-07-22T16:26:36 ResumeAfterTime=None > CfgTRES=cpu=48,mem=386000M,billing=48,gres/gpu=2 > AllocTRES= *Debug notes :* - When using the --gpus option instead of --gres everything works as expected : the second job is PENDING (Ressources) - Tried both, ConstrainDevices=on/off in cgroup.conf, same result - The same is happening on other multi-GPU nodes of the cluster - When the --gres=gpu:2 job is submitted first, i.e. when all GPUs are taken, the second (--gres=gpu:1) job waits correctly. - When both GPUs are free, the --gres=gpu:2 job, correctly gets both GPUs : CUDA_VISIBLE_DEVICES: 0,1 - It worked in slurm 22.05.9 (we recently upgraded in two steps -> 23.11.11 -> 25.05.0) - The only viable workaround I see for the moment is to intercept --gres and --gpus-per-node (don't even have it in job_desc I think!! :-/) options in job_submit.lua and force users to use the --gpus option, which seems to works fine. Anyone experienced similar issues? Any idea how to solve this would be highly appreciated. Jan == -- Jan Gmys Ingénieur de recherche Support HPC/IA pour la plateforme MesoNET Mésocentre de Calcul Scientifique Intensif de l'Université de Lille

1 0

Reserve memory for OS
by John Hearns 17 Jul '25

17 Jul '25

Someone remind me please what is the SLurm node parameter to reserve an amount of memory for the OS? Thanks

3 2

Integrating Fabric Modules with SLURM – Best Practices for HPC Networking?
by xoyeyor178＠coderdir.com 15 Jul '25

15 Jul '25

Our team is exploring ways to optimize our HPC cluster’s network performance, particularly for multi-node SLURM workloads. We’re considering Network Devices Expansion Modules Fabric Modules https://serverorbit.com/network-devices/network-expansion-modules/fabric-mo… to enhance scalability and reduce latency between compute nodes. Has anyone successfully deployed Fabric Modules (e.g., Cisco Nexus, Arista, or Mellanox solutions) in a SLURM environment? Specifically: Interconnect Strategies – Any tips for configuring Fabric Modules to handle SLURM’s bursty traffic patterns? Performance Gains – Measurable improvements in job throughput or MPI communication? Troubleshooting – Known conflicts with SLURM’s network topology detection?

1 0

Slurm-wlm depends on X11?
by Jesse Hayward 10 Jul '25

10 Jul '25

Hello group, hopefully a quick one: I am trying to install the suite of standard slurm packages (slurm, munge, slurmdbd, slurm-wlm-basic-plugins) on a compute node. Ubuntu 22.04.05 LTS just upgraded. *However, these packages now bring down x11 (libx11-6 and all of that other garbage in Ubuntu land). * Does slurm actually depend on x11 now? Or do I need to take a look at my apt config and see what's going on over there, instead. Thanks! Jess -- Jesse Hayward Systems Administrator for High Performance Computing Vassar College 845.437.7521 CIS 207

3 6

Is DefCpuPerGPU affected by --ntasks-per-gpu?
by Milad Alizadeh 10 Jul '25

10 Jul '25

I have defined DefCpuPerGPU for a partition but when a job specifies --ntasks-per-gpu then DefCpuPerGPU is ignored and I only get 1 cpu: ``` #SBATCH --gpus=1 #SBATCH --ntasks-per-gpu=1 ``` Is this the expected behaviour? The man page for DefCpuPerGPU says: "This value is used only if the job didn't specify --cpus-per-task and --cpus-per-gpu" which I do not specify, at least not directly. If yes is there a way to set something like DefCpuPerTask?

1 0

Setting memory by assigned node with a plugin
by laddaoui＠telecom-paris.fr 09 Jul '25

09 Jul '25

Hello everyone, I'm writing a job_submit/lua plugin to set the memory allocated to a job depending on which node is assigned by Slurm. However, it appears that the node where the job will run is not available at this step of the submission process. Would a SPANK plugin be able to access this information and modify the memory allocation accordingly? Or is there another approach you would recommend for dynamically setting memory based on the assigned node? Any insights would be greatly appreciated. Best regards, Nacereddine

1 0

Slurm versions 25.05.1 and 24.11.6 are now available
by Marshall Garey 08 Jul '25

08 Jul '25

We are pleased to announce the availability of Slurm versions 25.05.1 and 24.11.6. Changes in 25.05 include the following: * Fix many issues with the TLS Certificate Manager introduced in 25.05 * Optimize account deletion * Fix a bug when reordering the association hierarchy * Fix some issues that cause daemon crashes * Fix a variety of memory leaks Changes in 24.11 include the following: * Fix some issues that cause daemons to crash * Fix some race conditions on shutdown that cause daemons to crash or hang The full list of changes are available in the CHANGELOG for each version: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md https://github.com/SchedMD/slurm/blob/slurm-24.11/CHANGELOG/slurm-24.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Energy profile data not recorded in InfluxDB or HDF5
by Xand Meaden 02 Jul '25

02 Jul '25

Hi, Environment: Slurm 23.11.5, Ubuntu Linux 22.04 I'm attempting to setup accounting of job energy data, collected from IPMI (with acct_gather_energy/ipmi) and then written to InfluxDB (with acct_gather_profile/influxdb), using this documentation: https://slurm.schedmd.com/acct_gather.conf.html This is successfully working for non-energy data. However, no Energy data appears in the InfluxDB. If I switch to using HDF5 instead, there is also no Energy data recorded there. Here's the config being used: root@erc-hpc-comp06t:/etc/slurm# grep Acct /etc/slurm/slurm.conf JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/cgroup AcctGatherEnergyType=acct_gather_energy/ipmi AcctGatherProfileType=acct_gather_profile/influxdb root@erc-hpc-comp06t:/etc/slurm# cat acct_gather.conf # Energy accounting gatherer configuration EnergyIPMIFrequency=60 EnergyIPMICalcAdjustment=yes EnergyIPMIUsername=xx EnergyIPMIPassword=xxxxx # InfluxDB configuration - record Energy and Task data by default ProfileInfluxDBDefault=Energy,Task # Using stunnel to reach InfluxDB over TLS ProfileInfluxDBHost=localhost:8086 ProfileInfluxDBDatabase=slurmpr_hpc_test ProfileInfluxDBUser=slurmpr_hpc_test_rw ProfileInfluxDBPass=xx ProfileInfluxDBRTPolicy=autogen The Task data is recorded just fine. And I can see that the IPMI plugin is working: root@erc-hpc-comp06t:/etc/slurm# scontrol show node erc-hpc-comp06t | grep CurrentWatts CurrentWatts=85 AveWatts=84 Does anyone have this working? I really hope I'm missing something obvious here! Thanks, Xand -- Xand Meaden | Principal Research Infrastructure Engineer e-Research | King's College London

1 0

sbatch strange behavior with different --nodelist (-w) options
by Xinghong He 30 Jun '25

30 Jun '25

Dear Community, I'm seeing strange behavior from sbatch with different --nodelist (-w) options on my two node cluster. Here are my test scripts: *~/slurm$ cat mpirun.slm#!/bin/bash#SBATCH --job-name=mpirun_2x1#SBATCH --nodes=2#SBATCH --ntasks-per-node=1#SBATCH --exclusivesource /usr/mpi/gcc/openmpi-4.1.7a1/bin/mpivars.shmpirun ./a.sh* *~/slurm$ cat a.sh#!/bin/bashecho "`uname -n` OMPI_COMM_WORLD_RANK = $OMPI_COMM_WORLD_RANK SLURM_NODEID = $SLURM_NODEID SLURM_JOB_NODELIST = $SLURM_JOB_NODELIST"* If I do not specify any -w option or if I include both nodes in -w option, I get expected results. *~/slurm$ sbatch mpirun.slmSubmitted batch job 71~/slurm$ cat slurm-71.outstd-199 OMPI_COMM_WORLD_RANK = 0 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]std-271 OMPI_COMM_WORLD_RANK = 1 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]* However, if I specify only one node in -w option, but still want two nodes, I always get one expected result and one unexpected result. The unexpected one will dispatch both MPI tasks to the same node. This one is expected - running two MPI tasks across nodes. *~/slurm$ sbatch -w std-199 mpirun.slmSubmitted batch job 72~/slurm$ cat slurm-72.outstd-199 OMPI_COMM_WORLD_RANK = 0 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]std-271 OMPI_COMM_WORLD_RANK = 1 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]* This one if unexpected - ends up running two MPI tasks on the same node, though SLURM_JOB_NODELIST also gives the correct two nodes. *~/slurm$ sbatch -w std-271 mpirun.slmSubmitted batch job 73ubuntu@bright-anchovy-controller:~/slurm$ cat slurm-73.outstd-199 OMPI_COMM_WORLD_RANK = 0 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]std-199 OMPI_COMM_WORLD_RANK = 1 SLURM_NODEID = 0 SLURM_JOB_NODELIST = std-[199,271]* I first saw the problem on a larger cluster where I needed to specify both -w and -x options to include and exclude nodes. I then narrowed it down to a two node cluster. I tried adding options like --hostfile or --rankfile, or -npernode, all does not change how the tasks are dispatched to nodes. The problem is repeatable. Here are the tested systems: Slurm 23.02.5 on Ubuntu 22.04.5 LTS Slurm 24.05.1 on Ubuntu 22.04.4 LTS How to make the last one work. I.e., requesting *-w std-271* and make it run on two nodes. I'd appreciate any help! Regards, Xinghong

1 0

extern step does not exit
by william＠signalbox.org.uk 27 Jun '25

27 Jun '25

I am building a new small cluster on Rocky Linux 9 with Slurm 24.11.5. Slurm was compiled with the default options except that we added --with pmix We do use some additional complications - we use NFSv4 with AD authentication (so we use auks), and we use SElinux in enforcing mode. However, I do not currently see any evidence that those relate to the problem that we have; though note that the slurm account is an AD account. All relevant AD accounts have the RFC 2307 attributes (uid, gid etc.) configured. I had also configured pam_slurm_adopt but as that was causing issues I backed out the changes that enable it. It was too hard to troubleshoot without being able to login to the test compute node easily. The issue I am having in testing is that whatever job I launch, when it terminates the extern job step does not terminate. These are the more relevant slurm.conf settings: Epilog=/etc/slurm/epilog.sh JobAcctGatherType=jobacct_gather/cgroup JobCompType=jobcomp/none KillOnBadExit=1 KillWait=30 ProctrackType=proctrack/cgroup Prolog=/etc/slurm/prolog.sh PrologFlags=Alloc,Contain SelectTypeParameters=CR_Core_Memory SelectType=select/cons_tres SlurmUser=slurm TaskPlugin=task/affinity,task/cgroup TaskProlog=/etc/slurm/taskprolog.sh The slurmd.log on the compute node for the job shows a sequence like this (with -vv debug enabled): [2025-06-26T14:57:23.272] select/cons_tres: init: select/cons_tres loaded [2025-06-26T14:57:23.272] select/linear: init: Linear node selection plugin loaded with argument 20 [2025-06-26T14:57:23.272] cred/munge: init: Munge credential signature plugin loaded [2025-06-26T14:57:23.272] [34.extern] debug: auth/munge: init: loaded [2025-06-26T14:57:23.273] [34.extern] debug: Reading cgroup.conf file /etc/slurm/cgroup.conf [2025-06-26T14:57:23.279] [34.extern] debug: cgroup/v2: init: Cgroup v2 plugin loaded [2025-06-26T14:57:23.290] [34.extern] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2025-06-26T14:57:23.291] [34.extern] debug: CPUs:40 Boards:1 Sockets:2 CoresPerSocket:20 ThreadsPerCore:1 [2025-06-26T14:57:23.291] [34.extern] debug: task/cgroup: init: core enforcement enabled [2025-06-26T14:57:23.291] [34.extern] debug: task/cgroup: task_cgroup_memory_init: task/cgroup/memory: TotCfgRealMem:191587M allowed:100%(enforced), swap:0%(enforced), max:100%(191587M) max+swap:100%(383174M) min:30M [2025-06-26T14:57:23.291] [34.extern] debug: task/cgroup: init: memory enforcement enabled [2025-06-26T14:57:23.291] [34.extern] debug: task/cgroup: init: device enforcement enabled [2025-06-26T14:57:23.291] [34.extern] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2025-06-26T14:57:23.291] [34.extern] task/affinity: init: task affinity plugin loaded with CPU mask 0xffffffffff [2025-06-26T14:57:23.291] [34.extern] debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded [2025-06-26T14:57:23.291] [34.extern] topology/default: init: topology Default plugin loaded [2025-06-26T14:57:23.292] [34.extern] debug: gpu/generic: init: init: GPU Generic plugin loaded [2025-06-26T14:57:23.293] [34.extern] debug: Setting slurmstepd(2666071) oom_score_adj to -1000 [2025-06-26T14:57:23.293] [34.extern] debug: Message thread started pid = 2666071 [2025-06-26T14:57:23.294] [34.extern] debug: spank: opening plugin stack /etc/slurm/plugstack.conf [2025-06-26T14:57:23.294] [34.extern] debug: /etc/slurm/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf" [2025-06-26T14:57:23.294] [34.extern] debug: spank: opening plugin stack /etc/slurm/plugstack.conf.d/auks.conf [2025-06-26T14:57:23.298] [34.extern] debug: spank: /etc/slurm/plugstack.conf.d/auks.conf:57: Loaded plugin auks.so [2025-06-26T14:57:23.298] [34.extern] debug: SPANK: appending plugin option "auks" [2025-06-26T14:57:23.352] [34.extern] spank-auks: new unique ccache is KCM:71433:49222 <<<< This UID 71433 is the user running the job [2025-06-26T14:57:23.364] [34.extern] spank-auks: user '71433' cred stored in ccache KCM:71433:49222 [2025-06-26T14:57:23.380] [34.extern] debug: task/cgroup: task_cgroup_cpuset_create: job abstract cores are '0' [2025-06-26T14:57:23.381] [34.extern] debug: task/cgroup: task_cgroup_cpuset_create: step abstract cores are '0' [2025-06-26T14:57:23.381] [34.extern] debug: task/cgroup: task_cgroup_cpuset_create: job physical CPUs are '0' [2025-06-26T14:57:23.381] [34.extern] debug: task/cgroup: task_cgroup_cpuset_create: step physical CPUs are '0' [2025-06-26T14:57:23.381] [34.extern] task/cgroup: _memcg_initialize: job: alloc=1024MB mem.limit=1024MB memsw.limit=1024MB job_swappiness=18446744073709551614 [2025-06-26T14:57:23.381] [34.extern] task/cgroup: _memcg_initialize: step: alloc=1024MB mem.limit=1024MB memsw.limit=1024MB job_swappiness=18446744073709551614 [2025-06-26T14:57:23.385] [34.extern] debug: close_slurmd_conn: sending 0: No error [2025-06-26T14:57:24.371] launch task StepId=34.interactive request from UID:71433 GID:70668 HOST:172.17.11.22 PORT:46948 [2025-06-26T14:57:24.372] task/affinity: lllp_distribution: JobId=34 manual binding: mask_cpu,one_thread [2025-06-26T14:57:24.372] debug: Waiting for job 34's prolog to complete [2025-06-26T14:57:24.372] debug: Finished wait for job 34's prolog to complete [2025-06-26T14:57:24.377] select/cons_tres: init: select/cons_tres loaded [2025-06-26T14:57:24.377] select/linear: init: Linear node selection plugin loaded with argument 20 [2025-06-26T14:57:24.377] cred/munge: init: Munge credential signature plugin loaded [2025-06-26T14:57:24.377] [34.interactive] debug: auth/munge: init: loaded [2025-06-26T14:57:24.379] [34.interactive] debug: Reading cgroup.conf file /etc/slurm/cgroup.conf [2025-06-26T14:57:24.384] [34.interactive] debug: cgroup/v2: init: Cgroup v2 plugin loaded [2025-06-26T14:57:24.403] [34.interactive] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2025-06-26T14:57:24.404] [34.interactive] debug: CPUs:40 Boards:1 Sockets:2 CoresPerSocket:20 ThreadsPerCore:1 [2025-06-26T14:57:24.404] [34.interactive] debug: task/cgroup: init: core enforcement enabled [2025-06-26T14:57:24.404] [34.interactive] debug: task/cgroup: task_cgroup_memory_init: task/cgroup/memory: TotCfgRealMem:191587M allowed:100%(enforced), swap:0%(enforced), max:100%(191587M) max+swap:100%(383174M) min:30M [2025-06-26T14:57:24.404] [34.interactive] debug: task/cgroup: init: memory enforcement enabled [2025-06-26T14:57:24.404] [34.interactive] debug: task/cgroup: init: device enforcement enabled [2025-06-26T14:57:24.404] [34.interactive] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2025-06-26T14:57:24.404] [34.interactive] task/affinity: init: task affinity plugin loaded with CPU mask 0xffffffffff [2025-06-26T14:57:24.404] [34.interactive] debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded [2025-06-26T14:57:24.404] [34.interactive] topology/default: init: topology Default plugin loaded [2025-06-26T14:57:24.404] [34.interactive] debug: gpu/generic: init: init: GPU Generic plugin loaded [2025-06-26T14:57:24.406] [34.interactive] debug: close_slurmd_conn: sending 0: No error [2025-06-26T14:57:24.406] [34.interactive] debug: Message thread started pid = 2666089 [2025-06-26T14:57:24.406] [34.interactive] debug: Setting slurmstepd(2666089) oom_score_adj to -1000 [2025-06-26T14:57:24.407] [34.interactive] debug: spank: opening plugin stack /etc/slurm/plugstack.conf [2025-06-26T14:57:24.407] [34.interactive] debug: /etc/slurm/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf" [2025-06-26T14:57:24.407] [34.interactive] debug: spank: opening plugin stack /etc/slurm/plugstack.conf.d/auks.conf [2025-06-26T14:57:24.411] [34.interactive] debug: spank: /etc/slurm/plugstack.conf.d/auks.conf:57: Loaded plugin auks.so [2025-06-26T14:57:24.411] [34.interactive] debug: SPANK: appending plugin option "auks" [2025-06-26T14:57:24.461] [34.interactive] spank-auks: new unique ccache is KCM:71433:44156 [2025-06-26T14:57:24.467] [34.interactive] spank-auks: user '71433' cred stored in ccache KCM:71433:44156 [2025-06-26T14:57:24.480] [34.interactive] debug: task/cgroup: task_cgroup_cpuset_create: job abstract cores are '0' [2025-06-26T14:57:24.480] [34.interactive] debug: task/cgroup: task_cgroup_cpuset_create: step abstract cores are '0' [2025-06-26T14:57:24.480] [34.interactive] debug: task/cgroup: task_cgroup_cpuset_create: job physical CPUs are '0' [2025-06-26T14:57:24.480] [34.interactive] debug: task/cgroup: task_cgroup_cpuset_create: step physical CPUs are '0' [2025-06-26T14:57:24.481] [34.interactive] task/cgroup: _memcg_initialize: job: alloc=1024MB mem.limit=1024MB memsw.limit=1024MB job_swappiness=18446744073709551614 [2025-06-26T14:57:24.481] [34.interactive] task/cgroup: _memcg_initialize: step: alloc=1024MB mem.limit=1024MB memsw.limit=1024MB job_swappiness=18446744073709551614 [2025-06-26T14:57:24.501] [34.interactive] warning: restricted to a subset of cpus [2025-06-26T14:57:24.503] [34.interactive] debug: stdin uses a pty object [2025-06-26T14:57:24.504] [34.interactive] debug: init pty size 48:324 [2025-06-26T14:57:24.506] [34.interactive] debug levels are stderr='error', logfile='debug', syslog='quiet' [2025-06-26T14:57:24.507] [34.interactive] debug: IO handler started pid=2666089 [2025-06-26T14:57:24.517] [34.interactive] spank-auks: credential renewer launched (pid=2666107) [2025-06-26T14:57:24.517] [34.interactive] starting 1 tasks [2025-06-26T14:57:24.517] [34.interactive] task 0 (2666108) started 2025-06-26T14:57:24 [2025-06-26T14:57:24.537] [34.interactive] debug: task/affinity: task_p_pre_launch: affinity StepId=34.interactive, task:0 bind:mask_cpu,one_thread [2025-06-26T14:57:24.537] [34.interactive] debug: [job 34] attempting to run slurm task_prolog [/etc/slurm/taskprolog.sh] [2025-06-26T14:57:24.537] [34.interactive] debug: Sending launch resp rc=0 [2025-06-26T14:57:24.563] [34.interactive] debug: export name:TMPDIR:val:/local/scratch/34: [2025-06-26T14:59:42.726] [34.interactive] task 0 (2666108) exited with exit code 0. <<<< Here the interactive job was closed [2025-06-26T14:59:42.727] [34.interactive] spank-auks: all tasks exited, killing credential renewer (pid=2666107) [2025-06-26T14:59:42.729] [34.interactive] debug: task/affinity: task_p_post_term: affinity StepId=34.interactive, task 0 [2025-06-26T14:59:42.729] [34.interactive] debug: signaling condition [2025-06-26T14:59:42.729] [34.interactive] debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded [2025-06-26T14:59:42.729] [34.interactive] debug: Waiting for IO [2025-06-26T14:59:42.729] [34.interactive] debug: Closing debug channel [2025-06-26T14:59:42.729] [34.interactive] debug: IO handler exited, rc=0 [2025-06-26T14:59:42.729] [34.interactive] debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded [2025-06-26T14:59:42.731] [34.interactive] debug: slurm_recv_timeout at 0 of 4, recv zero bytes [2025-06-26T14:59:42.738] [34.interactive] spank-auks: Destroyed ccache KCM:71433:44156 [2025-06-26T14:59:42.740] debug: _rpc_terminate_job: uid = 71755 JobId=34 <<<< That uid 71755 is for the slurm account which does not run any daemons on a compute node [2025-06-26T14:59:42.740] debug: credential for job 34 revoked [2025-06-26T14:59:42.742] [34.extern] debug: Handling REQUEST_SIGNAL_CONTAINER [2025-06-26T14:59:42.742] [34.extern] debug: _handle_signal_container for StepId=34.extern uid=71755 signal=18 flag=0x0 [2025-06-26T14:59:42.742] [34.extern] Sent signal 18 to StepId=34.extern [2025-06-26T14:59:42.742] [34.interactive] debug: Handling REQUEST_SIGNAL_CONTAINER [2025-06-26T14:59:42.742] [34.interactive] debug: _handle_signal_container for StepId=34.interactive uid=71755 signal=18 flag=0x0 [2025-06-26T14:59:42.742] [34.interactive] Sent signal 18 to StepId=34.interactive [2025-06-26T14:59:42.743] [34.extern] debug: Handling REQUEST_SIGNAL_CONTAINER [2025-06-26T14:59:42.743] [34.extern] debug: _handle_signal_container for StepId=34.extern uid=71755 signal=15 flag=0x0 [2025-06-26T14:59:42.743] [34.extern] Sent signal 15 to StepId=34.extern [2025-06-26T14:59:42.743] [34.interactive] debug: Handling REQUEST_SIGNAL_CONTAINER [2025-06-26T14:59:42.743] [34.interactive] debug: _handle_signal_container for StepId=34.interactive uid=71755 signal=15 flag=0x0 [2025-06-26T14:59:42.743] [34.interactive] Sent signal 15 to StepId=34.interactive [2025-06-26T14:59:42.744] [34.interactive] debug: Handling REQUEST_STATE [2025-06-26T14:59:42.744] [34.interactive] debug: Message thread exited [2025-06-26T14:59:42.744] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:42.763] [34.interactive] done with step [2025-06-26T14:59:42.766] [34.extern] debug: signaling condition [2025-06-26T14:59:42.766] [34.extern] debug: task/affinity: task_p_post_term: affinity StepId=34.extern, task 0 [2025-06-26T14:59:42.766] [34.extern] debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded [2025-06-26T14:59:42.766] [34.extern] debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded [2025-06-26T14:59:42.766] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:42.766] [34.extern] debug: Terminate signal (SIGTERM) received [2025-06-26T14:59:42.768] [34.extern] error: setgroups: Operation not permitted [2025-06-26T14:59:42.768] [34.extern] error: _shutdown_x11_forward: Unable to drop privileges [2025-06-26T14:59:42.771] [34.extern] spank-auks: Destroyed ccache KCM:71433:49222 [2025-06-26T14:59:42.817] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:42.918] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:43.419] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:44.420] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:45.420] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:46.421] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:47.422] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:48.423] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:49.424] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:50.425] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:51.426] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:52.426] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T14:59:53.427] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:03.428] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:12.429] [34.extern] debug: Handling REQUEST_STEP_TERMINATE [2025-06-26T15:00:12.429] [34.extern] debug: _handle_terminate for StepId=34.extern uid=0 [2025-06-26T15:00:12.429] [34.extern] Sent SIGKILL signal to StepId=34.extern [2025-06-26T15:00:12.429] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:12.450] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:12.501] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:12.602] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:13.102] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:14.103] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:14.104] [34.extern] debug: Handling REQUEST_STEP_TERMINATE [2025-06-26T15:00:14.104] [34.extern] debug: _handle_terminate for StepId=34.extern uid=0 [2025-06-26T15:00:14.104] [34.extern] Sent SIGKILL signal to StepId=34.extern [2025-06-26T15:00:15.105] [34.extern] debug: Handling REQUEST_STATE [2025-06-26T15:00:15.105] [34.extern] debug: Handling REQUEST_STEP_TERMINATE . . We are left with a process "slurmstepd: [34.extern]" running, and a matching socket file for job 34 in /var/spool/slurmd; the job shows state 'CG' (Completing). I can clean up by issuing a kill -9 to the slurmstepd 'extern' job on the compute node. I do not know a way to get any logging output from that job step, but it has a file descriptor open to /var/log/slurm/slurmd.log (which is what we configured) so I assume that it is writing some of the above log. In some runs I also see this just as the main task exits (but from the .0 job step, not from .extern) : [2025-06-27T21:00:47.265] [39.0] error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/slurmstepd.scope/job_39/step_0/user/cgroup.free ze' for writing: Permission denied I can see that between reboots, the directories '/sys/fs/cgroup/system.slice/slurmstepd.scope/job_NN' remain, with the /sys/fs/cgroup/system.slice/slurmstepd.scope/job_NN/step_extern folder. The permissions of the files referred to as 'permission denied' seem fine to me: # ls -lhZ /sys/fs/cgroup/system.slice/slurmstepd.scope/job_42/step_extern/user/cgroup. freeze -rw-r--r--. 1 root root system_u:object_r:cgroup_t:s0 0 Jun 27 21:46 /sys/fs/cgroup/system.slice/slurmstepd.scope/job_42/step_extern/user/cgroup. freeze There are no SElinux alerts on the system. I am not sure whether the error messages about setgroups and _shutdown_x11_forward are actually the problem, or just something else being reported. The only system that I have to compare with is running Slurm 19.05 on CentOS 7 and is rather different. I would be interested to know if anyone else has had problems with extern job steps not shutting down. William

1 0

Job information if job is completed
by Gestió Servidors 24 Jun '25

24 Jun '25

Hello, Is there any way to get all information (like submit script or submit node) from a job that is completed? Something like "scontrol show jobid=XXX" when job is "running" or "pending". I need to inspect the submit script of a job but I only know job_id. Thanks.

6 5

Implementing a "soft" wall clock limit
by Davide DelVento 24 Jun '25

24 Jun '25

In the institution where I work, so far we have managed to live without mandatory wallclock limits (a policy decided well before I joined the organization), and that has been possible because the cluster was not very much utilized. Now that is changing, with more jobs being submitted and those being larger ones. As such I would like to introduce wallclock limits to allow slurm to be more efficient in scheduling jobs, including with backfill. My concern is that this user base is not used to it and therefore I want to make it easier for them, and avoid common complaints. I anticipate one of them would be "my job was cancelled even though there were enough nodes idle and no other job in line after mine" (since the cluster utilization is increasing, but not yet always full like it has been at most other places I know). So my question is: is it possible to implement "soft" wallclock limits in slurm, namely ones which would not be enforced unless necessary to run more jobs? In other words, is it possible to change the pre-emptability of a job only after some time has passed? I can think of some ways to hack this functionality myself with some cron or at jobs, and that might be easy enough to do, but I am not sure I can make it robust enough to cover all situations, so I'm looking for something either slurm-native or (if external solution) field-tested by someone else already, so that at least the worst kinks have been already ironed out. Thanks in advance for any suggestions you may provide!

7 14

enforce Qos to users
by laddaoui＠telecom-paris.fr 24 Jun '25

24 Jun '25

Hello everyone, I'm trying to use QoS to enforce resource limits on an association, but I'm having trouble with proper enforcement. I created a QoS with resource limits: ``` sacctmgr add qos qos_gpus flags=denyonlimit,overpartqos maxjobsperuser=4 maxtresperjob=gres/gpu=1 ``` Then I assigned it to an account: ``` sacctmgr modify account name=account-a set qos=qos_gpus defaultqos=qos_gpus systemctl restart slurmctld ``` Users in this account can bypass the QoS limits by explicitly specifying a different QoS when submitting jobs: ``` srun --qos=(normal|qos_gpus) ... ``` Even though I set `defaultqos=qos_gpus`, users can still choose any available QoS and bypass the intended resource limits. My question is: How can I restrict users to only using their assigned QoS and prevent them from specifying other QoS options? Is there a configuration I'm missing to enforce QoS restrictions properly? Best, --- info about my setup slurm version : tested on 23.11.4 and 23.02.7 AccountingStorageEnforce = associations,limits EnforcePartLimits = ALL

2 2

pam_slurm_adopt - ssh to compute nodes not working in slurm 24.11
by Marx, Wolfgang 23 Jun '25

23 Jun '25

Hi, We have defined in our cluster, that users can logon to a compute node from a login node,when they have actual a job running on the compute node. To get this functionallity working, we are usinge the pam_slurm_adopt. As long as we were using slurm 23.05 it all was working well. Now we have upgraded to slurm 24.11.5 and the ssh login to the compute nodes is not longer working. When a job of a user is running on a compute node the ssh of the user to this compute node is refused form the compute node. We have not changed anything in our configuration for the pam_slurm_adopt. Also there is no indication in the release notes that anything has changed regarding pam_slurm_adopt. Is this a known bug in Slurm 24.11 and has anyone facing the same problem. This is a very important feature, especially for our ANSYS users. Thanks Wolfgang Marx Wolfgang Marx, Basisdienste, Gruppe Hochleistungrechnen Technische Universität Darmstadt, Hochschulrechenzentrum Alexanderstraße 2, 64283 Darmstadt Tel.: +496151/16-71158 E-Mail: wolfgang.marx(a)tu-darmstadt.de Web: www.hrz.tu-darmstadt.de

2 1

read-only slurm user
by Hagdorn, Magnus Karl Moritz 23 Jun '25

23 Jun '25

Hi there, we use the slurm prometheus exporter to collect slurm metrics. This works pretty well. However, we have noticed that metrics for some of the restricted partitions are not collected. It occurred to me that this is because we are using an unprivileged user to run the exporter. I am trying to figure out the best way to allow an unprivileged user to collect all metrics. I could add the user to all the relevant groups. However, I am also thinking of using the new slurm exporter that uses the API in which case I need to somehow handle a token. It would be nice to have a readonly user, ie a user that cannot submit any jobs but only read the current state of the cluster. I guess setting MaxJobs and MaxSubmitJobs to 0 would do this. Any other suggestions? Regards magnus -- Dr. Magnus Hagdorn Charité – Universitätsmedizin Berlin Geschäftsbereich IT | Scientific Computing Campus Charité Mitte BALTIC - Invalidenstraße 120/121 10115 Berlin https://www.charite.de HPC Helpdesk: sc-hpc-helpdesk(a)charite.de

1 0

Doc Clarification: Heterogeneous Steps in Heterogeneous Job
by Steffen Christgau 19 Jun '25

19 Jun '25

Hi everybody, I am (along with others) a little bit puzzled by the meaning of a statement in the documentation concerning heterogeneous job steps inside het. jobs. The docs state (https://slurm.schedmd.com/archive/slurm-24.11.5/heterogeneous_jobs.html#het…) > You also cannot request heterogeneous steps from within a heterogeneous job. (A) On a very small Slurm test installation with just two nodes, the following het job that requests het steps (does it, right?!) runs fine: $ cat hetjob-steps.sh #!/bin/bash #SBATCH --mem-per-cpu=2g --nodes=1 --cpus-per-task=8 #SBATCH hetjob #SBATCH --mem-per-cpu=1g --nodes=1 --cpus-per-task=4 srun -l --cpus-per-task=4 nproc : -l --cpus-per-task=2 nproc $ cat slurm-125.out 1: 4 2: 2 3: 2 0: 4 The output looks reasonable and it looks like the above quote does not apply since one can apparently request het steps in a het job. Or am I wrong? The intro in the respective section also gives the impression that het jobsteps are a convenience feature that does not require het jobs, but it does not explicitly exclude the usage of het steps in het jobs: > Slurm version 20.11 introduces the ability to request heterogeneous job steps from within a non-homogeneous job allocation. This allows you the flexibility to have different layouts for job steps without requiring the use of heterogeneous jobs, where having separate jobs for the components may be undesirable. So what does the initial statement (A) actually mean then? Am I just using a lucky example which is actually not supported? A short clarification would be helpful. Thanks in advance Steffen

1 0

Wrong MaxRSS Behavior with cgroup v2 in Slurm
by Guillaume COCHARD 19 Jun '25

19 Jun '25

Hello, We've noticed a recent change in how MaxRSS is reported on our cluster. Specifically, the MaxRSS value for many jobs now often matches the allocated memory, which was not the case previously. It appears this change is due to how Slurm accounts for memory when copying large files, likely as a result of moving from cgroup v1 to cgroup v2. Here’s a simple example: copy_file.sh #!/bin/bash cp /distributed/filesystem/file5G /tmp cp /tmp/file5G ~ Two jobs with different memory allocations: Job 1 sbatch -c 1 --mem=1G copy_file.sh seff <jobid> Memory Utilized: 1021.87 MB Memory Efficiency: 99.79% of 1.00 GB Job 2 sbatch -c 1 --mem=10G copy_file.sh seff <jobid> Memory Utilized: 4.02 GB Memory Efficiency: 40.21% of 10.00 GB With cgroup v1, this script typically showed minimal memory usage. Now, under cgroup v2, memory usage appears inflated and depends on the allocated memory, which seems wrong. I believe this behavior aligns with similar issues raised by the Kubernetes community [1], and is consistent with how memory.current behaves in cgroup v2 [3]. According to Slurm’s documentation about cgroup v2, "this plugin provides cgroup's memory.current value from the memory interface, which is not equal to the RSS value provided by procfs. Nevertheless it is the same value that the kernel uses in its OOM killer logic." [2] While technically correct, this seems to mark a significant change in what MaxRSS and "Memory Efficiency" actually measure and renders those metrics almost useless. Our Configuration: ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup,task/affinity Question: Is there a way to restore more realistic MaxRSS values — specifically, ones that exclude file-backed page cache — while still using cgroup v2? Thanks, Guillaume References: [1] https://github.com/kubernetes/kubernetes/issues/118916 [2] https://slurm.schedmd.com/cgroup_v2.html#limitations [3] https://facebookmicrosites.github.io/cgroup2/docs/memory-controller.html

3 4

Re: Job information if job is completed
by Gestió Servidors 18 Jun '25

18 Jun '25

Hi, After reconfiguring slurm.conf to add script into the database, I have tried with a "normal" user to get that information (from a owned finished job). However, when I run "sacct -vvvv -B -j 92656", I get: sacct: Jobs Eligible in the time window from Epoch 0 to Wed Jun 18 15:33:03 2025 sacct: debug: Options selected: opt_completion=no opt_dup=no opt_field_list=User,JobID,Jobname%18,partition,state,time,submit,start,end,elapsed,nnodes,ncpus,nodelist, opt_help=0 opt_no_steps=yes opt_whole_hetjob=(null) sacct: debug: accounting_storage/slurmdbd: _connect_dbd_conn: Sent PersistInit msg sacct: debug2: Clusters requested: mycluster sacct: debug2: Userids requested: all sacct: debug2: Jobs requested: sacct: debug2: : 92656 sacct: error: Unknown error 1064 I have read that "sacct: error: Unknown error 1064" could be a error in the MySQL query syntax... but if I run same command "sacct -vvvv -B -j 92656" as root, I get submit script. I suppose problem is in database permissions, so I have added new permissions with "grant all on slurmdb.* TO 'user-sacct'@'localhost' identified by '..user-sacct..' with grant option;" (where user "user-sacct" is a user that acts as coordinator in SLURM). However, if I open a terminal with "user-sacct" and run "sacct -vvvv -B -j 92656", result is the same: "sacct: error: Unknown error 1064" How could give permissions to user-sacct to allow "sacct -B" command? Thanks.

1 0

Re: Job information if job is completed
by Gestió Servidors 18 Jun '25

18 Jun '25

Hello, So if there is no way to get all information from a finished job (specially submit script, not command line, but all content from submit script, like a copy of it), maybe a solution would be run a "prolog" script in each job to run a "cp" from the submit script. However, how could I copy the submit scrip from the prolog script? Because from the prolog script, I can access to some SLURM variables (https://slurm.schedmd.com/prolog_epilog.html) but I don't know how to know what the script is and run a simple "cp" to a destination folder. Thanks.

4 4

Re: [EXT] Re: slurm_pam_adopt module not working
by Ratnasamy, Fritz 17 Jun '25

17 Jun '25

Yes the file exists in /usr/lib64/security/. Best, *Fritz Ratnasamy*Data Scientist Information Technology On Tue, Jun 17, 2025 at 12:17 AM Sean Crosby <scrosby(a)unimelb.edu.au> wrote: > Hi Fritz, > > Does pam_slurm_adopt.so exist in the right location on the node? Normally > on EL hosts it would be /usr/lib64/security/pam_slurm_adopt.so > > # ls /usr/lib64/security/pam_slurm_adopt.so -la > -rwxr-xr-x 1 root root 291936 Mar 4 12:44 > /usr/lib64/security/pam_slurm_adopt.so > > If the file doesn't exist, pam would abnormally exit and not allow anyone > to log in. > > Sean > ------------------------------ > *From:* Ratnasamy, Fritz via slurm-users <slurm-users(a)lists.schedmd.com> > *Sent:* Tuesday, 17 June 2025 14:55 > *To:* Kevin Buckley <kevin.buckley.pawsey.org.au(a)gmail.com> > *Cc:* slurm-users(a)lists.schedmd.com <slurm-users(a)lists.schedmd.com> > *Subject:* [EXT] [slurm-users] Re: slurm_pam_adopt module not working > > * External email: Please exercise caution * > ------------------------------ > Thanks, for some reason I edited the /etc/pam.d/sshd via ansible but that > locked all users to the cluster. That same file works on a different > cluster where the files are pushed via puppet but with ansible it looks > like it is locking all users to the cluster. See below config file sshd: > > auth required pam_sepermit.so > auth substack password-auth > auth include postlogin > # Used with polkit to reauthorize users in remote sessions > -auth optional pam_reauthorize.so prepare > account required pam_nologin.so > ##SLURM > account sufficient pam_slurm_adopt.so action_no_jobs=deny > action_unknown=newest action_adopt_failure=deny action_generic_failure=deny > account sufficient pam_access.so > ##END SLURM > password include password-auth > # pam_selinux.so close should be the first session rule > session required pam_selinux.so close > session required pam_loginuid.so > # pam_selinux.so open should only be followed by sessions to be executed > in the user context > session required pam_selinux.so open env_params > session required pam_namespace.so > session optional pam_keyinit.so force revoke > session include password-auth > session include postlogin > # Used with polkit to reauthorize users in remote sessions > -session optional pam_reauthorize.so prepare > > > > *Fritz Ratnasamy *Data Scientist > Information Technology > > > > > On Wed, Jun 11, 2025 at 8:29 PM Kevin Buckley via slurm-users < > slurm-users(a)lists.schedmd.com> wrote: > > On 2025/06/11 12:46, Ratnasamy, Fritz via slurm-users wrote: > > > > We wanted to block users from ssh to a node where there are no jobs > > running however it looks like users are able to do so. We have installed > > the slurm_pam_adopt_module and set up the slurm.conf accordingly (the > same > > way we did on our first cluster where the pam module denies ssh access > > correctly). > > We saw a similar issue whereby the way that we had PAM setup, meant > that, and here I quote from SchedMD's Daniel Armengod: > > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---- > This is almost certainly caused by the fact that SSH's > `keyboard-interactive` > (not to be confused with `password`) AuthMethod forks a short-lived child > process that is involved in the authentication logic. Slurm's > pam_slurm_adopt > module latches on to that process (which is the wrong one, of course) and > things break in interesting ways from there. > > SSH authmethods `publickey` and `password` do not exhibit this behaviour > as SSH > does not fork a child process to offload the authentication > challenge-response > dialogue to. > > ... > > The key bit here is that in your last test you're forcing > `PreferredAuthentications=password`, which isn't actually the > `keyboard-interactive` AuthMethod that got picked before. > They work differently under the hood, even if as far as the > user is concerned, both methods just ask for a password. > > ... > > In summary: try disabling the `keyboard-interactive` authentication method > in > your compute nodes. pam_slurm_adopt should work correctly now. > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---- > > Maybe that's also your issue. > > > Daniel did say that SchedMD were going to update their documentation > to make that distinction, and it's effect, more explciit, so I would > expect it to be in the mainstream docs by now. > > HTH > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > CAUTION: This email has originated outside of University email systems. > Please do not click links or open attachments unless you recognize the > sender and trust the contents as safe. > > CAUTION: This email has originated outside of University email systems. > Please do not click links or open attachments unless you recognize the > sender and trust the contents as safe. > >

2 1

Re: slurm_pam_adopt module not working
by Ratnasamy, Fritz 17 Jun '25

17 Jun '25

Thanks, for some reason I edited the /etc/pam.d/sshd via ansible but that locked all users to the cluster. That same file works on a different cluster where the files are pushed via puppet but with ansible it looks like it is locking all users to the cluster. See below config file sshd: auth required pam_sepermit.so auth substack password-auth auth include postlogin # Used with polkit to reauthorize users in remote sessions -auth optional pam_reauthorize.so prepare account required pam_nologin.so ##SLURM account sufficient pam_slurm_adopt.so action_no_jobs=deny action_unknown=newest action_adopt_failure=deny action_generic_failure=deny account sufficient pam_access.so ##END SLURM password include password-auth # pam_selinux.so close should be the first session rule session required pam_selinux.so close session required pam_loginuid.so # pam_selinux.so open should only be followed by sessions to be executed in the user context session required pam_selinux.so open env_params session required pam_namespace.so session optional pam_keyinit.so force revoke session include password-auth session include postlogin # Used with polkit to reauthorize users in remote sessions -session optional pam_reauthorize.so prepare *Fritz Ratnasamy*Data Scientist Information Technology On Wed, Jun 11, 2025 at 8:29 PM Kevin Buckley via slurm-users < slurm-users(a)lists.schedmd.com> wrote: > On 2025/06/11 12:46, Ratnasamy, Fritz via slurm-users wrote: > > > > We wanted to block users from ssh to a node where there are no jobs > > running however it looks like users are able to do so. We have installed > > the slurm_pam_adopt_module and set up the slurm.conf accordingly (the > same > > way we did on our first cluster where the pam module denies ssh access > > correctly). > > We saw a similar issue whereby the way that we had PAM setup, meant > that, and here I quote from SchedMD's Daniel Armengod: > > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---- > This is almost certainly caused by the fact that SSH's > `keyboard-interactive` > (not to be confused with `password`) AuthMethod forks a short-lived child > process that is involved in the authentication logic. Slurm's > pam_slurm_adopt > module latches on to that process (which is the wrong one, of course) and > things break in interesting ways from there. > > SSH authmethods `publickey` and `password` do not exhibit this behaviour > as SSH > does not fork a child process to offload the authentication > challenge-response > dialogue to. > > ... > > The key bit here is that in your last test you're forcing > `PreferredAuthentications=password`, which isn't actually the > `keyboard-interactive` AuthMethod that got picked before. > They work differently under the hood, even if as far as the > user is concerned, both methods just ask for a password. > > ... > > In summary: try disabling the `keyboard-interactive` authentication method > in > your compute nodes. pam_slurm_adopt should work correctly now. > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---- > > Maybe that's also your issue. > > > Daniel did say that SchedMD were going to update their documentation > to make that distinction, and it's effect, more explciit, so I would > expect it to be in the mainstream docs by now. > > HTH > > -- > slurm-users mailing list -- slurm-users(a)lists.schedmd.com > To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com > CAUTION: This email has originated outside of University email systems. > Please do not click links or open attachments unless you recognize the > sender and trust the contents as safe. > >

2 1

MIG H100 with xeon Intel
by Richard Lefebvre 17 Jun '25

17 Jun '25

I'm having problems with Autodetect=nvml in gres.conf. I get on the controller log the following: error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 16-31 on node node001 doesn't match socket boundaries. (Socket 0 is cores 0-31). Consider setting SlurmdParameters=l3cache_as_socket (recommended) or override this by manually specifying core affinity in gres.conf. I did set l3cache_as_socket in the slurm.conf of the node, but I still get the error on the slurm controler I'm running 24.11.5 on AlmaLinux 9.5 Richard

2 1

How are the results produced by 'seff'?
by Loris Bennett 12 Jun '25

12 Jun '25

Hi, With Slurm 24.11.5 for some jobs I am seeing differences between the memory usage reported by 'seff' and that shown by Prometheus as 'cgroup_memory_rss_bytes' (and ultimately reported by 'jobstats' [1]). Certainly at the University of Delft they seem to feel that the memory usage reported by 'seff' is unreliable [2]. Is that indeed the case? Cheers, Loris Footnotes: [1] https://github.com/PrincetonUniversity/jobstats [2] https://doc.dhpc.tudelft.nl/delftblue/Slurm-trouble-shooting/ -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin

1 0

slurm_pam_adopt module not working
by Ratnasamy, Fritz 11 Jun '25

11 Jun '25

Hi, We wanted to block users from ssh to a node where there are no jobs running however it looks like users are able to do so. We have installed the slurm_pam_adopt_module and set up the slurm.conf accordingly (the same way we did on our first cluster where the pam module denies ssh access correctly). We have PrologFlags=contain in slurm.conf and inside /etc/pam.d/sshd, we set the last row to : account sufficient pam_slurm_adopt.so action_no_jobs=deny action_unknown=newest action_adopt_failure=deny action_generic_failure=deny Are we missing anything? Thanks *Fritz Ratnasamy*Data Scientist Information Technology

3 2

CR_CPU used but only cores used
by Adrian Sevcenco 11 Jun '25

11 Jun '25

Hi! i have a weird situation in which only cores are used instead of CPUs this is Alma9/slurm 22.05.9 (the last one from epel) I have: conf.d/resources.conf 9:SelectType=select/cons_res conf.d/resources.conf 11:SelectTypeParameters=CR_CPU conf.d/resources.conf 5:TaskPluginParam=autobind=threads conf.d/resources.conf 2:TaskPlugin=task/affinity,task/cgroup cgroup.conf 2:ConstrainCores=no conf.d/parts_issaf.conf 8:PartitionName=CLUSTER DEFAULT=YES State=UP nodes=DEFAULT DefaultTime=240:00:00 MaxTime=480:00:00 LLN=yes MaxNodes=1 What am i missing? Why are only cores used? Thanks a lot!! Adrian

2 2

Restrict and prioritize usage of certain nodes according to accounts
by thomas.hartmann＠th-ht.de 09 Jun '25

09 Jun '25

Hi, I'm going to have the following situation on my hands and I would like to ask for some suggestions how to solve this in slurm. We're talking about a cluster that is not yet operational so there are no real legacy configs that one needs to take into account. I have two sets of physical nodes: 1. node01 - nodeXX: General Purpose Nodes. Everybody can use them 2. project_A_01 - project_A_XX: These nodes come with some restrictions regarding who can and who must not run jobs on them. Users are organized into accounts, pseudohierarchically representing workgroups / projects etc within the university. For instance, there is going to be a "physics" account with "project_blackhole", "workgroup_miller" as children. Users are associated with one or more accounts. I have three sets of accounts (each can have child accounts): 1. "General" accounts: These are allowed to use all physical nodes. 2. "ForProfit" accounts: These absolutely must not use the project_A_* nodes 3. "Project_A" accounts: Their jobs should run first and foremost on the project_A_* nodes but they are also allowed to run on the node* nodes. My first idea would be to create two partitions, one with all nodes in there and a second one with only those nodes that the "ForProfit" are allowed to use. Using "AllowAccounts" or "DenyAccounts" would implement the restriction I need. In the future, there might be more project specific nodes for other projects. I would be grateful for any alternative solution or hints, how this could be solved optimally. Thanks a lot! Thomas

5 4

Problem querying slurm batch script archive
by Davide DelVento 09 Jun '25

09 Jun '25

I configured slurm.conf on our machine with AccountingStoreFlags=job_script,job_comment So that people can run sacct -Bj <job_ID> and check what the batch script was for the specified job_ID. This works fine for root and works great for many users (who can see their own job scripts, but not those of other users, as it ought to be). Unfortunately, for several users (but not even close to half of the users of the machine), the sacct -Bj command returns nothing, like if they queried someone else's job, even when querying their own. Does anybody have any clue on what might be wrong and triggering this odd behavior? Thanks!

1 0

seff command not found
by Ratnasamy, Fritz 09 Jun '25

09 Jun '25

Hi, We installed a new slurm version and it returns "command not found" for seff. I do not remember doing any manual installation for the previous versions, I thought it was coming with sacct, sbatch, ect... Any idea how I would need to set it up? I read online seff is actually a perl script. Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Re: Integration testing of Slinky images
by Nicolas Bock 06 Jun '25

06 Jun '25

Hi Brian, sorry for breaking the email thread, my subscription settings were not set correctly and I didn't receive your response as email. Thanks for the tips! I'll give that a try. Best, Nick > Nick, > Presuming you have followed the SchedMD instructions, you should be able > to get a session in your login container: > kubectl --namespace=slurm exec -it statefulsets/slurm-controller -- bash > --login > From there, you can do any standard testing you like. Simple 'srun > hostname' should work to let you know slurm itself is doing it's part. > You can also do commands such as 'scontrol show nodes' to see what and > how many resources you have configured. > Outside that, you need to ensure you have configured your slurm > containers to request the resources you plan on using (eg: gpus) and > that you have enough of them for the script(s) you wish to run. > Brian Andrus

1 0

X11 Forwarding Crashes X Session for too large windows
by Felix Winterhalter 06 Jun '25

06 Jun '25

Hello everyone, we have run into a peculiar issue with our setup. First and foremost let me say that X forwarding using ssh -X or ssh -Y works fine and doesn't show any irregularities. Trying to use the X forwarding of slurm however results in a weird situation. We are using it like: srun --pty --x11 bash And then run something like glxgears. The program starts up normal however whenever you increase the window size beyond a certain size the connection to the X server gets broken and glxgears crashes. The same is true for any other program using X, some don't even start as their inital window size is too large, others crash as soon as you resize them beyond a certain point. We have no clue what might be the cause only that it seems to be related to how slurm does the XForwarding. We've tried X11Parameters both with and without host_authority. We're running 25.05.0 Any help or idea would be very appreciated. Cheers, Felix This message and any attachments are for the sole use of the intended recipient(s). It may contain information that is confidential and privileged. If you are not the intended recipient of this message, you are prohibited from printing, copying, forwarding or saving it. Please delete the message and attachments and notify the sender immediately.

1 1

Integration testing of Slinky images
by Nicolas Bock 06 Jun '25

06 Jun '25

Hi, I was wondering whether there is any integration testing done on the containers and the operator published under the slinky project. Or asked differently, once I have deployed slinky, what tests should I run to confirm that the cluster is working as intended? Thanks! Nick

2 1

X11 performance terrible using plugin
by Simon Andrews 06 Jun '25

06 Jun '25

On our cluster we've noticed that if we use the native x11 slurm plugin (PrologFlags=x11) then X applications work, but are really slow and unresponsive. Even opening menus on graphical application is painfully slow. On the same system if I do a direct ssh connection with ssh -YC from the head node the same applications are quick and responsive. Any suggestions for what might be causing this, and how I can get the native x11 to have the same responsiveness as a direct ssh connection? Many thanks Simon.

6 8

Transfer from GPFS via slurm
by Ratnasamy, Fritz 06 Jun '25

06 Jun '25

Hi, We were told by our hardware provider that large datasets copied from NFS location to GPFS could be conducted via slurm to monitor the transfer. I am not sure of this works as I could not find much online. Otherwise apart from globus and rsync, what would you suggest as a tool/command to copy files from on a cluster between NFS and GPFS? Best, *Fritz Ratnasamy*Data Scientist Information Technology

2 1

Changing account names in sacctmgr
by Burian, John 05 Jun '25

05 Jun '25

All, For reasons I won’t go into, we find we need to change the POSIX group names for all of our users. My understanding is that in the absence of an --account option, sbatch/salloc assumes the account is the user’s primary POSIX group. So, to change group names, we’ll need to change the account names on all the associations in sacctmgr. And ideally, we would change account names in accounting records so we don’t lose accounting history. One is tempted to get into the Slurm DB and start running SQL updates. Is that path too fraught with peril to consider? Or should I just recreate the associations with the new names and take my lumps on the accounting history? John

4 4

error building debug packages on ubuntu 24.04
by Charles Hedrick 03 Jun '25

03 Jun '25

It was unable to find lib/security/pam_slurm_adopt.so It was built as lib64/security, not lib/security. ./debian/slurm-smd-libpam-slurm-adopt.install needs to be lib64/security/pam_slurm_adopt.so More likely, it needs to be built dynamically, so it will work everywhere.

1 0

Slurm version 25.05 is now available
by Tim Wickberg 29 May '25

29 May '25

We are pleased to announce the availability of Slurm 25.05. The release notes summarizing the new features, and including links to the corresponding documentation, can be found at: https://slurm.schedmd.com/release_notes.html A more extensive list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md The Slurm documentation has also been updated to the 25.05 release: https://slurm.schedmd.com Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support

1 0

Slurm accounts managed via ansible
by Ratnasamy, Fritz 28 May '25

28 May '25

Hi, I was wondering whether there might be built in support for managing slurm accounts,users,associations in ansible. it would be nice to be able to organize accounts in a yaml style file and modify accounts settings via gitlab CI/CD. For example, in a format as such: accounts: - name: "Boss1" parent: "root" description: "Big Boos" fairshare: 20 - name: "UnderBoss1" parent: "root" description: "working for Boss1" fairshare: 1 users: -username: "user1" -username: "user2" *Fritz Ratnasamy* Data Scientist Information Technology

3 2

Account getting duplicated
by Ratnasamy, Fritz 27 May '25

27 May '25

Hi, the slurm db duplicates all our account associations. one with cluster=cluster and another where cluster=venus (which is our actual cluster). is that intended? Or should I make any changes? *Fritz Ratnasamy*Data Scientist Information Technology

2 1

Slurm Reporting Difference between sreport and sacct
by Passant Hafez 23 May '25

23 May '25

Hi all, I was wondering if someone can help explaining this discrepancy. I have different values for project gpu consumption using sreport vs sacct (+ some calculations) This is an example that shows this: sreport -t hours -T gres/gpu cluster AccountUtilizationByuser start=2025-04-01 end=2025-04-05 | grep project1234 gives 178 while sacct -n -X --allusers --accounts=project1234 --start=2025-04-01 --end=2025-04-05 -o elapsedraw,AllocTRES%80,user,partition gives 213480 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus 249507 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus 13908 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus 9552 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus 4 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu 11 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu ... I will not bore you with the full output and its calculation, but the first job alone consumed 213480 seconds/60/60 * 8 gpus that's 474.4 gpu hours which is way more than the 178 hrs reported by sreport Any clue why these are inconsistent? or how sreport calculated the 178 value? All the best, Passant

3 3

Crash in "slurmd -C" when latest NVIDIA drivers are used
by Taras Shapovalov 22 May '25

22 May '25

Hello, Does someone have idea why "slurmd -C" crashes when it unloads gpu_nrt.so with latest NVIDIA drivers (570 and 575)? We checked, there is no crash in cuda at the moment and gpu_nvml.so works fine, all nvml calls finish successfully, dlclose on gpu_nvml.so works fine. The crash does not depend whether real GPUs present or not. Steps to reproduce: 1. Install Ubuntu 24.04 2. wget https://download.schedmd.com/slurm/slurm-24.11.4.tar.bz2<https://download.schedmd.com/slurm/slurm-24.11.4.tar.bz2;> 3. tar fx ./slurm-24.11.4.tar.bz2 4. cd slurm-24.11.4 5. apt-get install cuda-12-8 hwloc libmunge-dev -y 6. ./configure 7. make && make install 8. Run "slurmd -C", or sometimes "slurmd -vvv -C" to get the crash. Stack trace: #0 0x0000155555544b2a strlen (ld-linux-x86-64.so.2 + 0x28b2a) #1 0x000015555551fc08 __GI__dl_exception_create (ld-linux-x86-64.so.2 + 0x3c08) #2 0x000015555551d298 __GI__dl_signal_error (ld-linux-x86-64.so.2 + 0x1298) #3 0x000015555551e81d _dl_close (ld-linux-x86-64.so.2 + 0x281d) #4 0x000015555551d51c __GI__dl_catch_exception (ld-linux-x86-64.so.2 + 0x151c) #5 0x000015555551d669 _dl_catch_error (ld-linux-x86-64.so.2 + 0x1669) #6 0x0000155554e97c73 _dlerror_run (libc.so.6 + 0x97c73) #7 0x0000155554e979a6 __dlclose (libc.so.6 + 0x979a6) #8 0x0000155555388a25 gpu_plugin_fini (libslurmfull.so + 0x188a25) #9 0x000015555538f2ef gres_get_autodetected_gpus (libslurmfull.so + 0x18f2ef) #10 0x0000555555564828 _print_config (slurmd + 0x10828) #11 0x0000155554e2a1ca __libc_start_call_main (libc.so.6 + 0x2a1ca) #12 0x0000155554e2a28b __libc_start_main_impl (libc.so.6 + 0x2a28b) #13 0x000055555555fc75 _start (slurmd + 0xbc75) I don't really think the problem is in gpu_nrt itself, seems the problem is in memory corruption somewhere else, but I am not sure. The issue is reproduced constantly. A Best regards, Taras

2 3

Slurm 24.11.x + pyxis + TaskProlog -> fail
by Stefan Staeglich 21 May '25

21 May '25

Hi, we are currently evaluating Slurm 24.11. We are using pyxis in our cluster and have encountered the following problem in connection with TaskProlog: https://github.com/NVIDIA/pyxis/issues/163 Do you know more about the problem and the cause? Best, Stefan

1 0

sacctmgr: error: Sending PersistInit msg: Connection refused
by Ratnasamy, Fritz 18 May '25

18 May '25

Hi, We are working on a test cluster with slurm 24.11.3 and I am getting this error message from the login or compute nodes (note that the error does not show when run from the controller node): sacctmgr list associations tree format=cluster,account,user,maxnodes sacctmgr: error: _open_persist_conn: failed to open persistent connection to host:localhost:6819: Connection refused sacctmgr: error: Sending PersistInit msg: Connection refused Does someone know how fix that issue? Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

2 1

New version of slurm
by Ratnasamy, Fritz 16 May '25

16 May '25

Hi, I am trying to install the new version of slurm. Do you know if there is a way to find out what support is compiled into the executables? For example, apache has httpd -L which shows all the loaded modules. See below result: [image: image.png] *Fritz Ratnasamy*Data Scientist Information Technology

3 2

slurmdbd - error: We can't find assoc 1 on cluster slurm
by Edmundo Carmona Antoranz 14 May '25

14 May '25

Hello! We are hitting this problem when we are making a first run of slurmdbd: [2025-05-13T13:29:08.916] debug4: accounting_storage/as_mysql: _get_parent_id: 10(as_mysql_assoc.c:699) query select id_assoc, lineage from "slurm_assoc_table" where user='' and deleted!=1 and acct='root'; [2025-05-13T13:29:08.916] debug5: assoc_mgr_fill_in_assoc: looking for assoc of user=(null)(4294967294), acct=(null), cluster=slurm, partition=(null) [2025-05-13T13:29:08.916] debug2: _find_assoc_rec_id: no associations added yet [2025-05-13T13:29:08.916] error: We can't find assoc 1 on cluster slurm I checked and up to that moment the table slurm_assoc_table has not getting a record inserted after the table has been created. [root@ea-r9-29504-pid1 ~]# grep -C1 slurm_assoc_table /var/log/slurmdbd [2025-05-13T13:29:06.666] debug2: DBD_ADD_CLUSTERS: called in CONN 10 [2025-05-13T13:29:06.719] debug4: Table "slurm_assoc_table" doesn't exist, adding [2025-05-13T13:29:06.720] debug2: query alter table "slurm_assoc_table" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default '', modify `acct` tinytext not null, modify `partition` tinytext not null default '', modify `parent_acct` tinytext not null default '', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default '', modify `max_tres_pn` text not null default '', modify `max_tres_mins_pj` text not null default '', modify `max_tres_run_mins` text not null default '', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default '', modify `grp_tres_mins` text not null default '', modify `grp_tres_run_mins` text not null default '', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default '', modify `delta_qos` blob not null default '', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42)); [2025-05-13T13:29:06.729] debug3: query insert into table_defs_table (creation_time, mod_time, table_name, definition) values (1747135746, 1747135746, '"slurm_assoc_table"', 'alter table \"slurm_assoc_table\" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default \'\', modify `acct` tinytext not null, modify `partition` tinytext not null default \'\', modify `parent_acct` tinytext not null default \'\', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default \'\', modify `max_tres_pn` text not null default \'\', modify `max_tres_mins_pj` text not null default \'\', modify `max_tres_run_mins` text not null default \'\', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default \'\', modify `grp_tres_mins` text not null default \'\', modify `grp_tres_run_mins` text not null default \'\', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default \'\', modify `delta_qos` blob not null default \'\', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42));') on duplicate key update definition='alter table \"slurm_assoc_table\" modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0 not null, modify `comment` text, modify `flags` int unsigned default 0 not null, modify `is_def` tinyint default 0 not null, modify `id_assoc` int unsigned not null auto_increment, modify `user` tinytext not null default \'\', modify `acct` tinytext not null, modify `partition` tinytext not null default \'\', modify `parent_acct` tinytext not null default \'\', modify `id_parent` int unsigned not null, modify `lineage` text, modify `lft` int not null default 0, modify `rgt` int not null default 0, modify `shares` int default 1 not null, modify `max_jobs` int default NULL, modify `max_jobs_accrue` int default NULL, modify `min_prio_thresh` int default NULL, modify `max_submit_jobs` int default NULL, modify `max_tres_pj` text not null default \'\', modify `max_tres_pn` text not null default \'\', modify `max_tres_mins_pj` text not null default \'\', modify `max_tres_run_mins` text not null default \'\', modify `max_wall_pj` int default NULL, modify `grp_jobs` int default NULL, modify `grp_jobs_accrue` int default NULL, modify `grp_submit_jobs` int default NULL, modify `grp_tres` text not null default \'\', modify `grp_tres_mins` text not null default \'\', modify `grp_tres_run_mins` text not null default \'\', modify `grp_wall` int default NULL, modify `priority` int unsigned default NULL, modify `def_qos_id` int default NULL, modify `qos` blob not null default \'\', modify `delta_qos` blob not null default \'\', drop primary key, add primary key (id_assoc), drop index udex, add unique index udex (user(42), acct(42), `partition`(42)), drop key lft, add key lft (lft), drop key account, add key account (acct(42));', mod_time=1747135746; [2025-05-13T13:29:06.775] debug4: Table "slurm_assoc_usage_day_table" doesn't exist, adding -- [2025-05-13T13:29:08.916] debug4: accounting_storage/as_mysql: _get_parent_id: 10(as_mysql_assoc.c:699) query select id_assoc, lineage from "slurm_assoc_table" where user='' and deleted!=1 and acct='root'; [2025-05-13T13:29:08.916] debug5: assoc_mgr_fill_in_assoc: looking for assoc of user=(null)(4294967294), acct=(null), cluster=slurm, partition=(null) -- [2025-05-13T13:29:08.921] debug4: accounting_storage/as_mysql: _set_assoc_lft_rgt: 10(as_mysql_assoc.c:737) query select lft, rgt from "slurm_assoc_table" where id_assoc=2; [2025-05-13T13:29:08.922] debug4: accounting_storage/as_mysql: _make_sure_user_has_default_internal: 10(as_mysql_assoc.c:401) query select distinct is_def, acct from "slurm_assoc_table" where user='root' and deleted!=1 ORDER BY is_def desc, creation_time desc LIMIT 1; [2025-05-13T13:29:08.924] debug4: accounting_storage/as_mysql: as_mysql_add_assocs: 10(as_mysql_assoc.c:3999) query Even though the message shows up in the logs as an _error_, the service is able to come up fine, by the way, but is there an easy way to avoid the error message in the first place? BR!

1 0

Slurm release candidate version 25.05.0rc1 is available for testing
by Marshall Garey 13 May '25

13 May '25

We are pleased to announce the availability of Slurm release candidate 25.05.0rc1. To highlight some new features coming in 25.05: - Support for defining multiple topology configurations, and varying them by partition. - Support for tracking and allocating hierarchical resources. - Dynamic nodes can be dynamically added to the topology. - topology/block - Allow for gaps in the block layout. - Support for encrypting all network communication with TLS. - jobcomp/kafka - Optionally send job info at job start as well as job end. - Support an OR operator in --license requests. - switch/hpe_slingshot - Support for > 252 ranks per node. - switch/hpe_slingshot - Support mTLS authentication to the fabric manager. - sacctmgr - Add support for dumping and loading QOSes. - srun - Add new --wait-for-children option to keep the step running until all launched processes have been launched (cgroup/v2 only). - slurmrestd - Add new endpoint for creating reservations. This is the first release candidate of the upcoming 25.05 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://bugs.schedmd.com against the 25.05.x version and we will address them before the first production 25.05.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://www.schedmd.com/download-slurm/. The changelog for 25.05.0rc1 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-25.05.md#chang… -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Performance Issues after Update to 24.11.5
by Tilman Hoffbauer 13 May '25

13 May '25

Hello, we are running a SLURM-managed cluster with one control node (g-vm03) and 26 worker nodes (ouga[03-28]) on Rocky 8. We recently updated from 20.11.9 through 23.02.8 to 24.11.0 and then 24.11.5. Since then, we are experiencing performance issues - squeue and scontrol ping are slow to react and sometimes deliver "timeout on send/recv" messages, even with only very few parallel requests. We did not experience these issues with SLURM 20.11.9 before, we did not check the intermediate version 23.02.8 in detail before. In the log of slurmctld, we can also find messages like slurmctld: error: slurm_send_node_msg: [socket:[1272743]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error We thus implemented all recommendations from the high throughput documentation, and did achieve improvements with it (most notably by increasing the maximum number of open files and increasing MessageTimeout and TCPTimeout). For debugging, I attached the slurm.conf, the sdiag output (the server thread count is almost always 1 and sometimes increases to 2), the slurmctld log and the slurmdbd log from a time of high load. We would be very thankful for any input on how restore the old performance. Kind Regards, Tilman Hoffbauer

2 4

Do I have to hold back RAM for worker nodes?
by Xaver Stiensmeier 13 May '25

13 May '25

Dear Slurm-User List, currently, in our slurm.conf, we are setting: SelectType: select/cons_tres SelectTypeParameters: CR_Core and in our node configuration /RealMemory /was basically reduced by an amount to make sure the node always had enough RAM to run the OS. However, this is apparently now how it is supposed to be done: Lowering RealMemory with the goal of setting aside some amount for the OS and not available for job allocations will not work as intended if Memory is not set as a consumable resource in *SelectTypeParameters*. So one of the *_Memory options need to be enabled for that goal to be accomplished. (https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory) This leads to four questions regarding holding back RAM for worker nodes. Answers/help with any of those questions would be appreciated. *1.* Is reserving enough RAM for the worker node's OS and slurmd actually a thing you have to manage? *2.* If so how can we reserve enough RAM for the worker node's OS and slurmd when using CR_Core? *3.* Is that maybe a strong argument against using CR_Core that we overlooked? And semi-related: https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory talks about taking a value in megabytes. *4.* Is RealMemory really expecting megabytes or is it mebibytes? Best regards, Xaver

6 8

slurm failing to start as it doesnt know its node name??????
by Steven Jones 12 May '25

12 May '25

Hi, I am not sure what i have missed but I am getting this error on a compute node. ====== root@vm01no16 ~]# systemctl status slurmd.service × slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Sun 2025-05-11 23:01:29 UTC; 15min ago Process: 1178 ExecStart=/usr/sbin/slurmd --systemd $SLURMD_OPTIONS (code=exited, status=1/FAILURE) Main PID: 1178 (code=exited, status=1/FAILURE) CPU: 3ms May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: Starting Slurm node daemon... May 11 23:01:29 vm01no16.ods.vuw.ac.nz slurmd[1178]: slurmd: fatal: Unable to determine this slurmd's NodeName May 11 23:01:29 vm01no16.ods.vuw.ac.nz slurmd[1178]: fatal: Unable to determine this slurmd's NodeName May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: slurmd.service: Main process exited, code=exited, status=1/FAIL> May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: slurmd.service: Failed with result 'exit-code'. May 11 23:01:29 vm01no16.ods.vuw.ac.nz systemd[1]: Failed to start Slurm node daemon. [root@vm01no16 ~]# uname -a Linux vm01no16.ods.vuw.ac.nz 5.14.0-503.40.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 30 17:38:54 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux [root@vm01no16 ~]# vi /etc/hosts [root@vm01no16 ~]# cat /etc/hostname vm01no16.ods.vuw.ac.nz [root@vm01no16 ~]# tail /etc/hosts # Entry for int01no9.ods.vuw.ac.nz 130.195.87.29 int01no9.ods.vuw.ac.nz int01no9.ods.vuw.ac.nz-default # Entry for vm01no13.ods.vuw.ac.nz 130.195.87.41 vm01no13.ods.vuw.ac.nz vm01no13.ods.vuw.ac.nz-default # Entry for vm01no14.ods.vuw.ac.nz 130.195.87.42 vm01no14.ods.vuw.ac.nz vm01no14.ods.vuw.ac.nz-default # Entry for vm01no15.ods.vuw.ac.nz 130.195.87.43 vm01no15.ods.vuw.ac.nz vm01no15.ods.vuw.ac.nz-default # Entry for vm01no16.ods.vuw.ac.nz 130.195.87.44 vm01no16.ods.vuw.ac.nz vm01no16.ods.vuw.ac.nz-default [root@vm01no16 ~]# ====== regards Steven Jones B.Eng (Hons) Technical Specialist - Linux RHCE Victoria University, Digital Solutions, Level 8 Rankin Brown Building, Wellington, NZ 6012 0064 4 463 6272

2 1

Slurm versions 24.11.5, 24.05.8, and 23.11.11 are now available (CVE-2025-43904)
by Tim Wickberg 07 May '25

07 May '25

Slurm versions 24.11.5, 24.05.8, and 23.11.11 are now available and include a fix for a recently discovered security issue. SchedMD customers were informed on April 23rd and provided a patch on request; this process is documented in our security policy. [1] A mistake with permission handling for Coordinators within Slurm's accounting system can allow a Coordinator to promote a user to Administrator. (CVE-2025-43904) Thank you to Sekou Diakite (HPE) for reporting this. Downloads are available at https://www.schedmd.com/downloads.php . Release notes follow below. - Tim [1] https://www.schedmd.com/security-policy/ -- Tim Wickberg Chief Technology Officer, SchedMD LLC Commercial Slurm Development and Support > * Changes in Slurm 24.11.5 > ========================== > -- Return error to scontrol reboot on bad nodelists. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.40 > endpoints. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.41 > endpoints. > -- slurmrestd - Report an error when QOS resolution fails for v0.0.42 > endpoints. > -- data_parser/v0.0.42 - Added +inline_enums flag which modifies the > output when generating OpenAPI specification. It causes enum arrays to not > be defined in their own schema with references ($ref) to them. Instead they > will be dumped inline. > -- Fix binding error with tres-bind map/mask on partial node allocations. > -- Fix stepmgr enabled steps being able to request features. > -- Reject step creation if requested feature is not available in job. > -- slurmd - Restrict listening for new incoming RPC requests further into > startup. > -- slurmd - Avoid auth/slurm related hangs of CLI commands during startup > and shutdown. > -- slurmctld - Restrict processing new incoming RPC requests further into > startup. Stop processing requests sooner during shutdown. > -- slurmcltd - Avoid auth/slurm related hangs of CLI commands during > startup and shutdown. > -- slurmctld: Avoid race condition during shutdown or reconfigure that > could result in a crash due delayed processing of a connection while > plugins are unloaded. > -- Fix small memleak when getting the job list from the database. > -- Fix incorrect printing of % escape characters when printing stdio > fields for jobs. > -- Fix padding parsing when printing stdio fields for jobs. > -- Fix printing %A array job id when expanding patterns. > -- Fix reservations causing jobs to be held for Bad Constraints > -- switch/hpe_slingshot - Prevent potential segfault on failed curl > request to the fabric manager. > -- Fix printing incorrect array job id when expanding stdio file names. > The %A will now be substituted by the correct value. > -- Fix printing incorrect array job id when expanding stdio file names. > The %A will now be substituted by the correct value. > -- switch/hpe_slingshot - Fix vni range not updating on slurmctld restart > or reconfigre. > -- Fix steps not being created when using certain combinations of -c and > -n inferior to the jobs requested resources, when using stepmgr and nodes > are configured with CPUs == Sockets*CoresPerSocket. > -- Permit configuring the number of retry attempts to destroy CXI service > via the new destroy_retries SwitchParameter. > -- Do not reset memory.high and memory.swap.max in slurmd startup or > reconfigure as we are never really touching this in slurmd. > -- Fix reconfigure failure of slurmd when it has been started manually and > the CoreSpecLimits have been removed from slurm.conf. > -- Set or reset CoreSpec limits when slurmd is reconfigured and it was > started with systemd. > -- switch/hpe-slingshot - Make sure the slurmctld can free step VNIs after > the controller restarts or reconfigures while the job is running. > -- Fix backup slurmctld failure on 2nd takeover. > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904. > * Changes in Slurm 24.05.8 > ========================== > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904. > * Changes in Slurm 23.11.11 > =========================== > -- Fixed a job requeuing issue that merged job entries into the same SLUID > when all nodes in a job failed simultaneously. > -- Add ABORT_ON_FATAL environment variable to capture a backtrace from any > fatal() message. > -- Testsuite - fix python test 130_2. > -- Fix security issue where a coordinator could add a user with elevated > privileges. CVE-2025-43904.

1 0

Lots of RPC calls and REQUEST_GETPW calls
by Guillette, Jeremy 07 May '25

07 May '25

Hello, I’m trying to figure out why we’ve been seeing an increase in network traffic in our AWS-based cluster, which uses Amazon’s parallel cluster tool. After an incident a couple weeks ago, I turned on debug2 logging on the slurmd processes, and I’m seeing huge numbers of `REQUEST_GETPW` and `REQUEST_GETGR` requests going to the slurmd processes. I briefly turned on debug2 logging for `slurmctld` as well, and I’m seeing lots of RPC calls, but not as many as the `REQUEST_GETPW` requests that I’ve seen on compute node slurmd processes. Here’s a sample from the slurmctld log: ``` [2025-04-28T15:11:05.436] debug2: _slurm_rpc_dump_partitions, size=1253 usec=20 [2025-04-28T15:11:05.450] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2971 [2025-04-28T15:11:05.451] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2971 [2025-04-28T15:11:05.451] debug2: _slurm_rpc_dump_partitions, size=1253 usec=16 [2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2788 [2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2788 [2025-04-28T15:11:05.461] debug2: _slurm_rpc_dump_partitions, size=1253 usec=12 [2025-04-28T15:11:05.517] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2916 [2025-04-28T15:11:05.518] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2916 [2025-04-28T15:11:05.518] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.628] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3405 [2025-04-28T15:11:05.629] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3405 [2025-04-28T15:11:05.629] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2189 [2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2189 [2025-04-28T15:11:05.740] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:05.845] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2209 [2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2209 [2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=4106 [2025-04-28T15:11:05.846] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:05.847] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=4106 [2025-04-28T15:11:05.847] debug2: _slurm_rpc_dump_partitions, size=1253 usec=11 [2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3400 [2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3400 [2025-04-28T15:11:05.938] debug2: _slurm_rpc_dump_partitions, size=1253 usec=18 [2025-04-28T15:11:06.903] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3449 [2025-04-28T15:11:06.904] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3449 [2025-04-28T15:11:06.904] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.175] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3722 [2025-04-28T15:11:07.176] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3722 [2025-04-28T15:11:07.177] debug2: _slurm_rpc_dump_partitions, size=1253 usec=254 [2025-04-28T15:11:07.205] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=4040 [2025-04-28T15:11:07.206] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=4040 [2025-04-28T15:11:07.206] debug2: _slurm_rpc_dump_partitions, size=1253 usec=17 [2025-04-28T15:11:07.237] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2990 [2025-04-28T15:11:07.238] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2990 [2025-04-28T15:11:07.239] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.284] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2920 [2025-04-28T15:11:07.285] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2920 [2025-04-28T15:11:07.285] debug2: _slurm_rpc_dump_partitions, size=1253 usec=15 [2025-04-28T15:11:07.370] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3236 [2025-04-28T15:11:07.371] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3236 [2025-04-28T15:11:07.371] debug2: _slurm_rpc_dump_partitions, size=1253 usec=17 [2025-04-28T15:11:08.463] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2848 [2025-04-28T15:11:08.464] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2848 [2025-04-28T15:11:08.464] debug2: _slurm_rpc_dump_partitions, size=1253 usec=14 [2025-04-28T15:11:08.691] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=2627 [2025-04-28T15:11:08.692] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=2627 [2025-04-28T15:11:08.692] debug2: _slurm_rpc_dump_partitions, size=1253 usec=18 [2025-04-28T15:11:08.873] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3729 [2025-04-28T15:11:08.874] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3729 [2025-04-28T15:11:08.875] debug2: _slurm_rpc_dump_partitions, size=1253 usec=196 [2025-04-28T15:11:08.881] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE from UID=3461 [2025-04-28T15:11:08.882] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=3461 [2025-04-28T15:11:08.882] debug2: _slurm_rpc_dump_partitions, size=1253 usec=10 ``` And from slurmd: ``` [2025-04-27T19:45:01.353] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.475] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.491] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.501] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.504] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.507] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.513] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:02.606] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:02.607] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:04.988] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.992] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.995] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:04.999] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.011] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.016] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.033] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.045] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.048] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.057] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.073] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.077] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.110] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.144] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETGR [2025-04-27T19:45:05.170] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.172] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.174] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.203] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.204] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.207] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.316] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.318] [59253.batch] debug: Handling REQUEST_GETPW [2025-04-27T19:45:05.321] [59253.batch] debug: Handling REQUEST_GETPW ``` This level of debugging makes the logs pretty huge, but if seeing a whole log file is helpful, I can make something available. Any ideas on next steps for figuring out what’s going on? It seems like something is asking for authentication a whole lot, but it’s not clear to me what or why. We do use munge for Slurm authentication, and SSSD to work with LDAP for user authentication. -Jeremy Guillette — Jeremy Guillette Software Engineer, FAS Academic Technology | Academic Technology Harvard University Information Technology P: (617) 998-1826 | W: huit.harvard.edu (he/him/his)

4 6

Large memory jobs stuck Pending. Should use --time parameter?
by Mike 07 May '25

07 May '25

Greetings, We are new to Slurm and we are trying to better understand why we’re seeing high-mem jobs stuck in Pending state indefinitely. Smaller (mem) jobs in the queue will continue to pass by the high mem jobs even when we bump priority on a pending high-mem job way up. We have been reading over the backfill scheduling page and what we think we're seeing is that we need to require that users specify a --time parameter on their jobs so that Backfill works properly. None of our users specify a --time param because we have never required it. Is that what we need to require in order to fix this situation? From the backfill page: "Backfill scheduling is difficult without reasonable time limit estimates for jobs, but some configuration parameters that can help" and it goes on to list some config params that we have not set (DefaultTime, MaxTime, OverTimeLimit). We also see language such as, “Since the expected start time of pending jobs depends upon the expected completion time of running jobs, reasonably accurate time limits are important for backfill scheduling to work well.” So we suspect that we can achieve proper backfill scheduling by requiring that all users supply a "--time" parameter via a job submit plugin. Would that be a fair statement? Thank you in advance! -Mike Schor

4 4

Trying to using PMIx via srun for openmpi + pmix 5.0.7, PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 (but works with --mpi=pmix_v3?!)
by Dr. Thomas Orgis 06 May '25

06 May '25

Dear Slurm community, I am confused by the behaviour of a freshly built openmpi-5.0.7 with slurm-24.11.4. I can run a simple hello-world program via mpirun, but with really slow startup (a single process needing 1.6 s, 384 processes on two 192-core nodes need around half a minute). I guess there is a deeper issue here to work out with openmpi itself and why it needs 1.3 seconds to start a single process, even, outside a Slurm environment. So I investigate if running via srun changes things. Open-MPI docs recommend using mpirun and that has been traditionally our safe bet. But srun direct launch is supposed to work, too. I did duly build slurm-24.11.4 with ./configure --prefix=/syssw/slurm/24.11.4 \ --sysconfdir=/syssw/etc/slurm \ --with-munge=/syssw/munge/0.5.16 \ --with-hwloc=/syssw/hwloc/2.11.2 \ --disable-static --with-json \ --with-pmix=/syssw/pmix/3.2.5:/syssw/pmix/5.0.7 \ LDFLAGS=-Wl,--disable-new-dtags providing the two versions of pmix on our system currently. Now I am perplexed to observe that $ srun -vv --mpi=pmix_v5 -N 1 -n 1 mpihello does _not_ work, but produces c$ srun -vv -n 1 -N 1 --mpi=pmix_v5 mpihello srun: defined options srun: -------------------- -------------------- srun: (null) : n[164-165] srun: jobid : 671133 srun: job-name : interactive srun: mpi : pmix_v5 srun: nodes : 1 srun: ntasks : 1 srun: verbose : 2 srun: -------------------- -------------------- srun: end of defined options srun: debug: propagating SLURM_PRIO_PROCESS=0 srun: debug: propagating UMASK=0022 srun: jobid 671133: nodes(2):`n[164-165]', cpu counts: 192(x2) srun: debug: requesting job 671133, user 99, nodes 1 including ((null)) srun: debug: cpus 1, tasks 1, name mpihello, relative 65534 srun: CpuBindType=none srun: debug: Entering slurm_step_launch srun: debug: mpi/pmix_v5: pmixp_abort_agent_start: (null) [0]: pmixp_agent.c:417: Abort agent port: 36505 srun: debug: mpi/pmix_v5: _pmix_abort_thread: (null) [0]: pmixp_agent.c:356: Start abort thread srun: debug: mpi/pmix_v5: mpi_p_client_prelaunch: (null) [0]: mpi_pmix.c:282: setup process mapping in srun srun: debug: Entering _msg_thr_create() srun: debug: initialized stdio listening socket, port 38681 srun: debug: Started IO server thread srun: debug: Entering _launch_tasks srun: launching StepId=671133.3 on host n165, 1 tasks: 0 srun: topology/tree: init: topology tree plugin loaded srun: debug: launch returned msg_rc=0 err=0 type=8001 srun: Node n165, 1 tasks started [n165:2287661] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and MPI will try to terminate your MPI job as well) [n165:2287661] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! srun: Received task exit notification for 1 task of StepId=671133.3 (status=0x0e00). srun: error: n165: task 0: Exited with exit code 14 srun: debug: task 0 done srun: debug: IO thread exiting srun: debug: mpi/pmix_v5: _conn_readable: (null) [0]: pmixp_agent.c:109: false, shutdown srun: debug: mpi/pmix_v5: _pmix_abort_thread: (null) [0]: pmixp_agent.c:363: Abort thread exit Observe the line [n165:2287661] PMIX ERROR: PMIX_ERR_FILE_OPEN_FAILURE in file gds_shmem2.c at line 1056 anyone got an idea what that means, by what it's caused? And, what really confuses me, is that the test program _does_ work if I switch from pmix_v5 to pmix_v3: $ srun -vv -n 1 -N 1 --mpi=pmix_v3 mpihello srun: defined options srun: -------------------- -------------------- srun: (null) : n[164-165] srun: jobid : 671133 srun: job-name : interactive srun: mpi : pmix_v3 srun: nodes : 1 srun: ntasks : 1 srun: verbose : 2 srun: -------------------- -------------------- srun: end of defined options srun: debug: propagating SLURM_PRIO_PROCESS=0 srun: debug: propagating UMASK=0022 srun: jobid 671133: nodes(2):`n[164-165]', cpu counts: 192(x2) srun: debug: requesting job 671133, user 99, nodes 1 including ((null)) srun: debug: cpus 1, tasks 1, name mpihello, relative 65534 srun: CpuBindType=none srun: debug: Entering slurm_step_launch srun: debug: mpi/pmix_v3: pmixp_abort_agent_start: (null) [0]: pmixp_agent.c:417: Abort agent port: 43737 srun: debug: mpi/pmix_v3: _pmix_abort_thread: (null) [0]: pmixp_agent.c:356: Start abort thread srun: debug: mpi/pmix_v3: mpi_p_client_prelaunch: (null) [0]: mpi_pmix.c:282: setup process mapping in srun srun: debug: Entering _msg_thr_create() srun: debug: initialized stdio listening socket, port 43805 srun: debug: Started IO server thread srun: debug: Entering _launch_tasks srun: launching StepId=671133.4 on host n164, 1 tasks: 0 srun: topology/tree: init: topology tree plugin loaded srun: debug: launch returned msg_rc=0 err=0 type=8001 srun: Node n164, 1 tasks started hello world from processor n164, rank 0 out of 1 srun: Received task exit notification for 1 task of StepId=671133.4 (status=0x0000). srun: n164: task 0: Completed srun: debug: task 0 done srun: debug: IO thread exiting srun: debug: mpi/pmix_v3: _conn_readable: (null) [0]: pmixp_agent.c:109: false, shutdown srun: debug: mpi/pmix_v3: _pmix_abort_thread: (null) [0]: pmixp_agent.c:363: Abort thread exit How can a PMIx 5 MPI even work with the pmix_v3 plugin? Why does it _not_ work with the pmix_v5 plugin? I am also curious why the plugins don't link to the respective libpmix (are they using dlopen for their dependencies? Why?). $ ldd /syssw/slurm/24.11.4/lib/slurm/mpi_pmix*.so /syssw/slurm/24.11.4/lib/slurm/mpi_pmix.so: linux-vdso.so.1 (0x00007ffd19ffb000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x000014d199e56000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014d199c70000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014d199b90000) /lib64/ld-linux-x86-64.so.2 (0x000014d199edd000) /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v3.so: linux-vdso.so.1 (0x00007ffd265f2000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x00001553902c8000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00001553900e2000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000155390002000) /lib64/ld-linux-x86-64.so.2 (0x0000155390350000) /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v5.so: linux-vdso.so.1 (0x00007ffd862b7000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x0000145adc36d000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000145adc187000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000145adc0a7000) /lib64/ld-linux-x86-64.so.2 (0x0000145adc3f4000) But they do have the proper RPATH set up: $ readelf -d /syssw/slurm/24.11.4/lib/slurm/mpi_pmix*.so | grep -e ^File -e PATH File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/5.0.7/lib] File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v3.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/3.2.5/lib] File: /syssw/slurm/24.11.4/lib/slurm/mpi_pmix_v5.so 0x000000000000000f (RPATH) Library rpath: [/syssw/hwloc/2.11.2/lib:/syssw/pmix/5.0.7/lib] Which is important, since libpmix doesn't get sensible SONAME versioning (supposing they are supposed to be separate ABIs): $ find /syssw/pmix/* -name 'libpmix.so*' /syssw/pmix/3.2.5/lib/libpmix.so /syssw/pmix/3.2.5/lib/libpmix.so.2.2.35 /syssw/pmix/3.2.5/lib/libpmix.so.2 /syssw/pmix/5.0.7/lib/libpmix.so /syssw/pmix/5.0.7/lib/libpmix.so.2.13.7 /syssw/pmix/5.0.7/lib/libpmix.so.2 It's all libpmix.so.2. My mpihello program uses the 5.0.7 one, at least: $ ldd mpihello linux-vdso.so.1 (0x00007fff1f13f000) libmpi.so.40 => /sw/env/gcc-13.3.0/openmpi/5.0.7/lib/libmpi.so.40 (0x000014ed89d95000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014ed89baf000) libopen-pal.so.80 => /sw/env/gcc-13.3.0/openmpi/5.0.7/lib/libopen-pal.so.80 (0x000014ed89a25000) libfabric.so.1 => /syssw/fabric/1.21.0/lib/libfabric.so.1 (0x000014ed8989b000) libefa.so.1 => /lib/x86_64-linux-gnu/libefa.so.1 (0x000014ed8988d000) libibverbs.so.1 => /lib/x86_64-linux-gnu/libibverbs.so.1 (0x000014ed8986c000) libpsm2.so.2 => /syssw/psm2/12.0.1/lib/libpsm2.so.2 (0x000014ed89804000) libatomic.so.1 => /sw/compiler/gcc-13.3.0/lib64/libatomic.so.1 (0x000014ed897fb000) libnl-route-3.so.200 => /lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x000014ed8976a000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x000014ed89745000) libpmix.so.2 => /syssw/pmix/5.0.7/lib/libpmix.so.2 (0x000014ed8951e000) libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x000014ed894e8000) libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x000014ed894e3000) libhwloc.so.15 => /syssw/hwloc/2.11.2/lib/libhwloc.so.15 (0x000014ed89486000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014ed893a6000) /lib64/ld-linux-x86-64.so.2 (0x000014ed8a0d8000) libnuma.so.1 => /lib/x86_64-linux-gnu/libnuma.so.1 (0x000014ed89397000) Can someone shed light on how the differing PMIx plugins are supposed to work? Can someone share a setup where pmix_v5 does work with openmpi 5.x? Alrighty then, Thomas -- Dr. Thomas Orgis HPC @ Universität Hamburg

1 0

knl_generic.conf seems to be ignored and Slurm adopts default settings which specify Intel motherboard and syscfg - need help to figure out why Slurm is not reading my Dell knl_generic.conf parameters
by Bryan Johnston 01 May '25

01 May '25

Hi folks I'm in the process of standing up some Stampede2 Dell C3620p KNL nodes and I seem to be hitting a blind spot. I previously "successfully" configured KNL's on an Intel board (S7200AP), with OpenHPC and Rocky8. I say "successfully" because it works but evidently my latest troubleshooting has revealed that I may have been lucky rather than an expert KNL integrator :-) I thought I knew what I was doing, but after repeating my Intel KNL recipe with the Dell system, I have unearthed my ignorance with this wonderful (but deprecated) technology (anecdotally, the KNLs offer excellent performance and power efficiency for our workloads, particularly when contrasted with our alternative available hardware). The first discovery was the "syscfg" for Intel boards is not the same as the "syscfg" for Dell boards. I've since sorted this out. The second discovery was made while troubleshooting an issue that I'm hitting. After realising that the slurmd client nodes don't seem to be reading the "knl_generic.conf" parameters that are specified in /etc/slurm on the smshost (OpenHPC parlance for head node ... And it's a Slurm config less set up), I think my original Intel solution was working out of luck more than ingenuity. For reference , the Slurm configuration for KNL now includes: ``` NodeFeaturesPlugins=knl_generic DebugFlags=NodeFeatures GresTypes=hbm ``` And I've created a separate "knl_generic.conf" that points to the Dell specific tools and features. For the Dell system, slurmd seems to ignore my knl_generic.conf file and is drawing defaults from somewhere else. Slurm still considers SystemType to be Intel, SyscfgPath to be the default location, and SyscfgTimeout to be 1000. For Dell systems, Slurm needs to have SystemType=Dell and Timeout to be 1000. I don't understand why the nodes are not reading the knl_generic file - any help or clues would be appreciated. Here's my theory on what is happening: The Intel KNL system was successful by luck ... It probably exhibited the same ignore-the-config-file but ran default NodeFeatures for some generic knl_generic settings which are stored somewhere as default parameters. I must have just lucked out when I was using my Intel KNL system because it was using the defaults (that are compatible with Intel). If this assumption is correct, the Dell system is not working because it isn't compatible with the Intel defaults. Any clues on how to successfully invoke the config file (or better debuggingtechniwues to figure out why it isn't) would be appreciated. I can share journalctl feedback if necessary. For now, I've tried changing ownership of the config files to root:slurm, copied knl_generic.conf to the compute nodes' /etc/slurm/ and also tried to specify the config file by running (on the compute nodes) "slurmd" with "-f" ... No joy; if slurmd runs successfully (when I don't screw up some random experimental settings) then it always seems to ignore knl_generic.conf and loads some default settings from somewhere. A few questions: 1. Are there default settings stored somewhere? I might be barking up the wrong tree, although I've looked for files that may clash with the config file I've created but can't find any. 2. Is there a better way to force the knl_generic file to be loaded? 3. Is the configless Slurm somehow not reading the knl_generic file to the clients? I understand that all configuration files are read from the host server. Many thanks for any help! Regards / Groete / Sala(ni) Kahle - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Bryan Johnston Senior HPC Technologist II Lead: HPC Ecosystems Project HPCwire 2024's Outstanding Leader in HPC CHPC | www.chpc.ac.za | NICIS | nicis.ac.za Centre for High Performance Computing If you receive an email from me out of office hours for you, please do not feel obliged to respond during off-hours! Book time to meet with me<https://outlook.office.com/bookwithme/user/87af4906a703488386578f34e4473c74…>

1 1

Slurm and Kerberos/GSSAPI
by Burian, John 01 May '25

01 May '25

Does anyone have any experience with using Kerberos/GSSAPI and Slurm? I’m specifically wondering if there is a known mechanism for providing proper Kerberos credentials to Slurm batch jobs, such that those processes would be able to access a filesystem that requires Kerberos credentials. Some quick searching returned nothing useful. Interactive jobs have a similar problem, but I’m hoping that SSH credential forwarding can be leveraged there. I’m nothing like an expert in Kerberos, so forgive any apparent ignorance. Thanks, John

2 3

Can't specify multiple partitions when submitting GPU jobs
by milad＠earthspecies.org 29 Apr '25

29 Apr '25

Hi, I have defined a partition for each GPU type we have in the cluster. This was mainly because I have different Node types for each GPU type and I want to set `DefCpuPerGPU` `DefMemPerGPU` for each of them. Unfortunately one can't set them per node but can do that per partition. Now sometimes people don't care about the GPU type and would like any of the partitions to pick up the job. The `--partition` in `sbatch` does allow specifying multiple partitions and this works fine when I'm not specifying `--gpu`. However when I add do something like `sbatch -p A,B --gpus 1 script.sh` then I get "srun: job 6279 queued and waiting for resources" even though partition B does have a GPU to offer. Strangely if the first partition specified (i.e. A) had a free GPU it would allocate the GPU and run the job. Is this a bug? Perhaps related to this: https://groups.google.com/g/slurm-users/c/UOUVfkajUBQ

3 4