September 2025 - slurm-users - lists.schedmd.com

slurmctld hourly: Unexpected missing socket error
by Jason Ellul 12 Jan '26

12 Jan '26

Hi all, I am hoping someone can help with our problem. Every hour after restarting slurmctld the controller becomes unresponsive to commands for 1 sec, reporting errors such as: [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934767]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934760]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934875]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[934906]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error [2024-07-15T11:45:48.509] error: slurm_send_node_msg: [socket:[939016]] slurm_bufs_sendto(msg_type=RESPONSE_JOB_INFO) failed: Unexpected missing socket error It occurs consistently at around the hour mark, but generally not at other times, unless we run a reconfigure or restart the controller. We don’t see any issues in the slurmdbd.log and the errors are also always msg type RESPONSE. We have tried building a new server on different infrastructure, but the problem has persisted. Yesterday we even tried updating slurm to v24.05.1 in the hope that may provide a fix. During our troubleshooting we have: Set: * SchedulerParameters = max_rpc_cnt=400,sched_min_interval=50000,sched_max_job_start=300,batch_sched_delay=20,bf_resolution=600,bf_min_prio_reserve=2000,bf_min_age_reserve=600 * SlurmctldPort = 6808-6817 But although the stats in sdiag have improved we still see the errors. On our monitoring software we also see a drop in network and disk activity during this 1 second, always at approx. 1 hour after restarting the controller. Many Thanks in advance Jason Jason Ellul Head - Research Computing Facility Office of Cancer Research Peter MacCallum Cancer Centre

4 6

SRUN and SBATCH network issues on configless login node.
by Bruno Bruzzo 01 Oct '25

01 Oct '25

Hi all, first of all, sorry for my English, it's not my native language. We are currently experiencing an issue with srun and salloc on our login nodes, while sbatch works properly. slurm version 23.11.4. slurmctld runs on management node mmgt01. srun and salloc fail intermittently on login node, that means we can successfully use srun on login node from time to time, but it stops working for a while without us changing any configuration. login nodes reports the following to the user: $ srun -N 1 -n 64 --partition=gpunode --pty /bin/bash srun: job 4872 queued and waiting for resources srun: error: Security violation, slurm message from uid 202 srun: error: Security violation, slurm message from uid 202 srun: error: Security violation, slurm message from uid 202 srun: error: Task launch for StepId=4872.0 failed on node cn065: Invalid job credential srun: error: Application launch failed: Invalid job credential srun: Job step aborted uid 202 is the slurm user. On the server side, slurmctld logs show: sched: _slurm_rpc_allocate_resources JobId=4872 NodeList=(null) usec=228 sched: Allocate JobId=4872 NodeList=cn065 #CPUs=64 Partition=gpunode error: slurm_receive_msgs: [[snmgt01]:38727] failed: Zero Bytes were transmitted or received Killing interactive JobId=4872: Communication connection failure _job_complete: JobId=4872 WEXITSTATUS 1 _job_complete: JobId=4872 done step_partial_comp: JobId=4872 StepID=0 invalid; this step may have already completed _slurm_rpc_complete_job_allocation: JobId=4872 error Job/step already completing or completed We suspect it is a network issue regarding the Zero Bytes were transmitted or received. The configless system is working properly. Slurmd on login node can read changes made at slurm.conf after a scontrol reconfig. srun runs successfully from management nodes and from compute nodes. The issue is from the login node. scontrol ping always shows DOWN from login node, even when we can successfully run srun or salloc. $ scontrol ping Slurmctld(primary) at mmgt01 is DOWN We checked as well for munge consistency. mmgt and login nodes have the hostnames of their respective others on /etc/hosts. They can communicate. We would really appreciate some tips on what we could be missing. Best regards, Bruno Bruzzo System Administrator - Clementina XXI

4 7

Re: How to make TLS and PMIx v4 work together?
by Grigory Shamov 30 Sep '25

30 Sep '25

Forgot to add: the s2n-tls comes from EPEL and is ver 1.5.10. On 2025-09-25, 11:56 AM, "Grigory Shamov via slurm-users" <slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com>> wrote: Caution! This message was sent from outside the University of Manitoba. Hi All, We have updated SLURM to the current 25.05.x and tried to enable TLS on it. The OS is Alma 8.10, cgroups v1, and PMIx v 4. We see that srun fails for MPI jobs across the nodes, with TLS related errors when using PMIx (the default) but passes with srun --mpi=pmi2 or with mpirun . TLSType = tls/s2n TLSParameters = ca_cert_file= (has all the certs here under /etc/slurm/certs) And the errors when using PMIx are 025-09-25T11:04:43.894] error: con_close_on_poll_error: [n388:6818(fd:15)] socket error encountered while polling: Connection reset by peer [2025-09-25T11:04:50.102] [6451416.0] error: _negotiate: s2n_negotiate() failed S2N_ERR_CERT_UNTRUSTED[335544366]: Certificate is untrusted -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/tls/s2n_x509_validator.c:494 (couple of these) [2025-09-25T11:05:57.878] [6451416.0] error: tls_p_recv: s2n_recv() failed S2N_ERR_CLOSED[134217728]: connection is closed -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:37 [2025-09-25T11:05:57.883] [6451416.0] error: tls_p_send: s2n_send() failed S2N_ERR_IO[67108864]: underlying I/O operation failed, check system errno -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:28 (couple of these) [2025-09-25T11:05:59.076] error: wrap_on_data: [unix:/var/spool/slurmd/slurmd.socket(fd:17)] on_data returned rc: Unable to proxy slurmstepd message [2025-09-25T11:05:59.076] [6451416.0] error: _stepd_send_recv_msg: slurmd was unable to proxy request message to its final destination [2025-09-25T11:05:59.878] error: _slurmd_send_recv_msg: Failed to send/recv slurmstepd message MESSAGE_TASK_EXIT using proxy_type PROXY_TO_NODE_SEND_RECV 2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: pmixp_p2p_send: n388 [0]: pmixp_utils.c:469: send failed, rc=1001, exceeded the retry limit [2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: _slurm_send: n388 [0]: pmixp_server.c:1586: Cannot send message to /var/spool/slurmd/stepd.slurm.pmix.6451416.0, size = 27679, hostlist: (null) (and couple more PMIx errors). Looks like PMIx cannot talk to their peers now ? There was no specific configuration for the certgen plugin, because SLURM documentation seems to say it is optional(?). I wonder what do we miss here to have SLURM 25.05 in with TLS enabled and PMIx working? Any advice appreciated! Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com>

3 3

FUTURE nodes do not return to idle on slurmctld restart
by Steve Kirk 26 Sep '25

26 Sep '25

Hello, We've got a few nodes defined in our slurm.conf in 'FUTURE' state as it's a new hardware type we're working on brining into service. The nodes are currently all allocated to a dedicated partition. The partition is configured as 'state=UP'. As we've built the new nodes and started slurmd+munge, they've appeared in an idle state in the new partition as expected. All good so far. However if the slurmctld is restarted the nodes go back to being in 'FUTURE' state, and do not transition to idle, accept jobs etc. The slurm daemon on the new nodes can clearly still talk to the slurmctld, s* commands on the new nodes work as expected but remain in FUTURE state - until slurmd on each node is restarted. I could have misunderstood something about the FUTURE state but I was expecting them to go back to idle; I understand that slurmctld doesn't communicate out to nodes in FUTURE state but I at least expected them to be picked up when they communicate _in_ to the slurmctld. Is this expected behaviour or perhaps a bug? The reason I've defined the new nodes this way so I don't have to update slurm.conf and restart slurmctld as each is built, but can do that as a single job once everything is finished, however it seems less useful if they can 'disappear' from the cluster as far as users are concerned. Cheers, Steve

2 2

How to make TLS and PMIx v4 work together?
by Grigory Shamov 25 Sep '25

25 Sep '25

Hi All, We have updated SLURM to the current 25.05.x and tried to enable TLS on it. The OS is Alma 8.10, cgroups v1, and PMIx v 4. We see that srun fails for MPI jobs across the nodes, with TLS related errors when using PMIx (the default) but passes with srun --mpi=pmi2 or with mpirun . TLSType = tls/s2n TLSParameters = ca_cert_file= (has all the certs here under /etc/slurm/certs) And the errors when using PMIx are 025-09-25T11:04:43.894] error: con_close_on_poll_error: [n388:6818(fd:15)] socket error encountered while polling: Connection reset by peer [2025-09-25T11:04:50.102] [6451416.0] error: _negotiate: s2n_negotiate() failed S2N_ERR_CERT_UNTRUSTED[335544366]: Certificate is untrusted -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/tls/s2n_x509_validator.c:494 (couple of these) [2025-09-25T11:05:57.878] [6451416.0] error: tls_p_recv: s2n_recv() failed S2N_ERR_CLOSED[134217728]: connection is closed -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:37 [2025-09-25T11:05:57.883] [6451416.0] error: tls_p_send: s2n_send() failed S2N_ERR_IO[67108864]: underlying I/O operation failed, check system errno -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:28 (couple of these) [2025-09-25T11:05:59.076] error: wrap_on_data: [unix:/var/spool/slurmd/slurmd.socket(fd:17)] on_data returned rc: Unable to proxy slurmstepd message [2025-09-25T11:05:59.076] [6451416.0] error: _stepd_send_recv_msg: slurmd was unable to proxy request message to its final destination [2025-09-25T11:05:59.878] error: _slurmd_send_recv_msg: Failed to send/recv slurmstepd message MESSAGE_TASK_EXIT using proxy_type PROXY_TO_NODE_SEND_RECV 2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: pmixp_p2p_send: n388 [0]: pmixp_utils.c:469: send failed, rc=1001, exceeded the retry limit [2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: _slurm_send: n388 [0]: pmixp_server.c:1586: Cannot send message to /var/spool/slurmd/stepd.slurm.pmix.6451416.0, size = 27679, hostlist: (null) (and couple more PMIx errors). Looks like PMIx cannot talk to their peers now ? There was no specific configuration for the certgen plugin, because SLURM documentation seems to say it is optional(?). I wonder what do we miss here to have SLURM 25.05 in with TLS enabled and PMIx working? Any advice appreciated! Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

1 0

Node switching randomly to down state
by Julien Tailleur 24 Sep '25

24 Sep '25

Dear all, I am maintaining a small computing cluster and I have a weird behavior that I fail at debugging. My cluster comprise one master node and 16 computing servers, organized in two queues, each queue having 8 servers. All servers run up-to-date Debian bullseye. All but 3 servers work flawlessly. From the master node, I can see that 3 servers on one of the queue appear down: jtailleu@kandinsky:~$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 down* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 These servers are reachable by SSH/ping jtailleu@kandinsky:~$ ping -c 1 FX12 PING FX12 (192.168.6.22) 56(84) bytes of data. 64 bytes from FX12 (192.168.6.22): icmp_seq=1 ttl=64 time=0.070 ms --- FX12 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.070/0.070/0.070/0.000 ms ##### I can also put these nodes back into idle mode: root@kandinsky:~# scontrol update nodename=FX[12-14] state=idle root@kandinsky:~# sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 idle* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 But then, they switch back into down mode few minutes later: root@kandinsky:~# sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 down* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 root@kandinsky:~# sinfo -R REASON USER TIMESTAMP NODELIST Not responding slurm 2025-09-08T15:04:39 FX[12-14] I do not understand where the "not responding" comes from, nor how I can investigate that. Any idea what could trigger this behavior? Best wishes, Julien

4 4

No output and can't job by id
by Dhumal, Dr. Nilesh 22 Sep '25

22 Sep '25

Hello, Recently, we updated Slurm. We are running slrumcld on head node. The slurmdbd is disabled on the head node. The user submits the job and gets the job ID. The squeue is empty, and difficult to track the job progress. The job is not writing any output to a file. How can I resolve this issue? Nilesh Dhumal Associate Professor of Chemistry, http://faculty.fgcu.edu/ndhumal/ Coordinator, FGCU Computational Facility, https://www.fgcu.edu/cas/facultyresources/computationalfacility/ SH-430; Department of Chemistry and Physics Florida Gulf Coast University 10501 FGCU Boulevard South Fort Myers, FL 33965-6565 Phone: (239) 745-4394 Email: ndhumal(a)fgcu.edu

2 2

Node in drain state
by Gestió Servidors 22 Sep '25

22 Sep '25

Hello, I have got a node in "drain" state after finishing a job that was running on it. Log in node reports this information: [...] [2025-09-07T11:09:26.980] task/affinity: task_p_slurmd_batch_request: task_p_slurmd_batch_request: 59238 [2025-09-07T11:09:26.980] task/affinity: batch_bind: job 59238 CPU input mask for node: 0xFFF [2025-09-07T11:09:26.980] task/affinity: batch_bind: job 59238 CPU final HW mask for node: 0xFFF [2025-09-07T11:09:26.980] Launching batch job 59238 for UID 21310 [2025-09-07T11:09:27.006] cred/munge: init: Munge credential signature plugin loaded [2025-09-07T11:09:27.007] [59238.batch] debug: auth/munge: init: loaded [2025-09-07T11:09:27.009] [59238.batch] debug: Reading cgroup.conf file /soft/slurm-23.11.0/etc/cgroup.conf [2025-09-07T11:09:27.025] [59238.batch] debug: cgroup/v1: init: Cgroup v1 plugin loaded [2025-09-07T11:09:27.025] [59238.batch] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: core enforcement enabled [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: device enforcement enabled [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2025-09-07T11:09:27.026] [59238.batch] task/affinity: init: task affinity plugin loaded with CPU mask 0xfff [2025-09-07T11:09:27.027] [59238.batch] debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded [2025-09-07T11:09:27.027] [59238.batch] topology/default: init: topology Default plugin loaded [2025-09-07T11:09:27.030] [59238.batch] debug: gpu/generic: init: init: GPU Generic plugin loaded [2025-09-07T11:09:27.031] [59238.batch] debug: laying out the 12 tasks on 1 hosts clus09 dist 2 [2025-09-07T11:09:27.031] [59238.batch] debug: close_slurmd_conn: sending 0: No error [2025-09-07T11:09:27.031] [59238.batch] debug: Message thread started pid = 910040 [2025-09-07T11:09:27.031] [59238.batch] debug: Setting slurmstepd(910040) oom_score_adj to -1000 [2025-09-07T11:09:27.031] [59238.batch] debug: spank: opening plugin stack /soft/slurm-23.11.0/etc/plugstack.conf [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: job abstract cores are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: step abstract cores are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: job physical CPUs are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: step physical CPUs are '0-11' [2025-09-07T11:09:27.090] [59238.batch] debug levels are stderr='error', logfile='debug', syslog='fatal' [2025-09-07T11:09:27.090] [59238.batch] starting 1 tasks [2025-09-07T11:09:27.090] [59238.batch] task 0 (910044) started 2025-09-07T11:09:27 [2025-09-07T11:09:27.098] [59238.batch] debug: task/affinity: task_p_pre_launch: affinity StepId=59238.batch, task:0 bind:mask_cpu [2025-09-07T11:09:27.098] [59238.batch] _set_limit: RLIMIT_NPROC : reducing req:255366 to max:159631 [2025-09-07T11:09:27.398] [59238.batch] task 0 (910044) exited with exit code 2. [2025-09-07T11:09:27.399] [59238.batch] debug: task/affinity: task_p_post_term: affinity StepId=59238.batch, task 0 [2025-09-07T11:09:27.399] [59238.batch] debug: signaling condition [2025-09-07T11:09:27.399] [59238.batch] debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded [2025-09-07T11:09:27.400] [59238.batch] debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded [2025-09-07T11:09:27.400] [59238.batch] job 59238 completed with slurm_rc = 0, job_rc = 512 [2025-09-07T11:09:27.410] [59238.batch] debug: Message thread exited [2025-09-07T11:09:27.410] [59238.batch] stepd_cleanup: done with step (rc[0x200]:Unknown error 512, cleanup_rc[0x0]:No error) [2025-09-07T11:09:27.411] debug: _rpc_terminate_job: uid = 1000 JobId=59238 [2025-09-07T11:09:27.411] debug: credential for job 59238 revoked [...] "sinfo" shows: [root@login-node ~]# sinfo PARTITION TIMELIMIT AVAIL STATE NODELIST CPU_LOAD NODES(A/I) NODES(A/I/O/T) CPUS CPUS(A/I/O/T) REASON node.q* 4:00:00 up drained clus09 0.00 0/0 0/0/1/1 12 0/0/12/12 Kill task faile node.q* 4:00:00 up allocated clus[10-11] 13.82-15.8 2/0 2/0/0/2 12 24/0/0/24 none node.q* 4:00:00 up idle clus[01-06,12] 0.00 0/7 0/7/0/7 12 0/84/0/84 none But it seems there is no error in node... Slurmctld.log in server seems correct, too. Is there any way to reset node to "state=idle" after errors in the same way? Thanks.

3 4

Compute node not responding
by Dhumal, Dr. Nilesh 21 Sep '25

21 Sep '25

Hello, Recently, we installed slum 25 on our cluster. We are not monitoring the user's account. We didn't configure the sql database on the head node. We are running slurmcld on head node and slumd on the compute node. We are getting the following error Head node: compute node not responding. Compute node: 2025-09-19T15:30:23.461] error: Unable to register: Unable to contact slurm controller (connect failure) Do we need to run slumdbd on the head node? I checked the network connection by pinging the compute node from the head node. Do you have any suggestions to resolve this issue? Thanks Nilesh Get Outlook for Android<https://aka.ms/AAb9ysg>

4 5

Re: Node in drain state
by Gestió Servidors 19 Sep '25

19 Sep '25

Hi, After reading answer from Ole Holm Nielsen, I have increased "MessageTimeout" to 20s (by default is 5s) and "UnkillableStepTimeout" to 150s (by default is 60s and, always 5 times larger than "MessageTimeout"). However, I have also read that UnkillableStepProgram indicates the program to use in that cases... but, by default there is no program assigned to that parameter (no program to run). So my question is if someone uses a customized "UnkillableStepProgram" and if he/she could explain that. Thanks a lot!

3 2

seff for GPU
by Josu Lazkano Lete 18 Sep '25

18 Sep '25

Hello, We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff? It will be great to know how much GPU resources the users request and compare with how much GPU resources they use. Kind regards. [image: Vicomtech] <https://www.vicomtech.org> Josu Lazkano Lete Systems Manager Infrastructures and General Services jlazkano(a)vicomtech.org +(34) 943 30 92 30 The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it. Avoid printing this message if it is not strictly necessary.

7 9

Scheduling issues with multiple different types of GPU in one partition
by Kevin M. Hildebrand 17 Sep '25

17 Sep '25

We have several different types of GPUs in the same 'gpu' partition. The problem we're having occurs when one of those types of GPUs is fully occupied and there are a bunch of queued jobs waiting for those GPUs. If someone requests idle GPUs of a different type, those jobs end up getting stalled, even though there are plenty of GPUs available. For example, say we have 10 A100 GPUs and 10 H100 GPUs. If there are 10 H100 GPU jobs running and more in queue waiting for them, subsequently submitted A100 jobs will sit in queue even if there are plenty of idle A100 GPUs. The only way we can get the A100 jobs to run is by manually bumping their priority higher than the pending H100 jobs. Has anyone else encountered this issue? The only way we can think of to potentially solve it is to have separate partitions for each GPU type, but that seems unwieldy. We are currently running Slurm 24.05.8. Thanks, Kevin -- Kevin Hildebrand Director of Research Technology and HPC Services Division of IT

7 7

New "NOT-state" selection of the sinfo command in Slurm 25.05
by Ole Holm Nielsen 10 Sep '25

10 Sep '25

We just upgraded Slurm to 25.05.3, and I would like to highlight a new functionality of the "sinfo -t <state>" command in 25.05: > The state can be prefixed with '~' which will invert the result of match. We find this "NOT-state" selection useful together with Slurm power saving [1] where any idle nodes may be powered off to save electrical power or cloud costs. For example, the ClusterShell [2] clush command can now skip connections to the powered down nodes like in this example: $ clush -bw@slurmstate:~POWERED_DOWN uname -r You can even configure ClusterShell so that "clush -a" includes all nodes which are *not* powered down. The details are described in the Wiki page [3]. Best regards, Ole [1] https://slurm.schedmd.com/power_save.html [2] https://clustershell.readthedocs.io/en/latest/intro.html [3] https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_operations/#skipping-powere… -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

1 0

Development RPMs for cgroups v2
by John Hearns 10 Sep '25

10 Sep '25

I am building a version of Slurm on RHEL 9.4 When I run an rpmbuild, he slurm rpm does not contain /usr/lib64/slurm/cgroup_v2.so I have tried to look in the build logs. I suspect I am lacking some development RPMs - but which ones? All hints gratefully received John H

4 5

NHC skips nodes in "mix-" state
by blines＠tgen.org 08 Sep '25

08 Sep '25

The node-mark-offline script skips nodes in the "mix-" state with the error: State "mix-" not yet handled; ignoring. "mix-" seems to be a valid state indicating that a node is mixed and is "planned by the backfill scheduler for a higher priority job". Anyone know the reason that nodes in this state are being skipped? The responsible code appears to be: case "$STATUS" in *'@'*|*'#'*|boot*|*-*|plnd*) # These states aren't handled yet. echo "$0: State \"$STATUS\" not yet handled; ignoring." exit 0 As far as I can tell all the states being skipped are related to nodes being configured or rebooted or powering on/off except *-*. There are other states related to node power that include underscore in the state name; power_down, power_up, pow_dn, pow_up. Should the case statement be *_* instead of *-*?

1 0

Creating /run/user/$UID - for Podman runtime
by John Snowdon 08 Sep '25

08 Sep '25

We are in the middle of implementing an extensive range of container support on our new HPC platform and have decided to offer our users a wide suite of technologies to better support their workloads: * Apptainer * Podman (rootless) * Docker (rootless) We've already got a solution for automated entries in /etc/subuid and /etc/subgid on the head nodes (available here under GPL: https://github.com/megatron-uk/pam_subid), which is where we intend users to build their container images, and building and running containers using Apptainer and Podman in those environments works really well - we're happy that it should take care of 95% of our users needs (Docker is the last few percent....) and not involve giving them any special permissions. If I ssh directly to a compute node, then Podman also works there to run an existing image (podman container run ...). What I'm struggling with now is running Podman under Slurm itself on our compute nodes. It appears as though Podman (in rootless mode) wants to put the majority of its run time / state information under /run/user/$UID ... this is fine on the head nodes which have interactive logins hitting PAM modules which instantiate the /run/user/$UID directories, but not under sbatch/srun which doesn't create them by default. I've not been able to find a single, magical setting which will move all of the Podman state information out from /run/user to another location - there are 3 or 4 settings involved, and even then I still find various bits of Podman want to create stuff under there. Rather than hacking away at getting Podman changed to move all settings and state information elsewhere, it seems like the cleanest solution would just be to put the regular /run/user/$UID directory in place at the point Slurm starts the job instead. What's the best way to get Slurm to create this and clean-up afterwards? Should this be in a prolog/epilog wrapper (e.g. directly calling loginctl) or is it cleaner to get Slurm to trigger the usual PAM session machinery in some manner? John Snowdon Senior Research Infrastructure Engineer (HPC) Research Software Engineering Catalyst Building, Room 2.01 Newcastle University 3 Science Square Newcastle Helix Newcastle upon Tyne NE4 5TG https://hpc.researchcomputing.ncl.ac.uk

5 7

Slurm version 25.05.3 is now available
by Marshall Garey 04 Sep '25

04 Sep '25

We are pleased to announce the availability of Slurm version 25.05.3. This version fixes an issue that prevented deleting a QOS when running with MySQL servers (MariaDB is was unaffected). Please note that the slurmdbd will require MySQL 8.0.4+ or MariaDB 10.0.5+ to function correctly. This version also fixes heterogeneous jobs when TLS is enabled, a logging issue with syslog, and various mild to moderate stability fixes. The full list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Discussion about the --cpus-per-task flag
by Michele Esposito Marzino 04 Sep '25

04 Sep '25

Hi everyone, I would like to ask about the rationale behind the --cpus-per-task flag. While I do understand the meaning and its intended goal, I was wondering how it is managed in practice. I tried inferring this from the source code, but it is quite intricate, and not really straightforward by simply searching for the string match. Does it affect the CPUSET of each task? Who is responsible for setting this? The slurmcltd or slurmd? And how? Or at least, point me where in the code it is handled, so that I can reconstruct the whole process myself. Thanks

1 0

sacctmgr output with and without associations
by Rémy Dernat 04 Sep '25

04 Sep '25

Hi, I have a stupid issue about sacctmgr output and associations. I would like to retrieve my accounts with all the following data : "Account","Clusters","DefaultQOS","Description","Fairshare","Flags","GrpTRESMins","GrpTRES","GrpJobs","GrpSubmitJob",GrpWall", "MaxTRESMins","MaxTRES","MaxJobs","MaxNodes","MaxSubmitJobs","MaxWall","Organization","ParentName","Priority","QosLevel" So my command is the following : sacctmgr list accounts --assoc format="Account","Clusters","DefaultQOS","Description","Fairshare","Flags","GrpTRESMins","GrpTRES","GrpJobs","GrpSubmitJob","GrpWall","MaxTRESMins","MaxTRES","MaxJobs","MaxNodes","MaxSubmitJobs","MaxWall","Organization","ParentName","Priority","QosLevel" -n -P However, I have a lot of outputs corresponding to each User association entry. That would be fine if a basic `|uniq` would remove everything I don't need, but I am stuck with grptres, which can be seen in the main account, but not in the children associations with the Users. On the opposite, if I remove `--assoc`, I don't have anymore grptres, ParentName, or related Qos values. I had to add `where User=""` to my first command to just retrieve my main accounts, without User associations. I dont' know if that is clear enough; is there something I missed with the sacctmgr output ? Thanks, Best regards, Rémy

2 1

Can't remove account
by Matthias Leopold 01 Sep '25

01 Sep '25

Hi, I have a cluster with account foo. I want to delete this account. For every user that had associations in this account I created new associations in a different account and changed the DefaultAccount of the user. Then I deleted all associations in account foo. Now I want to delete account foo: sacctmgr delete account name=foo This gives: Error with request: You can not remove the default account of a user ... Please either remove the accounts listed above from list and resubmit, or change these users' default accounts to remove the account(s). Changes Discarded It names all associations for users that FORMERLY had DefaultAccount foo?? NO user has DefaultAccount foo anymore... I have to mention that I also have partition foo and affected users have an association in their new account with partition foo. But that shouldn't matter IMHO Has anybody seen this? What can I do? Am I getting something wrong? Slurm version is 24.05.7 thanks Matthias

1 0

2026

2025

2024

slurm-users September 2025