- slurm-users - lists.schedmd.com

Unexpected missing socket error
by Ozeryan, Vladimir 07 Oct '25

07 Oct '25

Hello everyone, Not sure if you guys have heard this tune already but did anyone come across a solution for "Unexpected missing socket error". There is nothing useful in the logs but the message appears on compute nodes and slurm controller node. Thank you, Vlad Ozeryan AMDS - AB1 Linux-Support Vladimir.Ozeryan(a)jhuapl.edu Ext. 23966

3 2

Setting GrpTRES for specific Account only for specific Partition(s)
by Paul Raines 06 Oct '25

06 Oct '25

I am trying to figure out how one can limit resources like with GrpTRES for specific accounts only for specific partitions in the cluster Lets say I have a cluster with 3 partitions Part1, Part2, and Part3 and 3 accounts Acct1, Acct2, and Acct3 I want Acct1 to be limited to 10 cpus on Part1, Acct2 to be limited to 5 cpus on Part1, and Acct3 to be limited to 8 cpus on Part1. No limits for any account on the other two partitions at present. I have tried # sacctmgr modify account Acct1 where partition=Part1 set GrpTRES=cpu=10 Modified account associations... Nothing modified I have tried creating a QOS to assign # sacctmgr add qos cpus10 # sacctmgr modify qos cpus10 set GrpTRES=cpu=10 # sacctmgr modify account Acct1 where partition=Part1 set qos=cpus10 Modified account associations... Nothing modified I cannot set a Partition QOS since that would apply to all accounts on the partition giving them the same limit I cannot just set a general Account limit (or QOS) since that would apply to all Partitions. I really don't want to have to have different Accounts for each partition so I can set Account wide limits. That would make users have to remember which Account to use with which Partiton. It looks like the following works but requires having to repeat for all users in every Account sacctmgr -i add user User1 account=Acct1 partition=Part1 \ fairshare=parent qos=cpus10 # limits on the Acct1 partition sacctmgr -i add user User1 account=Acct1 \ fairshare=parent # no limits on any other partition But in essense is not much better than having different Accounts for each partition as I now have to maintain multiple user entries per Account: one for partitios with no-limits and and additional one for each partition that needs limits. I guess at least users don't have to remember what account to use when submitting to a partition. Its seems there should be some simpler way to do this. A way to say "this is the limit for this Account on this Partion for all current users and any future users added to the Account" Am I missing something? The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

cloud compute limits/budget
by Tom Sparks 03 Oct '25

03 Oct '25

I am setting up a single user slurm I have limited budget for cloud gpu compute ($40) I am wondering if slurm has ability to stop me from going over my budget? I have read the slurm's powersaving / configureless nodes guides Are they other guides I should look at?

2 1

SRUN and SBATCH network issues on configless login node.
by Bruno Bruzzo 01 Oct '25

01 Oct '25

Hi all, first of all, sorry for my English, it's not my native language. We are currently experiencing an issue with srun and salloc on our login nodes, while sbatch works properly. slurm version 23.11.4. slurmctld runs on management node mmgt01. srun and salloc fail intermittently on login node, that means we can successfully use srun on login node from time to time, but it stops working for a while without us changing any configuration. login nodes reports the following to the user: $ srun -N 1 -n 64 --partition=gpunode --pty /bin/bash srun: job 4872 queued and waiting for resources srun: error: Security violation, slurm message from uid 202 srun: error: Security violation, slurm message from uid 202 srun: error: Security violation, slurm message from uid 202 srun: error: Task launch for StepId=4872.0 failed on node cn065: Invalid job credential srun: error: Application launch failed: Invalid job credential srun: Job step aborted uid 202 is the slurm user. On the server side, slurmctld logs show: sched: _slurm_rpc_allocate_resources JobId=4872 NodeList=(null) usec=228 sched: Allocate JobId=4872 NodeList=cn065 #CPUs=64 Partition=gpunode error: slurm_receive_msgs: [[snmgt01]:38727] failed: Zero Bytes were transmitted or received Killing interactive JobId=4872: Communication connection failure _job_complete: JobId=4872 WEXITSTATUS 1 _job_complete: JobId=4872 done step_partial_comp: JobId=4872 StepID=0 invalid; this step may have already completed _slurm_rpc_complete_job_allocation: JobId=4872 error Job/step already completing or completed We suspect it is a network issue regarding the Zero Bytes were transmitted or received. The configless system is working properly. Slurmd on login node can read changes made at slurm.conf after a scontrol reconfig. srun runs successfully from management nodes and from compute nodes. The issue is from the login node. scontrol ping always shows DOWN from login node, even when we can successfully run srun or salloc. $ scontrol ping Slurmctld(primary) at mmgt01 is DOWN We checked as well for munge consistency. mmgt and login nodes have the hostnames of their respective others on /etc/hosts. They can communicate. We would really appreciate some tips on what we could be missing. Best regards, Bruno Bruzzo System Administrator - Clementina XXI

4 7

Re: How to make TLS and PMIx v4 work together?
by Grigory Shamov 30 Sep '25

30 Sep '25

Forgot to add: the s2n-tls comes from EPEL and is ver 1.5.10. On 2025-09-25, 11:56 AM, "Grigory Shamov via slurm-users" <slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com>> wrote: Caution! This message was sent from outside the University of Manitoba. Hi All, We have updated SLURM to the current 25.05.x and tried to enable TLS on it. The OS is Alma 8.10, cgroups v1, and PMIx v 4. We see that srun fails for MPI jobs across the nodes, with TLS related errors when using PMIx (the default) but passes with srun --mpi=pmi2 or with mpirun . TLSType = tls/s2n TLSParameters = ca_cert_file= (has all the certs here under /etc/slurm/certs) And the errors when using PMIx are 025-09-25T11:04:43.894] error: con_close_on_poll_error: [n388:6818(fd:15)] socket error encountered while polling: Connection reset by peer [2025-09-25T11:04:50.102] [6451416.0] error: _negotiate: s2n_negotiate() failed S2N_ERR_CERT_UNTRUSTED[335544366]: Certificate is untrusted -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/tls/s2n_x509_validator.c:494 (couple of these) [2025-09-25T11:05:57.878] [6451416.0] error: tls_p_recv: s2n_recv() failed S2N_ERR_CLOSED[134217728]: connection is closed -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:37 [2025-09-25T11:05:57.883] [6451416.0] error: tls_p_send: s2n_send() failed S2N_ERR_IO[67108864]: underlying I/O operation failed, check system errno -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:28 (couple of these) [2025-09-25T11:05:59.076] error: wrap_on_data: [unix:/var/spool/slurmd/slurmd.socket(fd:17)] on_data returned rc: Unable to proxy slurmstepd message [2025-09-25T11:05:59.076] [6451416.0] error: _stepd_send_recv_msg: slurmd was unable to proxy request message to its final destination [2025-09-25T11:05:59.878] error: _slurmd_send_recv_msg: Failed to send/recv slurmstepd message MESSAGE_TASK_EXIT using proxy_type PROXY_TO_NODE_SEND_RECV 2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: pmixp_p2p_send: n388 [0]: pmixp_utils.c:469: send failed, rc=1001, exceeded the retry limit [2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: _slurm_send: n388 [0]: pmixp_server.c:1586: Cannot send message to /var/spool/slurmd/stepd.slurm.pmix.6451416.0, size = 27679, hostlist: (null) (and couple more PMIx errors). Looks like PMIx cannot talk to their peers now ? There was no specific configuration for the certgen plugin, because SLURM documentation seems to say it is optional(?). I wonder what do we miss here to have SLURM 25.05 in with TLS enabled and PMIx working? Any advice appreciated! Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com>

3 3

FUTURE nodes do not return to idle on slurmctld restart
by Steve Kirk 26 Sep '25

26 Sep '25

Hello, We've got a few nodes defined in our slurm.conf in 'FUTURE' state as it's a new hardware type we're working on brining into service. The nodes are currently all allocated to a dedicated partition. The partition is configured as 'state=UP'. As we've built the new nodes and started slurmd+munge, they've appeared in an idle state in the new partition as expected. All good so far. However if the slurmctld is restarted the nodes go back to being in 'FUTURE' state, and do not transition to idle, accept jobs etc. The slurm daemon on the new nodes can clearly still talk to the slurmctld, s* commands on the new nodes work as expected but remain in FUTURE state - until slurmd on each node is restarted. I could have misunderstood something about the FUTURE state but I was expecting them to go back to idle; I understand that slurmctld doesn't communicate out to nodes in FUTURE state but I at least expected them to be picked up when they communicate _in_ to the slurmctld. Is this expected behaviour or perhaps a bug? The reason I've defined the new nodes this way so I don't have to update slurm.conf and restart slurmctld as each is built, but can do that as a single job once everything is finished, however it seems less useful if they can 'disappear' from the cluster as far as users are concerned. Cheers, Steve

2 2

How to make TLS and PMIx v4 work together?
by Grigory Shamov 25 Sep '25

25 Sep '25

Hi All, We have updated SLURM to the current 25.05.x and tried to enable TLS on it. The OS is Alma 8.10, cgroups v1, and PMIx v 4. We see that srun fails for MPI jobs across the nodes, with TLS related errors when using PMIx (the default) but passes with srun --mpi=pmi2 or with mpirun . TLSType = tls/s2n TLSParameters = ca_cert_file= (has all the certs here under /etc/slurm/certs) And the errors when using PMIx are 025-09-25T11:04:43.894] error: con_close_on_poll_error: [n388:6818(fd:15)] socket error encountered while polling: Connection reset by peer [2025-09-25T11:04:50.102] [6451416.0] error: _negotiate: s2n_negotiate() failed S2N_ERR_CERT_UNTRUSTED[335544366]: Certificate is untrusted -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/tls/s2n_x509_validator.c:494 (couple of these) [2025-09-25T11:05:57.878] [6451416.0] error: tls_p_recv: s2n_recv() failed S2N_ERR_CLOSED[134217728]: connection is closed -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:37 [2025-09-25T11:05:57.883] [6451416.0] error: tls_p_send: s2n_send() failed S2N_ERR_IO[67108864]: underlying I/O operation failed, check system errno -> Error encountered in /builddir/build/BUILD/s2n-tls-1.5.10/utils/s2n_io.c:28 (couple of these) [2025-09-25T11:05:59.076] error: wrap_on_data: [unix:/var/spool/slurmd/slurmd.socket(fd:17)] on_data returned rc: Unable to proxy slurmstepd message [2025-09-25T11:05:59.076] [6451416.0] error: _stepd_send_recv_msg: slurmd was unable to proxy request message to its final destination [2025-09-25T11:05:59.878] error: _slurmd_send_recv_msg: Failed to send/recv slurmstepd message MESSAGE_TASK_EXIT using proxy_type PROXY_TO_NODE_SEND_RECV 2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: pmixp_p2p_send: n388 [0]: pmixp_utils.c:469: send failed, rc=1001, exceeded the retry limit [2025-09-25T11:07:36.335] [6451416.0] error: mpi/pmix_v4: _slurm_send: n388 [0]: pmixp_server.c:1586: Cannot send message to /var/spool/slurmd/stepd.slurm.pmix.6451416.0, size = 27679, hostlist: (null) (and couple more PMIx errors). Looks like PMIx cannot talk to their peers now ? There was no specific configuration for the certgen plugin, because SLURM documentation seems to say it is optional(?). I wonder what do we miss here to have SLURM 25.05 in with TLS enabled and PMIx working? Any advice appreciated! Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

1 0

Node switching randomly to down state
by Julien Tailleur 24 Sep '25

24 Sep '25

Dear all, I am maintaining a small computing cluster and I have a weird behavior that I fail at debugging. My cluster comprise one master node and 16 computing servers, organized in two queues, each queue having 8 servers. All servers run up-to-date Debian bullseye. All but 3 servers work flawlessly. From the master node, I can see that 3 servers on one of the queue appear down: jtailleu@kandinsky:~$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 down* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 These servers are reachable by SSH/ping jtailleu@kandinsky:~$ ping -c 1 FX12 PING FX12 (192.168.6.22) 56(84) bytes of data. 64 bytes from FX12 (192.168.6.22): icmp_seq=1 ttl=64 time=0.070 ms --- FX12 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.070/0.070/0.070/0.000 ms ##### I can also put these nodes back into idle mode: root@kandinsky:~# scontrol update nodename=FX[12-14] state=idle root@kandinsky:~# sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 idle* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 But then, they switch back into down mode few minutes later: root@kandinsky:~# sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST Volume* up infinite 8 alloc FX[21-24,31-34] Speed up infinite 3 down* FX[12-14] Speed up infinite 4 alloc FX[41-44] Speed up infinite 1 idle FX11 root@kandinsky:~# sinfo -R REASON USER TIMESTAMP NODELIST Not responding slurm 2025-09-08T15:04:39 FX[12-14] I do not understand where the "not responding" comes from, nor how I can investigate that. Any idea what could trigger this behavior? Best wishes, Julien

4 4

No output and can't job by id
by Dhumal, Dr. Nilesh 22 Sep '25

22 Sep '25

Hello, Recently, we updated Slurm. We are running slrumcld on head node. The slurmdbd is disabled on the head node. The user submits the job and gets the job ID. The squeue is empty, and difficult to track the job progress. The job is not writing any output to a file. How can I resolve this issue? Nilesh Dhumal Associate Professor of Chemistry, http://faculty.fgcu.edu/ndhumal/ Coordinator, FGCU Computational Facility, https://www.fgcu.edu/cas/facultyresources/computationalfacility/ SH-430; Department of Chemistry and Physics Florida Gulf Coast University 10501 FGCU Boulevard South Fort Myers, FL 33965-6565 Phone: (239) 745-4394 Email: ndhumal(a)fgcu.edu

2 2

Node in drain state
by Gestió Servidors 22 Sep '25

22 Sep '25

Hello, I have got a node in "drain" state after finishing a job that was running on it. Log in node reports this information: [...] [2025-09-07T11:09:26.980] task/affinity: task_p_slurmd_batch_request: task_p_slurmd_batch_request: 59238 [2025-09-07T11:09:26.980] task/affinity: batch_bind: job 59238 CPU input mask for node: 0xFFF [2025-09-07T11:09:26.980] task/affinity: batch_bind: job 59238 CPU final HW mask for node: 0xFFF [2025-09-07T11:09:26.980] Launching batch job 59238 for UID 21310 [2025-09-07T11:09:27.006] cred/munge: init: Munge credential signature plugin loaded [2025-09-07T11:09:27.007] [59238.batch] debug: auth/munge: init: loaded [2025-09-07T11:09:27.009] [59238.batch] debug: Reading cgroup.conf file /soft/slurm-23.11.0/etc/cgroup.conf [2025-09-07T11:09:27.025] [59238.batch] debug: cgroup/v1: init: Cgroup v1 plugin loaded [2025-09-07T11:09:27.025] [59238.batch] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: core enforcement enabled [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: device enforcement enabled [2025-09-07T11:09:27.026] [59238.batch] debug: task/cgroup: init: Tasks containment cgroup plugin loaded [2025-09-07T11:09:27.026] [59238.batch] task/affinity: init: task affinity plugin loaded with CPU mask 0xfff [2025-09-07T11:09:27.027] [59238.batch] debug: jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded [2025-09-07T11:09:27.027] [59238.batch] topology/default: init: topology Default plugin loaded [2025-09-07T11:09:27.030] [59238.batch] debug: gpu/generic: init: init: GPU Generic plugin loaded [2025-09-07T11:09:27.031] [59238.batch] debug: laying out the 12 tasks on 1 hosts clus09 dist 2 [2025-09-07T11:09:27.031] [59238.batch] debug: close_slurmd_conn: sending 0: No error [2025-09-07T11:09:27.031] [59238.batch] debug: Message thread started pid = 910040 [2025-09-07T11:09:27.031] [59238.batch] debug: Setting slurmstepd(910040) oom_score_adj to -1000 [2025-09-07T11:09:27.031] [59238.batch] debug: spank: opening plugin stack /soft/slurm-23.11.0/etc/plugstack.conf [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: job abstract cores are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: step abstract cores are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: job physical CPUs are '0-11' [2025-09-07T11:09:27.031] [59238.batch] debug: task/cgroup: task_cgroup_cpuset_create: step physical CPUs are '0-11' [2025-09-07T11:09:27.090] [59238.batch] debug levels are stderr='error', logfile='debug', syslog='fatal' [2025-09-07T11:09:27.090] [59238.batch] starting 1 tasks [2025-09-07T11:09:27.090] [59238.batch] task 0 (910044) started 2025-09-07T11:09:27 [2025-09-07T11:09:27.098] [59238.batch] debug: task/affinity: task_p_pre_launch: affinity StepId=59238.batch, task:0 bind:mask_cpu [2025-09-07T11:09:27.098] [59238.batch] _set_limit: RLIMIT_NPROC : reducing req:255366 to max:159631 [2025-09-07T11:09:27.398] [59238.batch] task 0 (910044) exited with exit code 2. [2025-09-07T11:09:27.399] [59238.batch] debug: task/affinity: task_p_post_term: affinity StepId=59238.batch, task 0 [2025-09-07T11:09:27.399] [59238.batch] debug: signaling condition [2025-09-07T11:09:27.399] [59238.batch] debug: jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded [2025-09-07T11:09:27.400] [59238.batch] debug: task/cgroup: fini: Tasks containment cgroup plugin unloaded [2025-09-07T11:09:27.400] [59238.batch] job 59238 completed with slurm_rc = 0, job_rc = 512 [2025-09-07T11:09:27.410] [59238.batch] debug: Message thread exited [2025-09-07T11:09:27.410] [59238.batch] stepd_cleanup: done with step (rc[0x200]:Unknown error 512, cleanup_rc[0x0]:No error) [2025-09-07T11:09:27.411] debug: _rpc_terminate_job: uid = 1000 JobId=59238 [2025-09-07T11:09:27.411] debug: credential for job 59238 revoked [...] "sinfo" shows: [root@login-node ~]# sinfo PARTITION TIMELIMIT AVAIL STATE NODELIST CPU_LOAD NODES(A/I) NODES(A/I/O/T) CPUS CPUS(A/I/O/T) REASON node.q* 4:00:00 up drained clus09 0.00 0/0 0/0/1/1 12 0/0/12/12 Kill task faile node.q* 4:00:00 up allocated clus[10-11] 13.82-15.8 2/0 2/0/0/2 12 24/0/0/24 none node.q* 4:00:00 up idle clus[01-06,12] 0.00 0/7 0/7/0/7 12 0/84/0/84 none But it seems there is no error in node... Slurmctld.log in server seems correct, too. Is there any way to reset node to "state=idle" after errors in the same way? Thanks.

3 4

Compute node not responding
by Dhumal, Dr. Nilesh 21 Sep '25

21 Sep '25

Hello, Recently, we installed slum 25 on our cluster. We are not monitoring the user's account. We didn't configure the sql database on the head node. We are running slurmcld on head node and slumd on the compute node. We are getting the following error Head node: compute node not responding. Compute node: 2025-09-19T15:30:23.461] error: Unable to register: Unable to contact slurm controller (connect failure) Do we need to run slumdbd on the head node? I checked the network connection by pinging the compute node from the head node. Do you have any suggestions to resolve this issue? Thanks Nilesh Get Outlook for Android<https://aka.ms/AAb9ysg>

4 5

Re: Node in drain state
by Gestió Servidors 19 Sep '25

19 Sep '25

Hi, After reading answer from Ole Holm Nielsen, I have increased "MessageTimeout" to 20s (by default is 5s) and "UnkillableStepTimeout" to 150s (by default is 60s and, always 5 times larger than "MessageTimeout"). However, I have also read that UnkillableStepProgram indicates the program to use in that cases... but, by default there is no program assigned to that parameter (no program to run). So my question is if someone uses a customized "UnkillableStepProgram" and if he/she could explain that. Thanks a lot!

3 2

seff for GPU
by Josu Lazkano Lete 18 Sep '25

18 Sep '25

Hello, We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff? It will be great to know how much GPU resources the users request and compare with how much GPU resources they use. Kind regards. [image: Vicomtech] <https://www.vicomtech.org> Josu Lazkano Lete Systems Manager Infrastructures and General Services jlazkano(a)vicomtech.org +(34) 943 30 92 30 The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it. Avoid printing this message if it is not strictly necessary.

7 9

Scheduling issues with multiple different types of GPU in one partition
by Kevin M. Hildebrand 17 Sep '25

17 Sep '25

We have several different types of GPUs in the same 'gpu' partition. The problem we're having occurs when one of those types of GPUs is fully occupied and there are a bunch of queued jobs waiting for those GPUs. If someone requests idle GPUs of a different type, those jobs end up getting stalled, even though there are plenty of GPUs available. For example, say we have 10 A100 GPUs and 10 H100 GPUs. If there are 10 H100 GPU jobs running and more in queue waiting for them, subsequently submitted A100 jobs will sit in queue even if there are plenty of idle A100 GPUs. The only way we can get the A100 jobs to run is by manually bumping their priority higher than the pending H100 jobs. Has anyone else encountered this issue? The only way we can think of to potentially solve it is to have separate partitions for each GPU type, but that seems unwieldy. We are currently running Slurm 24.05.8. Thanks, Kevin -- Kevin Hildebrand Director of Research Technology and HPC Services Division of IT

7 7

New "NOT-state" selection of the sinfo command in Slurm 25.05
by Ole Holm Nielsen 10 Sep '25

10 Sep '25

We just upgraded Slurm to 25.05.3, and I would like to highlight a new functionality of the "sinfo -t <state>" command in 25.05: > The state can be prefixed with '~' which will invert the result of match. We find this "NOT-state" selection useful together with Slurm power saving [1] where any idle nodes may be powered off to save electrical power or cloud costs. For example, the ClusterShell [2] clush command can now skip connections to the powered down nodes like in this example: $ clush -bw@slurmstate:~POWERED_DOWN uname -r You can even configure ClusterShell so that "clush -a" includes all nodes which are *not* powered down. The details are described in the Wiki page [3]. Best regards, Ole [1] https://slurm.schedmd.com/power_save.html [2] https://clustershell.readthedocs.io/en/latest/intro.html [3] https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_operations/#skipping-powere… -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark

1 0

Development RPMs for cgroups v2
by John Hearns 10 Sep '25

10 Sep '25

I am building a version of Slurm on RHEL 9.4 When I run an rpmbuild, he slurm rpm does not contain /usr/lib64/slurm/cgroup_v2.so I have tried to look in the build logs. I suspect I am lacking some development RPMs - but which ones? All hints gratefully received John H

4 5

NHC skips nodes in "mix-" state
by blines＠tgen.org 08 Sep '25

08 Sep '25

The node-mark-offline script skips nodes in the "mix-" state with the error: State "mix-" not yet handled; ignoring. "mix-" seems to be a valid state indicating that a node is mixed and is "planned by the backfill scheduler for a higher priority job". Anyone know the reason that nodes in this state are being skipped? The responsible code appears to be: case "$STATUS" in *'@'*|*'#'*|boot*|*-*|plnd*) # These states aren't handled yet. echo "$0: State \"$STATUS\" not yet handled; ignoring." exit 0 As far as I can tell all the states being skipped are related to nodes being configured or rebooted or powering on/off except *-*. There are other states related to node power that include underscore in the state name; power_down, power_up, pow_dn, pow_up. Should the case statement be *_* instead of *-*?

1 0

Creating /run/user/$UID - for Podman runtime
by John Snowdon 08 Sep '25

08 Sep '25

We are in the middle of implementing an extensive range of container support on our new HPC platform and have decided to offer our users a wide suite of technologies to better support their workloads: * Apptainer * Podman (rootless) * Docker (rootless) We've already got a solution for automated entries in /etc/subuid and /etc/subgid on the head nodes (available here under GPL: https://github.com/megatron-uk/pam_subid) which is where we intend users to build their container images, and building and running containers using Apptainer and Podman in those environments works really well - we're happy that it should take care of 95% of our users needs (Docker is the last few percent....) and not involve giving them any special permissions. If I ssh directly to a compute node, then Podman also works there to run an existing image (podman container run ...). What I'm struggling with now is running Podman under Slurm itself on our compute nodes. It appears as though Podman (in rootless mode) wants to put the majority of its run time / state information under /run/user/$UID ... this is fine on the head nodes which have interactive logins hitting PAM modules which instantiate the /run/user/$UID directories, but not under sbatch/srun which doesn't create them by default. I've not been able to find a single, magical setting which will move all of the Podman state information out from /run/user to another location - there are 3 or 4 settings involved, and even then I still find various bits of Podman want to create stuff under there. Rather than hacking away at getting Podman changed to move all settings and state information elsewhere, it seems like the cleanest solution would just be to put the regular /run/user/$UID directory in place at the point Slurm starts the job instead. What's the best way to get Slurm to create this and clean-up afterwards? Should this be in a prolog/epilog wrapper (e.g. directly calling loginctl) or is it cleaner to get Slurm to trigger the usual PAM session machinery in some manner? John Snowdon Senior Research Infrastructure Engineer (HPC) Research Software Engineering Catalyst Building, Room 2.01 Newcastle University 3 Science Square Newcastle Helix Newcastle upon Tyne NE4 5TG https://hpc.researchcomputing.ncl.ac.uk

5 7

Slurm version 25.05.3 is now available
by Marshall Garey 04 Sep '25

04 Sep '25

We are pleased to announce the availability of Slurm version 25.05.3. This version fixes an issue that prevented deleting a QOS when running with MySQL servers (MariaDB is was unaffected). Please note that the slurmdbd will require MySQL 8.0.4+ or MariaDB 10.0.5+ to function correctly. This version also fixes heterogeneous jobs when TLS is enabled, a logging issue with syslog, and various mild to moderate stability fixes. The full list of changes are available in the CHANGELOG: https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support

1 0

Discussion about the --cpus-per-task flag
by Michele Esposito Marzino 04 Sep '25

04 Sep '25

Hi everyone, I would like to ask about the rationale behind the --cpus-per-task flag. While I do understand the meaning and its intended goal, I was wondering how it is managed in practice. I tried inferring this from the source code, but it is quite intricate, and not really straightforward by simply searching for the string match. Does it affect the CPUSET of each task? Who is responsible for setting this? The slurmcltd or slurmd? And how? Or at least, point me where in the code it is handled, so that I can reconstruct the whole process myself. Thanks

1 0

sacctmgr output with and without associations
by Rémy Dernat 04 Sep '25

04 Sep '25

Hi, I have a stupid issue about sacctmgr output and associations. I would like to retrieve my accounts with all the following data : "Account","Clusters","DefaultQOS","Description","Fairshare","Flags","GrpTRESMins","GrpTRES","GrpJobs","GrpSubmitJob",GrpWall", "MaxTRESMins","MaxTRES","MaxJobs","MaxNodes","MaxSubmitJobs","MaxWall","Organization","ParentName","Priority","QosLevel" So my command is the following : sacctmgr list accounts --assoc format="Account","Clusters","DefaultQOS","Description","Fairshare","Flags","GrpTRESMins","GrpTRES","GrpJobs","GrpSubmitJob","GrpWall","MaxTRESMins","MaxTRES","MaxJobs","MaxNodes","MaxSubmitJobs","MaxWall","Organization","ParentName","Priority","QosLevel" -n -P However, I have a lot of outputs corresponding to each User association entry. That would be fine if a basic `|uniq` would remove everything I don't need, but I am stuck with grptres, which can be seen in the main account, but not in the children associations with the Users. On the opposite, if I remove `--assoc`, I don't have anymore grptres, ParentName, or related Qos values. I had to add `where User=""` to my first command to just retrieve my main accounts, without User associations. I dont' know if that is clear enough; is there something I missed with the sacctmgr output ? Thanks, Best regards, Rémy

2 1

Can't remove account
by Matthias Leopold 01 Sep '25

01 Sep '25

Hi, I have a cluster with account foo. I want to delete this account. For every user that had associations in this account I created new associations in a different account and changed the DefaultAccount of the user. Then I deleted all associations in account foo. Now I want to delete account foo: sacctmgr delete account name=foo This gives: Error with request: You can not remove the default account of a user ... Please either remove the accounts listed above from list and resubmit, or change these users' default accounts to remove the account(s). Changes Discarded It names all associations for users that FORMERLY had DefaultAccount foo?? NO user has DefaultAccount foo anymore... I have to mention that I also have partition foo and affected users have an association in their new account with partition foo. But that shouldn't matter IMHO Has anybody seen this? What can I do? Am I getting something wrong? Slurm version is 24.05.7 thanks Matthias

1 0

slurmrestd unable to find plugin rest_auth/jwt
by Jon Marshall 28 Aug '25

28 Aug '25

Hi, I feel like I must have missed something obvious here, so hopefully someone can spot it! We recently updated to Slurm 24.11.6 on our cluster and I made sure to compile --with slurmrestd and --with jwt, as there are a few things I've been meaning to try out with the REST API. From the rpmbuild config.log I get: $ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc/slurm --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-systemdsystemunitdir=/usr/lib/systemd/system --enable-pkgconfig --with-jwt -- configure:21020: checking for jwt.h configure:21020: gcc -c -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer conftest.c >&5 configure:21020: $? = 0 configure:21020: result: yes configure:21028: checking for jwt_add_header in -ljwt configure:21051: gcc -o conftest -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -Wl,-z,relro -Wl,-z,lazy conftest.c -ljwt -ldl -lrt -lm -lresolv >&5 configure:21051: $? = 0 configure:21061: result: yes -- config.status:1746: creating src/plugins/auth/jwt/Makefile -- config.status:1746: creating src/slurmrestd/plugins/auth/jwt/Makefile -- ac_cv_header_jwt_h=yes -- ac_cv_lib_jwt_jwt_add_header=yes -- JWT_LDFLAGS='-ljwt' And: configure:26873: checking whether to compile slurmrestd configure:26904: result: yes -- configure:26915: checking for slurmrestd default port configure:26933: result: 6820 And indeed I get a slurmrestd package out of it, which I've successfully installed with no errors. I've created a JWT key and rolled it out to the controller node, but when I go to check auth types I get: slurmrestd -a rest_auth/jwt slurmrestd: fatal: Unable to find plugin: rest_auth/jwt Running an strace on this process shows that it does in fact stat the .so that does in fact exist: stat("/usr/lib64/slurm/auth_jwt.so", {st_mode=S_IFREG|0755, st_size=283728, ...}) = 0 And I'm kind of stumped now. I've checked through all the resolved bug tickets to see if I can spot what is wrong but I'm getting nothing, and I can't find anything in the mailing lists either, other than things I've checked. Has anyone else had similar issues, or can someone see the blindingly obvious thing I'm missing? Cheers Jon Jon Marshall High Performance Computing Specialist IT and Scientific Computing Team Cancer Research UK Cambridge Institute Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE Web<http://www.cruk.cam.ac.uk/> | Facebook<http://www.facebook.com/cancerresearchuk> | Twitter<http://twitter.com/CR_UK> [Description: CRI Logo]<http://www.cruk.cam.ac.uk/>

2 4

Assistance with Node Restrictions and Priority for Users in Floating Partition
by Manisha Yadav 28 Aug '25

28 Aug '25

Dear Team, I have a scenario where I need to provide priority access to multiple users from different projects for only 3 nodes. This means that, at any given time, only 3 nodes can be used in that partition, and if one user is utilizing all 3 nodes, no other user should be able to submit jobs to that partition, or their jobs should remain in the queue. To achieve this, I attempted to use QoS by creating a floating partition with some of the nodes and configuring a QoS with priority. I also set a limit with GrpTRES=gres/gpu=24, given that each node has 8 GPUs, and there are 3 nodes in total. I then attached the QoS to the partition and assigned it to the users who need access. I Also tried MaxTRES=gres/gpu=24 While this setup works as expected in the testing environment for CPUs, it is not functioning as intended in production, and it is not effectively restricting node usage in the partition. Could anyone provide suggestions or guidance on how to properly implement node restrictions along with priority? Thank you for your assistance. Best regards, Manisha Yadav ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

4 8

Node Health Check Program
by Paul Edmon 27 Aug '25

27 Aug '25

We've been using NHC (https://github.com/mej/nhc) for years with much success. However that project hasn't had a release in 2 years and the various Issues filed indicate that there might be problems with Rocky 9 (which we are looking to upgrade to). Do people that are at EL9 use NHC? Is there a fork? Is there a different code that people use for doing node health checks? -Paul Edmon-

8 16