- slurm-users - lists.schedmd.com

System limits propagation
by Fatih Ertinaz 31 Jan '25

31 Jan '25

Hi everyone, I saw similar discussions in the archives, however I could not see a clear explanation therefore decided to open up a new thread. The problem I have is that default ulimits defined in the compute node OS image are overridden by Slurm and somehow changes in slurm.conf does not resolve the problem. This is specifically about max memory size, but the same issue applies to stack size and user processes as well. From the head node or from one of the compute nodes if I bypass slurm: max memory size (kbytes, -m) unlimited But going through salloc / sbatch it becomes: max memory size (kbytes, -m) 1024 As an initial attempt I tried setting `PropagateResourceLimitsExcept=NONE` but it didn't help. Then I tried `PropagateResourceLimits=ALL` also with no luck. So in theory PropagateResourceLimits should propagate all limits, but I'm not sure if that's really the case. I'm open to suggestions, Fatih Ertinaz Note: Slurm version is 24.05.1 OS is SLES SP5

2 1

Issues starting the SLURMCTLD on Rocky
by Vitorio Cargnini 31 Jan '25

31 Jan '25

Hello, I just deployed the latest SLURM and I am getting some odd issues restating it. Anyone saw this before, how can I fix it? Jan 30 17:22:09 slurmctrl01 systemd[1]: slurmctld.service: Main process exited, code=dumped, status=6/ABRT Jan 30 17:22:09 slurmctrl01 systemd[1]: slurmctld.service: Failed with result 'core-dump'. Jan 30 17:22:09 slurmctrl01 systemd[1]: Failed to start Slurm controller daemon. Jan 30 17:22:09 slurmctrl01 systemd[1]: systemd-coredump(a)8-92126-0.service: Succeeded. journalctl -xe #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92119: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92112: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92122: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92111: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92115: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92123: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92104: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) -- Subject: Process 92054 (slurmctld) dumped core -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- Documentation: man:core(5) -- -- Process 92054 (slurmctld) crashed and dumped core. -- -- This usually indicates a programming error in the crashing program and -- should be reported to its vendor as a bug. Jan 30 17:22:09 slurmctrl01.internal.samsung systemd[1]: slurmctld.service: Main process exited, code=dumped, status=6/ABRT Jan 30 17:22:09 slurmctrl01.internal.samsung systemd[1]: slurmctld.service: Failed with result 'core-dump'. -- Subject: Unit failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- The unit slurmctld.service has entered the 'failed' state with result 'core-dump'. Jan 30 17:22:09 slurmctrl01.internal.samsung systemd[1]: Failed to start Slurm controller daemon. -- Subject: Unit slurmctld.service has failed -- Defined-By: systemd Regards, Vitorio

2 1

Re: RHEL8.10 V slurmctld
by Steven Jones 30 Jan '25

30 Jan '25

Hi, Already done as part of the build process. regards Steven ________________________________ From: Williams, Jenny Avis <jenny_williams(a)unc.edu> Sent: Friday, 31 January 2025 9:06 am To: Steven Jones <steven.jones(a)vuw.ac.nz>; John Hearns <hearnsj(a)gmail.com> Cc: slurm-users(a)schedmd.com <slurm-users(a)schedmd.com> Subject: RE: [slurm-users] Re: RHEL8.10 V slurmctld You don't often get email from jenny_williams(a)unc.edu. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> First I’d verify munge functionality in the updated environment – https://github.com/dun/munge/wiki/Installation-Guide#troubleshooting From: Steven Jones <steven.jones(a)vuw.ac.nz> Sent: Thursday, January 30, 2025 2:55 PM To: Williams, Jenny Avis <jenny_williams(a)unc.edu>; John Hearns <hearnsj(a)gmail.com> Cc: slurm-users(a)schedmd.com Subject: Re: [slurm-users] Re: RHEL8.10 V slurmctld Hi, Hmmm, yes I am using munge. [root@node1 ~]# strings `which slurmd` |egrep -i munge [root@node1 ~]# Does not return anything on the nodes, but worked fine for RHEL9.5 [root@xxxunidrslurmd2 munge]# scontrol show config |egrep -i auth AuthAltTypes = (null) AuthAltParameters = (null) AuthInfo = (null) AuthType = auth/munge [root@vuwunidrslurmd2 munge]# Munge logs are 0 length ============= slurmd2 slurm]# rpm -qi munge Name : munge Version : 0.5.13 Release : 2.el8 Architecture: x86_64 Install Date: Wed 15 Jan 2025 02:11:46 AM UTC Group : Unspecified Size : 320124 License : GPLv3+ and LGPLv3+ Signature : RSA/SHA256, Mon 27 Apr 2020 11:43:24 PM UTC, Key ID 199e2f91fd431d51 Source RPM : munge-0.5.13-2.el8.src.rpm Build Date : Fri 24 Apr 2020 07:37:08 AM UTC Build Host : x86-vm-02.build.eng.bos.redhat.com Relocations : (not relocatable) Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : https://dun.github.io/munge/ Summary : Enables uid & gid authentication across a host cluster Description : MUNGE (MUNGE Uid 'N' Gid Emporium) is an authentication service for creating and validating credentials. It is designed to be highly scalable for use in an HPC cluster environment. It allows a process to authenticate the UID and GID of another local or remote process within a group of hosts having common users and groups. These hosts form a security realm that is defined by a shared cryptographic key. Clients within this security realm can create and validate credentials without the use of root privileges, reserved ports, or platform-specific methods. ========== [root@node1 ~]# rpm -qi munge Name : munge Version : 0.5.13 Release : 2.el8 Architecture: x86_64 Install Date: Sun Jan 26 22:14:28 2025 Group : Unspecified Size : 319876 License : GPLv3+ and LGPLv3+ Signature : RSA/SHA256, Mon Apr 12 06:46:59 2021, Key ID 15af5dac6d745a60 Source RPM : munge-0.5.13-2.el8.src.rpm Build Date : Mon Apr 12 05:07:29 2021 Build Host : ord1-prod-x86build003.svc.aws.rockylinux.org Relocations : (not relocatable) Packager : infrastructure(a)rockylinux.org<mailto:infrastructure@rockylinux.org> Vendor : Rocky URL : https://dun.github.io/munge/ Summary : Enables uid & gid authentication across a host cluster Description : MUNGE (MUNGE Uid 'N' Gid Emporium) is an authentication service for creating and validating credentials. It is designed to be highly scalable for use in an HPC cluster environment. It allows a process to authenticate the UID and GID of another local or remote process within a group of hosts having common users and groups. These hosts form a security realm that is defined by a shared cryptographic key. Clients within this security realm can create and validate credentials without the use of root privileges, reserved ports, or platform-specific methods. [root@node1 ~]# ========== [root@node1 ~]# rpm -qi slurm-slurmd Name : slurm-slurmd Version : 20.11.9 Release : 1.el8 Architecture: x86_64 Install Date: Sun Jan 26 22:15:34 2025 Group : Unspecified Size : 517922 License : GPLv2 and BSD Signature : RSA/SHA256, Fri May 6 00:03:49 2022, Key ID 21ea45ab2f86d6a1 Source RPM : slurm-20.11.9-1.el8.src.rpm Build Date : Thu May 5 23:33:30 2022 Build Host : buildvm-x86-18.iad2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://slurm.schedmd.com/ Bug URL : https://bugz.fedoraproject.org/slurm Summary : Slurm compute node daemon Description : Slurm compute node daemon. Used to launch jobs on compute nodes [root@node1 ~]# ======= slurmd2 ~]# rpm -qi slurm-slurmctld Name : slurm-slurmctld Version : 20.11.9 Release : 1.el8 Architecture: x86_64 Install Date: Wed 15 Jan 2025 02:11:48 AM UTC Group : Unspecified Size : 1097306 License : GPLv2 and BSD Signature : RSA/SHA256, Fri 06 May 2022 12:03:49 AM UTC, Key ID 21ea45ab2f86d6a1 Source RPM : slurm-20.11.9-1.el8.src.rpm Build Date : Thu 05 May 2022 11:33:30 PM UTC Build Host : buildvm-x86-18.iad2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://slurm.schedmd.com/ Bug URL : https://bugz.fedoraproject.org/slurm Summary : Slurm controller daemon Description : Slurm controller daemon. Used to manage the job queue, schedule jobs, and dispatch RPC messages to the slurmd processon the compute nodes to launch jobs. regards Steven ________________________________ From: Williams, Jenny Avis <jenny_williams(a)unc.edu<mailto:jenny_williams@unc.edu>> Sent: Friday, 31 January 2025 8:36 am To: Steven Jones <steven.jones(a)vuw.ac.nz<mailto:steven.jones@vuw.ac.nz>>; John Hearns <hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>> Cc: slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com> <slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com>> Subject: RE: [slurm-users] Re: RHEL8.10 V slurmctld You don't often get email from jenny_williams(a)unc.edu<mailto:jenny_williams@unc.edu>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> On both a compute node and the controller rpm -qi slurm-slurmctld rpm -qi slurm-slurmd check what the auth type is – for example, we still use munge, which in my compile is also the default auth type. : # strings `which slurmd` |egrep -i munge DEFAULT_AUTH_TYPE "auth/munge" DEFAULT_CRED_TYPE "cred/munge" #scontrol show config |egrep -i auth AuthAltTypes = (null) AuthAltParameters = (null) AuthInfo = (null) AuthType = auth/munge From: Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Sent: Thursday, January 30, 2025 2:07 PM To: John Hearns <hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>> Cc: slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com> Subject: [slurm-users] Re: RHEL8.10 V slurmctld Hi, Yes, even ssh works OK. [root@xxxunicobuildt1 warewulf]# ssh xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1> (xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1>) Password: Last login: Wed Jan 29 01:26:21 2025 from 130.195.87.12 [xxxjonesst@xxx.ac.nz@node1 ~]$ xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1> ~]$ whoami | id uid=1204805830(xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz>) gid=1204805830(xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz>) tail -f /var/log/secure ========= Jan 30 18:19:56 node1 sshd[15443]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser= rhost=130.195.87.12 user=xxxjonesst(a)xxx.ac.nz<mailto:user=xxxjonesst@xxx.ac.nz> Jan 30 18:19:56 node1 sshd[15440]: Accepted keyboard-interactive/pam for xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz> from 130.195.87.12 port 59402 ssh2 Would there be any relevant changes between RHEL8's slurm and RHEL9's slurm? [root@node1 ~]# rpm -qa |grep slurm slurm-libs-20.11.9-1.el8.x86_64 slurm-slurmd-20.11.9-1.el8.x86_64 slurm-20.11.9-1.el8.x86_64 [root@node1 ~]# I would have to go back and check but I do not think I hit this on RHEL9 what I did get was srun ver22 on the RHEL9 server didnt like srun ver20 on the rocky8 node. Can I compile / rpm build srun ver22 to run on rocky8? or is that part of slurmd? regards Steven ________________________________ From: John Hearns <hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>> Sent: Thursday, 30 January 2025 10:53 pm To: Steven Jones <steven.jones(a)vuw.ac.nz<mailto:steven.jones@vuw.ac.nz>> Cc: slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com> <slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com>> Subject: Re: [slurm-users] RHEL8.10 V slurmctld You don't often get email from hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Have you run id on a computer node? On Wed, Jan 29, 2025, 6:47 PM Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> wrote: I am using Redhat's IdM/IPA for users Slurmctld is failing to run jobs and it is getting "invalid user id". "2025-01-28T21:48:50.271] sched: Allocate JobId=4 NodeList=node4 #CPUs=1 Partition=debug [2025-01-28T21:48:50.280] Killing non-startable batch JobId=4: Invalid user id" id on the slurm controller works fine. [xxxjoness@xxx.ac.nz@hpcunidrslurmd2 ~]$ id xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz> uid=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) gid=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) groups=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) 8><--- Any ideas please? because I am out..... I have tried RHEL9.5, this seemed to run but srun is version 22 and on rocky8 it is version20 so fails. regards Steven -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>

1 0

Re: RHEL8.10 V slurmctld
by Steven Jones 30 Jan '25

30 Jan '25

Hi, Hmmm, yes I am using munge. [root@node1 ~]# strings `which slurmd` |egrep -i munge [root@node1 ~]# Does not return anything on the nodes, but worked fine for RHEL9.5 [root@xxxunidrslurmd2 munge]# scontrol show config |egrep -i auth AuthAltTypes = (null) AuthAltParameters = (null) AuthInfo = (null) AuthType = auth/munge [root@vuwunidrslurmd2 munge]# Munge logs are 0 length ============= slurmd2 slurm]# rpm -qi munge Name : munge Version : 0.5.13 Release : 2.el8 Architecture: x86_64 Install Date: Wed 15 Jan 2025 02:11:46 AM UTC Group : Unspecified Size : 320124 License : GPLv3+ and LGPLv3+ Signature : RSA/SHA256, Mon 27 Apr 2020 11:43:24 PM UTC, Key ID 199e2f91fd431d51 Source RPM : munge-0.5.13-2.el8.src.rpm Build Date : Fri 24 Apr 2020 07:37:08 AM UTC Build Host : x86-vm-02.build.eng.bos.redhat.com Relocations : (not relocatable) Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : https://dun.github.io/munge/ Summary : Enables uid & gid authentication across a host cluster Description : MUNGE (MUNGE Uid 'N' Gid Emporium) is an authentication service for creating and validating credentials. It is designed to be highly scalable for use in an HPC cluster environment. It allows a process to authenticate the UID and GID of another local or remote process within a group of hosts having common users and groups. These hosts form a security realm that is defined by a shared cryptographic key. Clients within this security realm can create and validate credentials without the use of root privileges, reserved ports, or platform-specific methods. ========== [root@node1 ~]# rpm -qi munge Name : munge Version : 0.5.13 Release : 2.el8 Architecture: x86_64 Install Date: Sun Jan 26 22:14:28 2025 Group : Unspecified Size : 319876 License : GPLv3+ and LGPLv3+ Signature : RSA/SHA256, Mon Apr 12 06:46:59 2021, Key ID 15af5dac6d745a60 Source RPM : munge-0.5.13-2.el8.src.rpm Build Date : Mon Apr 12 05:07:29 2021 Build Host : ord1-prod-x86build003.svc.aws.rockylinux.org Relocations : (not relocatable) Packager : infrastructure(a)rockylinux.org Vendor : Rocky URL : https://dun.github.io/munge/ Summary : Enables uid & gid authentication across a host cluster Description : MUNGE (MUNGE Uid 'N' Gid Emporium) is an authentication service for creating and validating credentials. It is designed to be highly scalable for use in an HPC cluster environment. It allows a process to authenticate the UID and GID of another local or remote process within a group of hosts having common users and groups. These hosts form a security realm that is defined by a shared cryptographic key. Clients within this security realm can create and validate credentials without the use of root privileges, reserved ports, or platform-specific methods. [root@node1 ~]# ========== [root@node1 ~]# rpm -qi slurm-slurmd Name : slurm-slurmd Version : 20.11.9 Release : 1.el8 Architecture: x86_64 Install Date: Sun Jan 26 22:15:34 2025 Group : Unspecified Size : 517922 License : GPLv2 and BSD Signature : RSA/SHA256, Fri May 6 00:03:49 2022, Key ID 21ea45ab2f86d6a1 Source RPM : slurm-20.11.9-1.el8.src.rpm Build Date : Thu May 5 23:33:30 2022 Build Host : buildvm-x86-18.iad2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://slurm.schedmd.com/ Bug URL : https://bugz.fedoraproject.org/slurm Summary : Slurm compute node daemon Description : Slurm compute node daemon. Used to launch jobs on compute nodes [root@node1 ~]# ======= slurmd2 ~]# rpm -qi slurm-slurmctld Name : slurm-slurmctld Version : 20.11.9 Release : 1.el8 Architecture: x86_64 Install Date: Wed 15 Jan 2025 02:11:48 AM UTC Group : Unspecified Size : 1097306 License : GPLv2 and BSD Signature : RSA/SHA256, Fri 06 May 2022 12:03:49 AM UTC, Key ID 21ea45ab2f86d6a1 Source RPM : slurm-20.11.9-1.el8.src.rpm Build Date : Thu 05 May 2022 11:33:30 PM UTC Build Host : buildvm-x86-18.iad2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://slurm.schedmd.com/ Bug URL : https://bugz.fedoraproject.org/slurm Summary : Slurm controller daemon Description : Slurm controller daemon. Used to manage the job queue, schedule jobs, and dispatch RPC messages to the slurmd processon the compute nodes to launch jobs. regards Steven ________________________________ From: Williams, Jenny Avis <jenny_williams(a)unc.edu> Sent: Friday, 31 January 2025 8:36 am To: Steven Jones <steven.jones(a)vuw.ac.nz>; John Hearns <hearnsj(a)gmail.com> Cc: slurm-users(a)schedmd.com <slurm-users(a)schedmd.com> Subject: RE: [slurm-users] Re: RHEL8.10 V slurmctld You don't often get email from jenny_williams(a)unc.edu. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> On both a compute node and the controller rpm -qi slurm-slurmctld rpm -qi slurm-slurmd check what the auth type is – for example, we still use munge, which in my compile is also the default auth type. : # strings `which slurmd` |egrep -i munge DEFAULT_AUTH_TYPE "auth/munge" DEFAULT_CRED_TYPE "cred/munge" #scontrol show config |egrep -i auth AuthAltTypes = (null) AuthAltParameters = (null) AuthInfo = (null) AuthType = auth/munge From: Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com> Sent: Thursday, January 30, 2025 2:07 PM To: John Hearns <hearnsj(a)gmail.com> Cc: slurm-users(a)schedmd.com Subject: [slurm-users] Re: RHEL8.10 V slurmctld Hi, Yes, even ssh works OK. [root@xxxunicobuildt1 warewulf]# ssh xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1> (xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1>) Password: Last login: Wed Jan 29 01:26:21 2025 from 130.195.87.12 [xxxjonesst@xxx.ac.nz@node1 ~]$ xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1> ~]$ whoami | id uid=1204805830(xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz>) gid=1204805830(xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz>) tail -f /var/log/secure ========= Jan 30 18:19:56 node1 sshd[15443]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser= rhost=130.195.87.12 user=xxxjonesst(a)xxx.ac.nz<mailto:user=xxxjonesst@xxx.ac.nz> Jan 30 18:19:56 node1 sshd[15440]: Accepted keyboard-interactive/pam for xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz> from 130.195.87.12 port 59402 ssh2 Would there be any relevant changes between RHEL8's slurm and RHEL9's slurm? [root@node1 ~]# rpm -qa |grep slurm slurm-libs-20.11.9-1.el8.x86_64 slurm-slurmd-20.11.9-1.el8.x86_64 slurm-20.11.9-1.el8.x86_64 [root@node1 ~]# I would have to go back and check but I do not think I hit this on RHEL9 what I did get was srun ver22 on the RHEL9 server didnt like srun ver20 on the rocky8 node. Can I compile / rpm build srun ver22 to run on rocky8? or is that part of slurmd? regards Steven ________________________________ From: John Hearns <hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>> Sent: Thursday, 30 January 2025 10:53 pm To: Steven Jones <steven.jones(a)vuw.ac.nz<mailto:steven.jones@vuw.ac.nz>> Cc: slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com> <slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com>> Subject: Re: [slurm-users] RHEL8.10 V slurmctld You don't often get email from hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Have you run id on a computer node? On Wed, Jan 29, 2025, 6:47 PM Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> wrote: I am using Redhat's IdM/IPA for users Slurmctld is failing to run jobs and it is getting "invalid user id". "2025-01-28T21:48:50.271] sched: Allocate JobId=4 NodeList=node4 #CPUs=1 Partition=debug [2025-01-28T21:48:50.280] Killing non-startable batch JobId=4: Invalid user id" id on the slurm controller works fine. [xxxjoness@xxx.ac.nz@hpcunidrslurmd2 ~]$ id xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz> uid=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) gid=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) groups=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) 8><--- Any ideas please? because I am out..... I have tried RHEL9.5, this seemed to run but srun is version 22 and on rocky8 it is version20 so fails. regards Steven -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>

1 0

Behavior of 'afterok' in cloud clusters
by Thompson, Hoot (GSFC-606.0)[ADNET SYSTEMS INC] 30 Jan '25

30 Jan '25

The behavior of slurm jobs using the ‘afterok’ dependency seems to have issues with ephemeral compute nodes such as those in a cloud cluster. If the specified jobid dependency is associated with a compute node that has already spun down then a subsequent job that requires successful completion of the prior job will fail with a “Job dependency problem”. This occurs when the subsequent job is tied to a node that must spin up before beginning execution. This phenomenon does not occur if the ‘afterany’ dependency is used. It seems that job completion status is retained when a node is spun down but no information as to whether the job was successfully executed is saved. There are perhaps other scenarios that coud cause the same issue. Has anyone else witnessed this problem? How can it be avoided? afterany This job can begin execution after the specified jobs have terminated. aftercorr A task of this job array can begin execution after the corresponding task ID in the specified job has completed successfully afternotok This job can begin execution after the specified jobs have terminated in some failed state afterok This job can begin execution after the specified jobs have successfully executed

1 0

Optimizing CPU allocation in Slurm with hyperthreading enabled
by Guillaume COCHARD 30 Jan '25

30 Jan '25

Hello, The Slurm documentation states that "The default allocation method within a node is cyclic allocation (allocate available CPUs in a round-robin fashion across the sockets within a node)." In our case, we are using hyperthreading, which means that if two jobs requesting one CPU each arrive at the same time on an empty node, they will be assigned to the two threads of the same physical core. To improve performance, we would like Slurm to allocate CPUs (in our case, threads) in a way that prioritizes using one thread per physical core first, and only once all cores are occupied, assign the second thread of each core. This would be similar to the Least Loaded Node (LLN) approach, but instead of distributing jobs across nodes, it would distribute them across physical cores (Least Loaded Core). We do not want to disable hyperthreading; we simply want to optimize core usage when a machine is not fully loaded. We also want this to be a cluster-wide setting, and not a user and/or job setting. Our current slurm.conf contains: ConstrainCores = no SelectType = select/cons_tres SelectTypeParameters = CR_CPU_MEMORY And our cgroup.conf contains: ConstrainDevices=yes ConstrainCores=yes Does anyone know how to achieve this? Thanks, Guillaume

1 0

Cloud elastic help
by mark.w.moorcroft＠ama-inc.com 29 Jan '25

29 Jan '25

I have a new Slurm setup in AWS gov cloud that is not quite working. I will list a few factoids and maybe someone can suggest where to look next. The Troubleshooting page really has nothing relevant for elastic cloud deployments. The nodes are getting set to DOWN+CLOUD+POWERED_DOWN. Running a job does not launch a node in this state. I can force the nodes to launch with scontrol POWER_UP. The jobs will claim to run, then re-queue, but never complete. When the nodes boot I see them appear in slurmctl, but it soon reports connection lost. The node slurmd claims to be healthy, but the controller eventually just terminates them. You can ping in both directions with the hostnames. I've done 4 clusters. The first was torque/maui and the rest were slurm, but all were bare metal. This is my first attempt at cloud. We have ITAR data, so I can't use Amazon Parallel Computing because it is not offered in GovCloud. https://cluster-in-the-cloud.readthedocs.io/en/latest/running.html I had to fork this project because so much is obsolete, but it's mostly working for me now. https://github.com/mntbighker [root@mgmt ~]# journalctl -fu slurmctld Jan 29 19:16:15 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: 1 jobs to backfill Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: Updating partition uid access list Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: purge_old_job: job file deletion is falling behind, 1 left to remove Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched: Running job scheduler for full queue. Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: beginning Jan 29 19:16:45 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: 1 jobs to backfill Jan 29 19:16:50 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sackd_mgr_dump_state: saved state of 0 nodes Jan 29 19:17:15 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: beginning Jan 29 19:17:15 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: debug: sched/backfill: _attempt_backfill: 1 jobs to backfill Jan 29 19:17:26 mgmt.many-antelope.citc.local slurmctld[3403]: slurmctld: POWER: Power save mode: 4 nodes [root@mgmt ~]# scontrol show node many-antelope-c5n-2xlarge-0001 NodeName=many-antelope-c5n-2xlarge-0001 CoresPerSocket=4 CPUAlloc=0 CPUEfctv=8 CPUTot=8 CPULoad=0.00 AvailableFeatures=shape=c5n.2xlarge,ad=None,arch=x86_64 ActiveFeatures=shape=c5n.2xlarge,ad=None,arch=x86_64 Gres=(null) NodeAddr=many-antelope-c5n-2xlarge-0001 NodeHostName=many-antelope-c5n-2xlarge-0001 RealMemory=20034 AllocMem=0 FreeMem=N/A Sockets=1 Boards=1 State=DOWN+CLOUD+POWERED_DOWN ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=production,debug,batch,long BootTime=None SlurmdStartTime=None LastBusyTime=Unknown ResumeAfterTime=None CfgTRES=cpu=8,mem=20034M,billing=8 AllocTRES= CurrentWatts=0 AveWatts=0 Reason=Not responding [slurm@2025-01-29T17:49:26] [root@mgmt ~]# scontrol show jobs JobId=30 JobName=test.sl UserId=mwmoorcroft(1106) GroupId=nssam(1101) MCS_label=N/A Priority=1 Nice=0 Account=nssam QOS=(null) JobState=PENDING Reason=Nodes_required_for_job_are_DOWN,_DRAINED_or_reserved_for_jobs_in_higher_priority_partitions Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=01:00:00 TimeMin=N/A SubmitTime=2025-01-29T18:58:26 EligibleTime=2025-01-29T18:58:26 AccrueTime=2025-01-29T18:58:26 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-01-29T19:19:15 Scheduler=Backfill:* Partition=production AllocNode:Sid=ip-172-16-2-14.us-gov-east-1.compute.internal:7090 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1-1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=20034M,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/mnt/shared/home/mwmoorcroft/test.sl WorkDir=/mnt/shared/home/mwmoorcroft StdErr=/mnt/shared/home/mwmoorcroft/slurm-30.out StdIn=/dev/null StdOut=/mnt/shared/home/mwmoorcroft/slurm-30.out

2 2

Adding a permanent node Feature to slurm in AWS parallelcluster
by luigi＠scorzato.it 28 Jan '25

28 Jan '25

I need to add a node Feature to my SLURM AWS parallel-cluster I understand from here: https://groups.google.com/g/slurm-users/c/LgL6hq3XxzE that to add a node feature permanently I have to edit slurm.conf, because the effects of the command: scontrol update NodeName=[node(s)] AvailableFeatures=[comma separated list of features] will not persist after restart (besides, I could not find how to ADD a feature without deleting the existing AvailableFeatures). However, AWS parallelcluster slurm.conf already includes files that contain lines like: NodeName=nn77[1-100] CPUs=2 RealMemory=186777 State=CLOUD Feature=dynamic,g4dn.large,gpu Weight=1000 Gres=gpu:t4:1 which is automatically generated and should not be edited. What is the recommended way to add a feature without breaking the parallelcluster set up? My I put somewhere something like NodeName=nn77[1-100] Feature+=cascadelake ? thank you!

1 0

Automate job lmod environment setup based on node
by Daniel UB 27 Jan '25

27 Jan '25

Greetings ! We're using Slurm 23.11.8 in a small cluster . The control node shares via NFS the directory /clusterprograms with the compute nodes (using the same name as mountpoint) to provide access to available software. The users are instructed to use lmod<https://lmod.readthedocs.io/en/latest/060_locating.html> (module load/avail/purge) to setup their job's runtime environment. We have the software and modules at /clusterprograms distributed like this: /clusterprograms/ ├── common ├── gpu/ │ ├── cuda8/ │ └── cuda10/ └── cpu/ ├── intel/ └── amd/ We configured the /etc/profile.d/ directory on each node so ,when we access them directly (via ssh) the paths used by module (MODULEPATH) are only these: * /home/<user>/modulefiles * /clusterprograms/common/modules * /clusterprograms/<processor vendor or cuda version>/modules The exception is the control/login node, which only has /home/<user>/modulefiles and /clusterprograms/common/modules The issue we have now is this: We are looking for a safe and practical way to automate the reconfiguration of MODULEPATH<https://lmod.readthedocs.io/en/latest/077_ref_counting.html> in the compute nodes when a job is submitted. For now, the only method that is "working" is the following TaskProlog script, but it forces the users to load the modules inside the scripts that may be called by srun steps, rather than being able to load them in the job base step: #!/bin/bash if [[ -n "${SLURM_JOB_CONSTRAINTS}" ]]; then echo "export ORIGINAL_MPATH=${MODULEPATH}" IFS=',' read -ra JOB_ARCH_CONSTRAINTS <<< "$SLURM_JOB_CONSTRAINTS" for constraint in "${JOB_ARCH_CONSTRAINTS[@]}"; do case "${constraint}" in cuda*) echo "export MODULEPATH=/clusterprograms/gpu/${constraint}:${MODULEPATH}" ;; intel-*) echo "export MODULEPATH=/clusterprograms/cpu/intel/${constraint}:${MODULEPATH}" ;; amd-*) echo "export MODULEPATH=/clusterprograms/cpu/amd/${constraint}:${MODULEPATH}" ;; esac done fi Is there any better way to do this ? How is that it doesn't work if we use Prolog ? I tried using the --export option but it doesn't work, since the control node has MODULEPATH set via the main shell profiles. Many thanks !!! -- Daniel Garrapucho Lévy Técnic informàtic Departament de Física de la Matèria Condensada Facultat de Física Martí i Franquès, 1 08028 Barcelona, SPAIN Despatx V302 Email: daniel.garrapucho(a)ub.edu<mailto:daniel.garrapucho@ub.edu> [https://estatics.web.ub.edu/image/company_logo?img_id=2946262&t=17001439433…] Aquest missatge, i els fitxers adjunts que hi pugui haver, pot contenir informació confidencial o protegida legalment i s’adreça exclusivament a la persona o entitat destinatària. Si no consteu com a destinatari final o no teniu l’encàrrec de rebre’l, no esteu autoritzat a llegir-lo, retenir-lo, modificar-lo, distribuir-lo, copiar-lo ni a revelar-ne el contingut. Si l’heu rebut per error, informeu-ne el remitent i elimineu del sistema tant el missatge com els fitxers adjunts que hi pugui haver. Este mensaje, y los ficheros adjuntos que pueda incluir, puede contener información confidencial o legalmente protegida y está exclusivamente dirigido a la persona o entidad destinataria. Si usted no consta como destinatario final ni es la persona encargada de recibirlo, no está autorizado a leerlo, retenerlo, modificarlo, distribuirlo o copiarlo, ni a revelar su contenido. Si lo ha recibido por error, informe de ello al remitente y elimine del sistema tanto el mensaje como los ficheros adjuntos que pueda contener. This email message and any attachments it carries may contain confidential or legally protected material and are intended solely for the individual or organization to whom they are addressed. If you are not the intended recipient of this message or the person responsible for processing it, then you are not authorized to read, save, modify, send, copy or disclose any part of it. If you have received the message by mistake, please inform the sender of this and eliminate the message and any attachments it carries from your account.

1 0

Problems with slurmdbd on Ubuntu 24.04
by karl＠lovink.net 27 Jan '25

27 Jan '25

Hello, Last weekend, I updated my server to ubuntu 24.04 and instead of using my own compiled version of slurm, I switched to the version (23.11.4) that comes with Ubuntu 24.04. Since then, I have been having problems with slurmdbd. I get the following error message: slurmdbd: error: issue making get_lineage procedure slurmdbd: accounting_storage/as_mysql: init: Accounting storage MYSQL plugin failed slurmdbd: debug2: wsrep_on=0 slurmdbd: error: Couldn't load specified plugin name for accounting_storage/mysql: Plugin init() callback failed slurmdbd: error: cannot create accounting_storage context for accounting_storage/mysql slurmdbd: fatal: Unable to initialize accounting_storage/mysql accounting storage plugin My /etc/slurm/slurmdbd.conf looks like this: AuthType=auth/munge DebugLevel=debug5 DbdHost=localhost DbdPort=3306 StorageLoc=slurm_acct_db StoragePass=<password> StorageType=accounting_storage/mysql StorageUser=slurm SlurmUser=admina PluginDir=/usr/lib/x86_64-linux-gnu/slurm-wlm ArchiveEvents=yes ArchiveJobs=yes ArchiveResvs=yes ArchiveSteps=no ArchiveSuspend=no ArchiveTXN=no ArchiveUsage=no AuthInfo=/var/run/munge/munge.socket.2 PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month PurgeTXNAfter=12month PurgeUsageAfter=24month LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurmdbd.pid NOTE: Removing the PluginDir directive makes no difference. And the Plugin directory contains the accounting/MySQL file: ls -al /usr/lib/x86_64-linux-gnu/slurm-wlm/*mysql* -rw-r--r-- 1 root root 3145992 Apr 1 2024 /usr/lib/x86_64-linux-gnu/slurm-wlm/accounting_storage_mysql.a -rw-r--r-- 1 root root 1071 Apr 1 2024 /usr/lib/x86_64-linux-gnu/slurm-wlm/accounting_storage_mysql.la -rw-r--r-- 1 root root 501848 Apr 1 2024 /usr/lib/x86_64-linux-gnu/slurm-wlm/accounting_storage_mysql.so I checked already the threat on: https://groups.google.com/g/slurm-users/c/wRV8o7qqGi4 Any idea what is going on here? Regards, Karl

3 4

2025

2024

slurm-users ----- 2025 ----- May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users