- slurm-users - lists.schedmd.com

Re: how to set slurmdbd.conf if using two slurmdb node with HA database?
by taleintervenor＠sjtu.edu.cn 27 Feb '25

27 Feb '25

Do you mean the second configuration scheme? I think configuring `dbdhost=localhost` is the same as configuring ` DbdAddr =nodeC` and ` DbdAddr =nodeD` on the two nodes respectively. The key point is whether we should set the DbdBackupHost option and how it work? 发件人: Daniel Letai <dani(a)letai.org.il> 发送时间: 2025年2月19日 18:21 收件人: slurm-users(a)lists.schedmd.com 主题: [slurm-users] Re: how to set slurmdbd.conf if using two slurmdb node with HA database? I'm not sure it will work, didn't test it, but could you just do `dbdhost=localhost` to solve this? On 18/02/2025 11:59, hermes via slurm-users wrote: The deployment scenario is as follows: nodeA nodeB (slurmctld) (backup slurmctld) | \-------------------------------/ | | / \ | nodeC nodeD (slurmdbd) (backup slurmdbd) (mysql) <--multi master replica--> (mysql) Since the database is multi-master replicated, the slurmdbd should only talk to the mysql on its own node. In such case, how should we set the slurmdbd.conf? The conf file contains options “DbdAddr”, “DbdHost” and “DbdBackupHost”. Should they be consistent between nodeA-2 and nodeB-2? Such as: DbdAddr = nodeC | DbdAddr = nodeC DbdHost = nodeC | DbdHost = nodeC DbdBackupHost = nodeD | DbdBackupHost = nodeD StorageHost = nodeC | StorageHost = nodeD Or maybe just set different conf and don’t use the “DbdBackupHost” like: DbdAddr = nodeC | DbdAddr = nodeD DbdHost = nodeC | DbdHost = nodeD StorageHost = nodeC | StorageHost = nodeD I’m quite confused about the usage of DbdAddr and DbdHost. What is the difference between them and why only DbdHost has the backup one? Another confusing point is how DbdBackupHost work. I guess It is slurmctld that is responsible for selecting the available slurmdbd. Since the slurm.conf already contains “AccountingStorageHost” and “AccountingStorageBackupHost”, why we need set backupdbd again on slurmdbd side?

4 10

Limit CPUs per job (but not per user, partition or node)
by Herbert Fruchtl 26 Feb '25

26 Feb '25

We have a cluster with multi-core nodes (168) that can be shared by multiple jobs at the same time. How do I configure a partition such that it only accepts jobs requesting up to (say) 8 cores, but will run multiple jobs at the same time? The following is apparently not working: PartitionName=debug Nodes=node01 MaxTime=02:00:00 DefMemPerCPU=1000 MaxCPUsPerNode=8 Default=NO It allows one job using 8 cores, but a second one will not start because the limit is apparently for the partition as a whole. Thanks in advance, Herbert -- Herbert Fruchtl (he/him) Senior Scientific Computing Officer / HPC Administrator School of Chemistry, IT Services University of St Andrews -- The University of St Andrews is a charity registered in Scotland: No SC013532

3 2

Re: MinTRES in QoS and power saving
by Patryk Bełzak 26 Feb '25

26 Feb '25

Hi, there was this issue raised some time ago: https://www.mail-archive.com/slurm-users@lists.schedmd.com/msg10799.html We're experiencing exactly the same issue now with GPU nodes in power saving, some (but not all) jobs doesn't start because of that, and it's annoying users - badly. Anyone experienced something similiar? Or maybe Stefan has a solution? (we're on version 24.05.5) Best regards Patryk Belzak. -- HPC Administrator Wroclaw Centre for Networking and Supercomputing

1 0

Shard conf weirdness
by Reed Dier 24 Feb '25

24 Feb '25

Hoping someone can help me pin down the weirdness I’m experiencing. There are actually two issues, I’ve run into, the root issue, and then something odd when trying to work around the root issue. v23.11.10 - Ubuntu 22.04 - slurm-smd debs built <https://slurm.schedmd.com/archive/slurm-23.11.10/quickstart_admin.html#debu…> from the tarball. I have 2 slurmctld daemons, and 1 slurmdbd daemon. The slurm.conf is consistent across the cluster, all ctld,dbd,slurmd, have the same shasum hash. The backup slurmctld does not appear to like the gres configured on my gpu nodes, despite the primary slurmctld having no issues. When failing over with scontrol takeover, I get the following log messages on the secondary slurmctld, where it complains about a reported, but not configured typed gres/shard. > [2025-02-21T05:02:02.017] error: Setting node $HOST2 state to INVAL with reason:gres/shard type (p100) reported but not configured > [2025-02-21T05:02:02.018] drain_nodes: node $HOST2 state set to DRAIN > [2025-02-21T05:02:02.018] error: _slurm_rpc_node_registration node=$HOST2: Invalid argument > [2025-02-21T05:02:02.020] error: Setting node $HOST3 state to INVAL with reason:gres/shard type (p40) reported but not configured > [2025-02-21T05:02:02.020] drain_nodes: node $HOST3 state set to DRAIN > [2025-02-21T05:02:02.020] error: _slurm_rpc_node_registration node=$HOST3: Invalid argument > [2025-02-21T05:02:02.023] error: Setting node $HOST1 state to INVAL with reason:gres/shard type (t4) reported but not configured > [2025-02-21T05:02:02.020] drain_nodes: node $HOST1 state set to DRAIN > [2025-02-21T05:02:02.023] error: _slurm_rpc_node_registration node=$HOST1: Invalid argument And looking at those hosts in the slurm.conf, the shards are not typed, but generic. > NodeName=$HOST1 [SNIP] State=UNKNOWN Gres=gpu:t4:2,shard:8 > NodeName=$HOST2 [SNIP] State=UNKNOWN Gres=gpu:p100:2,shard:8 > NodeName=$HOST3 [SNIP] State=UNKNOWN Gres=gpu:p40:2,gpu:p100:2,shard:16 I should also point out at this point that my gres.conf is just AutoDetect=nvml. I am not explicitly mapping any devices. So this is problem 1, the slurmctld behaving differently between daemons, where the primary has zero issue with the configuration, but the secondary balking and draining the nodes consistently. Moving on to problem 2, which is me trying to solve the issue, and running into a different issue. I decided to try to add the typed gres/shards to both the AccountingStorageTRES, and the NodeName lists. > NodeName=$HOST1 [SNIP] State=UNKNOWN Gres=gpu:t4:2,shard:t4:8 > NodeName=$HOST2 [SNIP] State=UNKNOWN Gres=gpu:p100:2,shard:p100:8 > NodeName=$HOST3 [SNIP] State=UNKNOWN Gres=gpu:p40:2,gpu:p100:2,shard:p40:8,shard:p100:8 Now the secondary slurmctld is happy and no longer immediately drains the gpu nodes for the invalid gres. However, the node ($HOST3) with a mixed set of gpus is now complaining. > [2025-02-23T21:41:05.311] gpu/nvml: _get_system_gpu_list_nvml: 4 GPU system device(s) detected > [2025-02-23T21:41:05.311] fatal: _build_shared_list: bad configuration, multiple configurations without "File" No matter what I tried, any line of the NodeName with multiple shard:$type:$num generates the _build_shared_list error above. I tried re-ordering the list as gpu,shard,gpu,shard to no avail, I tried having untyped shards, with typed shards after that, but then it complained about too many shards (double the intended number, $untyped + $typed = too many) So my current solution is to have the slurm conf in sync everywhere BUT the host(s) with multiple gpu models in the same host, where I had to revert to untyped gres on those slurmd’s, but on the slurmctlds they are typed gres. Hopefully I’ve done a decent job of explaining the corner case, enough that someone can point me in the direction of figuring out whats going on, and what is the “correct” way of doing this. I tried increasing the slurmd and slurmctld logging to debug2, to nothing that stood out beyond what was already gathered. > [2025-02-23T21:46:44.895] debug: GRES[shard] Type:p100 Count:8 Cores(88):(null) Links:(null) Flags:HAS_TYPE File:(null) UniqueId:(null) > [2025-02-23T21:46:44.895] debug: GRES[shard] Type:p40 Count:8 Cores(88):(null) Links:(null) Flags:HAS_TYPE File:(null) UniqueId:(null) > [2025-02-23T21:46:44.895] fatal: _build_shared_list: bad configuration, multiple configurations without "File" Any ideas are greatly appreciated, Reed

1 0

Plese help [CPUs=24 Boards=1 SocketsPerBoard=1 CoresPerSocket=16 ThreadsPerCore=1]
by Hugo Solís 24 Feb '25

24 Feb '25

I am installing slurm on a small cluster of 3 identical computers at the university. It turns out that I get the message when starting: error: NodeNames=hugo-big-1 CPUs=24 match no Sockets, Sockets*CoresPerSocket or Sockets*CoresPerSocket*ThreadsPerCore. Resetting CPUs. error: NodeNames=hugo-big-2 CPUs=24 match no Sockets, Sockets*CoresPerSocket or Sockets*CoresPerSocket*ThreadsPerCore. Resetting CPUs. error: NodeNames=hugo-big-3 CPUs=24 match no Sockets, Sockets*CoresPerSocket or Sockets*CoresPerSocket*ThreadsPerCore. Resetting CPUs. if I run sudo slurmd -C I get NodeName=hugo-big-1 CPUs=24 Boards=1 SocketsPerBoard=1 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=128586 reading many post it seems there is a bug starting in version 18. However I had modify the slurm.conf file NodeName=hugo-big-1 CPUs=24 Boards=1 SocketsPerBoard=1 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=128586 Gres=gpu:nvidia_geforce_rtx_3090:2 to several different version where I do not specify CPUs or sockets and I failed everytime. I do need help, any idea? warm regards Hugo Solis BTW. if I run lscpu I got that any product of Thread, cores or sockets do not result in 24. Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Vendor ID: GenuineIntel Model name: 12th Gen Intel(R) Core(TM) i9-12900K CPU family: 6 Model: 151 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 2 CPU max MHz: 5200.0000 CPU min MHz: 800.0000 BogoMIPS: 6374.40 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_ti mer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault cat_l2 cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdt_a rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_e pp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq tme rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities Virtualization features: Virtualization: VT-x Caches (sum of all): L1d: 640 KiB (16 instances) L1i: 768 KiB (16 instances) L2: 14 MiB (10 instances) L3: 30 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-23

2 2

Please help - Building Slurm-24.11.1 Failed
by Zhang, Yuan 23 Feb '25

23 Feb '25

Hello, I got errors about missing perl modules when building slurm24.11.1 rpm packages. Has anyone seen this error before? And how to fix it? Here are the error messages: -- Processing files: slurm-perlapi-24.11.1-1.el8.x86_64 error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Bitstr.pm error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Constant.pm error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Hostlist.pm error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurm/Slurm.so error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurmdb/Slurmdb.so error: File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurmdb/autosplit.ix RPM build errors: Macro expanded in comment on line 31: %_prefix path install path for commands, libraries, etc. Macro expanded in comment on line 240: %define _unpackaged_files_terminate_build 0 File listed twice: /usr/lib/.build-id/49/9394dac8b96530e1a29d0e35608ef9acedec70 File listed twice: /usr/lib/.build-id/88/4e3a854911920ae0c77c4b32d5af70dad538a9 File listed twice: /usr/lib/.build-id/c1/72dbf51fe1c17868a4e0dbf0aa55a7923f6d02 File listed twice: /usr/lib/.build-id/dc/67c93438d36986971444d218350d27b413c2cb File listed twice: /usr/lib/.build-id/ee/e79f70a62bf62e74705a75daa87d26857452ef Deprecated external dependency generator is used! File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Bitstr.pm File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Constant.pm File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/Slurm/Hostlist.pm File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurm/Slurm.so File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurmdb/Slurmdb.so File not found: /home/gdhpcgroup/yxz006/rpmbuild/BUILDROOT/slurm-24.11.1-1.el8.x86_64/usr/lib64/perl5/auto/Slurmdb/autosplit.ix -- Thanks! Yuan -- Yuan Zhang Senior HPC System Programmer Nationwide Children’s Hospital Research Institute Columbus, Ohio 43215

3 5

Running SLURM in a laptop
by Antonio Rius 19 Feb '25

19 Feb '25

For testing purposes I am using SLURM installed in a lenovo laptop running ubuntu 24-04. I am trying this configuration as part of my slurm learning process before using a real HPC facility. In my trial I want to submit an array of 9 independent processes. The CPU information of such machine is: Total Logical CPUs: 16 Physical Cores: 12 Threads per Core: 2 ================================= and the slurm script (attached) that I am using contain the sentences : #SBATCH --job-name=process_pkl #SBATCH --array=1-9 #SBATCH --output=/tmp/process_pkl_%A_%a.out #SBATCH --error=/tmp/process_pkl_%A_%a.err #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --mem=2G =================================== but in practice, using htop, only 4 processes are running concurrently, and there are other 12 CPUs without activity. My question is: What am I missing in my setup?

4 3

slurmrestd health check
by Hagdorn, Magnus Karl Moritz 19 Feb '25

19 Feb '25

Hi there, we use haproxy to distribute SLURM REST API requests to multiple instances of slurmrestd. For the haproxy we need a health check. At the moment we are just checking that we get a 401 status. This works but we are ending up with a lot of noise in the log files. It would be very nice if there was an unauthenticated REST API endpoint that can be used to check the daemon is up. Has anybody solved this issue? Refards magnus -- Dr. Magnus Hagdorn Charité – Universitätsmedizin Berlin Geschäftsbereich IT | Scientific Computing

1 0

/etc/passwd sync?
by mark.w.moorcroft＠ama-inc.com 19 Feb '25

19 Feb '25

If you set up slurm elastic cloud in EC2 without LDAP, what is the recommended method for sync of the passwd/group files? Is this necessary to get openmpi jobs to run. I would swear I had this working last week without synced passwd on two nodes. But thinking about it now I'm not sure how this could have worked. My home directories are in an NFS mount, but the user accounts don't exist on the node AMI. I'm using ansible/packer to manage the AMI's. When I ran OpenHPC / Slurm on bare metal there was a sync process. This is my first AWS Slurm cluster rodeo. I can't use the Amazon Parallel Computing tools because we are forced to be in GovCloud. I started with "ClusterInTheCloud", but it's all 4 years old, and semi-broken out of the box. My manager had me ditch a lot of it (including LDAP). So I'm building out a fork that is getting heavily modded for our situation. An ORTE daemon has unexpectedly failed after launch and before communicating back to mpirun. This could be caused by a number of factors, including an inability to create a connection back to mpirun due to a lack of common network interfaces and/or no route found between them. Please check network connectivity

4 4

how to set slurmdbd.conf if using two slurmdb node with HA database?
by taleintervenor＠sjtu.edu.cn 19 Feb '25

19 Feb '25

The deployment scenario is as follows: nodeA nodeB (slurmctld) (backup slurmctld) | \-------------------------------/ | | / \ | nodeC nodeD (slurmdbd) (backup slurmdbd) (mysql) <--multi master replica--> (mysql) Since the database is multi-master replicated, the slurmdbd should only talk to the mysql on its own node. In such case, how should we set the slurmdbd.conf? The conf file contains options "DbdAddr", "DbdHost" and "DbdBackupHost". Should they be consistent between nodeA-2 and nodeB-2? Such as: DbdAddr = nodeC | DbdAddr = nodeC DbdHost = nodeC | DbdHost = nodeC DbdBackupHost = nodeD | DbdBackupHost = nodeD StorageHost = nodeC | StorageHost = nodeD Or maybe just set different conf and don't use the "DbdBackupHost" like: DbdAddr = nodeC | DbdAddr = nodeD DbdHost = nodeC | DbdHost = nodeD StorageHost = nodeC | StorageHost = nodeD I'm quite confused about the usage of DbdAddr and DbdHost. What is the difference between them and why only DbdHost has the backup one? Another confusing point is how DbdBackupHost work. I guess It is slurmctld that is responsible for selecting the available slurmdbd. Since the slurm.conf already contains "AccountingStorageHost" and "AccountingStorageBackupHost", why we need set backupdbd again on slurmdbd side?

2 1

Run only one time on a node
by John Hearns 19 Feb '25

19 Feb '25

I am running single node tests on a cluster. I can select named nodes using the -2 flag with sbatch. However - if I want to submit perhaps 20 test jobs is there any smart way to run only one time on a node? I know I could touch a file with the hostname and test for that file. I am just wondering if there is a smarter way to do this.\ I should point out that the tests take a few minutes to run, so if a node finishes a test run it could become idle and run the tests again.

3 2

Create filenames based on slurm hosts
by John Hearns 15 Feb '25

15 Feb '25

I am working on power logging of a GPU cluster I am working with. I am running jobs on multiple hosts. I wanst to create a file , one for each host, which has a unique filename containing the host name. Something like clush -w $SLURM_JOB_NODELIST "touch file$(hostname)" My foo is weak today. Help me Ole Wan Neilsen or any $Jedi

3 3

avoid using same GPU by the interactive job
by navin srivastava 13 Feb '25

13 Feb '25

hi, facing an issue in my environment where the batch job and the interactive job use the same gpu. Each server has 2 gpu. When 2 batch jobs are running it works fine and use the 2 different gpu's. but if one batch job is running and another job is submitted interactively then it uses the same GPU . Is there a way to avoid this? GresTypes=gpu NodeName=node[01-02] NodeAddr=node[01-02] CPUs=48 Boards=1 SocketsPerBoard=2 CoresPerSocket=24 ThreadsPerCore=1 TmpDisk=6000000 RealMemory=515634 Feature=A100 Gres=gpu:2 PartitionName=onprem Nodes=node[01-10] Default=YES MaxTime=21-00:00:00 DefaultTime=3-00:00:00 State=UP Shared=YES OverSubscribe=NO gres.conf: Name=gpu File=/dev/nvidia0 Name=gpu File=/dev/nvidia1 Any suggestions on this. Regards Navin

3 3

error no error
by Ricardo Román-Brenes 13 Feb '25

13 Feb '25

Hello. Could someone enlighten me as to what this error message is? Feb 13 10:02:00 gpu1 slurmd[573705]: slurmd: error: slurm_msg_sendto: address:port=192.168.9.1:36698 msg_type=8001: No error

1 0

Re: /etc/passwd sync?
by Mark W. Moorcroft 13 Feb '25

13 Feb '25

I made some progress without need for the /etc/passwd sync. Sbatch is working fine on multi-node jobs it appears. Now only salloc runs fail, which I guess is expected behavior without user account sync and ssh key setup. On bare metal OpenHPC, warewulf handles all that for me. So presumably I have MPI keys, but not user account sync and keys. Again, on OpenHPC they populated keys on account creation, and passwd was synced to the boot image. Can anyone tell me what this is about? slurmctld[46449]: slurmctld: error: failed to generate json for resume job/node list Mark Moorcroft Senior Linux Administrator Analytical Mechanics Associates

1 0

Re: /etc/passwd sync?
by Mark W. Moorcroft 12 Feb '25

12 Feb '25

The only service found in gov cloud with a search for “parallel” is Batch. And the awscli commands are absent there as well. In commercial cloud “parallel” yields many services. Mark Moorcroft Senior Linux Administrator Analytical Mechanics Associates e. mark.w.moorcroft(a)ama-inc.com<mailto:mark.w.moorcroft@ama-inc.com> w. ama-inc.com<https://ama-inc.com/> t. +1 408-290-5455 _____________________________________ Confidentiality Notice: This e-mail is intended only for the addressee named above. It contains information that is privileged, confidential or otherwise protected from use and disclosure. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, or dissemination of this transmission, or taking of any action in reliance on its contents, or other use is strictly prohibited. If you have received this transmission in error, please reply to the sender listed above immediately and permanently delete this message from your inbox. Thank you for your cooperation.

2 1

Force a refresh for sreport
by Jackson, Gary L. 10 Feb '25

10 Feb '25

As I understand it, slurmdbd will compile statistics for sreport once an hour. Is there any way I can force that to happen immediately? Restarting slurmdbd doesn’t seem to do anything. I don’t want to use this for anything operational. Just testing. -- Gary

1 0

jobs getting stuck in CG
by Ricardo Román-Brenes 10 Feb '25

10 Feb '25

Hello everyone. I have a cluster composed of 16 nodes, with 4 of them having GPUs with no particular configuration to manage them. The filesystem is gluster, authentication via slapd/munge. My problem is that very frequently, let's say at least a job daily, gets stuck in CG. I have no idea why this happens. Manually killing the slurmstep process releases the node but this is in no way a manageable solution. Has anyone experienced this (and fixed it?) Thank you. -Ricardo

4 5

MemSpecLimit Error
by Chase Schuette 07 Feb '25

07 Feb '25

After a Bright Computing/Base Command update. I'm encountering a slurm.conf error as seen below. I tried removing the MemSpecLimit parameter from the node names but the changes don't seem to be taking affect. Even after restarting slurmd across compute nodes and restarting slurmctld on the head node. I'm also suspicious of the RealMemory being set to zero. Any insight? Open to suggestions. Thanks ahead of time! sbatch: error: NodeNames=ch3lahpccn[001-032] MemSpecLimit=0 is invalid, reset to 0 sbatch: error: NodeNames=ch3lahpcgpu1 MemSpecLimit=0 is invalid, reset to 0 sbatch: error: Memory specification can not be satisfied sbatch: error: Batch job submission failed: Requested node configuration is not available Caterpillar: Confidential Green

2 1

Issue running slurm commands as normal account but work as root.
by Kent L. Hanson 05 Feb '25

05 Feb '25

Hello all! I have an odd question. On my headnode, also my login nodes, I can ping my outside network and inside network using any DNS name I choose. It is dual homed. When I run squeue, sinfo, or any slurm command as my regular account I get the below error. I can run srun, squeue, sinfo, or any slurm command and it works as expected. Does anyone know what is going on? I feel like this is a permissions issue on something. On the compute nodes I have no problem running slurm commands as my regular account or root. Everything works well on my compute nodes. Error: [user.name@headnode ~]$ squeue squeue: error: resolve_ctls_from_dns_srv: res_nsearch error: Hostname lookup failure squeue: error: fetch_config: DNS SRV lookup failed squeue: error: _establish_config_source: failed to fetch config squeue: fatal: Could not establish a configuration source Thanks! Kent Hanson

2 2

Submit multiple jobs odd info on squeue
by Vitorio Cargnini 04 Feb '25

04 Feb '25

Hi, I submit 2 or more jobs to test the scheduler and how it is handling multiple job requests and overall info. Now I submit two or more jobs I get the following odd output: [vitorio@dev35 verif_dev]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 2 sim run.sh vitorio PD 0:00 1 (InvalidAccount) 1 sim run.sh vitorio R 15:35 1 dev35 [vitorio@dev35 verif_dev]$ ^C What about this InvalidAccount ?? All subsequent jobs show up like that Regards, Vitorio

1 0

RHEL8.10 V slurmctld
by Steven Jones 04 Feb '25

04 Feb '25

I am using Redhat's IdM/IPA for users Slurmctld is failing to run jobs and it is getting "invalid user id". "2025-01-28T21:48:50.271] sched: Allocate JobId=4 NodeList=node4 #CPUs=1 Partition=debug [2025-01-28T21:48:50.280] Killing non-startable batch JobId=4: Invalid user id" id on the slurm controller works fine. [xxxjoness@xxx.ac.nz@hpcunidrslurmd2 ~]$ id xxxjoness(a)xxx.ac.nz uid=1204805830(xxxjoness(a)xxx.ac.nz) gid=1204805830(xxxjoness(a)xxx.ac.nz) groups=1204805830(xxxjoness(a)xxx.ac.nz) 8><--- Any ideas please? because I am out..... I have tried RHEL9.5, this seemed to run but srun is version 22 and on rocky8 it is version20 so fails. regards Steven

8 24

System limits propagation
by Fatih Ertinaz 31 Jan '25

31 Jan '25

Hi everyone, I saw similar discussions in the archives, however I could not see a clear explanation therefore decided to open up a new thread. The problem I have is that default ulimits defined in the compute node OS image are overridden by Slurm and somehow changes in slurm.conf does not resolve the problem. This is specifically about max memory size, but the same issue applies to stack size and user processes as well. From the head node or from one of the compute nodes if I bypass slurm: max memory size (kbytes, -m) unlimited But going through salloc / sbatch it becomes: max memory size (kbytes, -m) 1024 As an initial attempt I tried setting `PropagateResourceLimitsExcept=NONE` but it didn't help. Then I tried `PropagateResourceLimits=ALL` also with no luck. So in theory PropagateResourceLimits should propagate all limits, but I'm not sure if that's really the case. I'm open to suggestions, Fatih Ertinaz Note: Slurm version is 24.05.1 OS is SLES SP5

2 1

Issues starting the SLURMCTLD on Rocky
by Vitorio Cargnini 31 Jan '25

31 Jan '25

Hello, I just deployed the latest SLURM and I am getting some odd issues restating it. Anyone saw this before, how can I fix it? Jan 30 17:22:09 slurmctrl01 systemd[1]: slurmctld.service: Main process exited, code=dumped, status=6/ABRT Jan 30 17:22:09 slurmctrl01 systemd[1]: slurmctld.service: Failed with result 'core-dump'. Jan 30 17:22:09 slurmctrl01 systemd[1]: Failed to start Slurm controller daemon. Jan 30 17:22:09 slurmctrl01 systemd[1]: systemd-coredump(a)8-92126-0.service: Succeeded. journalctl -xe #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92119: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92112: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92122: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92111: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92115: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92123: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) Stack trace of thread 92104: #0 0x00007f8954cf347c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007f89550c2cb5 _wait (libslurmfull.so) #2 0x00007f89550cc5cf _worker (libslurmfull.so) #3 0x00007f8954ced1ca start_thread (libpthread.so.0) #4 0x00007f8953fa28d3 __clone (libc.so.6) -- Subject: Process 92054 (slurmctld) dumped core -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- Documentation: man:core(5) -- -- Process 92054 (slurmctld) crashed and dumped core. -- -- This usually indicates a programming error in the crashing program and -- should be reported to its vendor as a bug. Jan 30 17:22:09 slurmctrl01.internal.samsung systemd[1]: slurmctld.service: Main process exited, code=dumped, status=6/ABRT Jan 30 17:22:09 slurmctrl01.internal.samsung systemd[1]: slurmctld.service: Failed with result 'core-dump'. -- Subject: Unit failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- The unit slurmctld.service has entered the 'failed' state with result 'core-dump'. Jan 30 17:22:09 slurmctrl01.internal.samsung systemd[1]: Failed to start Slurm controller daemon. -- Subject: Unit slurmctld.service has failed -- Defined-By: systemd Regards, Vitorio

2 1

Re: RHEL8.10 V slurmctld
by Steven Jones 30 Jan '25

30 Jan '25

Hi, Already done as part of the build process. regards Steven ________________________________ From: Williams, Jenny Avis <jenny_williams(a)unc.edu> Sent: Friday, 31 January 2025 9:06 am To: Steven Jones <steven.jones(a)vuw.ac.nz>; John Hearns <hearnsj(a)gmail.com> Cc: slurm-users(a)schedmd.com <slurm-users(a)schedmd.com> Subject: RE: [slurm-users] Re: RHEL8.10 V slurmctld You don't often get email from jenny_williams(a)unc.edu. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> First I’d verify munge functionality in the updated environment – https://github.com/dun/munge/wiki/Installation-Guide#troubleshooting From: Steven Jones <steven.jones(a)vuw.ac.nz> Sent: Thursday, January 30, 2025 2:55 PM To: Williams, Jenny Avis <jenny_williams(a)unc.edu>; John Hearns <hearnsj(a)gmail.com> Cc: slurm-users(a)schedmd.com Subject: Re: [slurm-users] Re: RHEL8.10 V slurmctld Hi, Hmmm, yes I am using munge. [root@node1 ~]# strings `which slurmd` |egrep -i munge [root@node1 ~]# Does not return anything on the nodes, but worked fine for RHEL9.5 [root@xxxunidrslurmd2 munge]# scontrol show config |egrep -i auth AuthAltTypes = (null) AuthAltParameters = (null) AuthInfo = (null) AuthType = auth/munge [root@vuwunidrslurmd2 munge]# Munge logs are 0 length ============= slurmd2 slurm]# rpm -qi munge Name : munge Version : 0.5.13 Release : 2.el8 Architecture: x86_64 Install Date: Wed 15 Jan 2025 02:11:46 AM UTC Group : Unspecified Size : 320124 License : GPLv3+ and LGPLv3+ Signature : RSA/SHA256, Mon 27 Apr 2020 11:43:24 PM UTC, Key ID 199e2f91fd431d51 Source RPM : munge-0.5.13-2.el8.src.rpm Build Date : Fri 24 Apr 2020 07:37:08 AM UTC Build Host : x86-vm-02.build.eng.bos.redhat.com Relocations : (not relocatable) Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : https://dun.github.io/munge/ Summary : Enables uid & gid authentication across a host cluster Description : MUNGE (MUNGE Uid 'N' Gid Emporium) is an authentication service for creating and validating credentials. It is designed to be highly scalable for use in an HPC cluster environment. It allows a process to authenticate the UID and GID of another local or remote process within a group of hosts having common users and groups. These hosts form a security realm that is defined by a shared cryptographic key. Clients within this security realm can create and validate credentials without the use of root privileges, reserved ports, or platform-specific methods. ========== [root@node1 ~]# rpm -qi munge Name : munge Version : 0.5.13 Release : 2.el8 Architecture: x86_64 Install Date: Sun Jan 26 22:14:28 2025 Group : Unspecified Size : 319876 License : GPLv3+ and LGPLv3+ Signature : RSA/SHA256, Mon Apr 12 06:46:59 2021, Key ID 15af5dac6d745a60 Source RPM : munge-0.5.13-2.el8.src.rpm Build Date : Mon Apr 12 05:07:29 2021 Build Host : ord1-prod-x86build003.svc.aws.rockylinux.org Relocations : (not relocatable) Packager : infrastructure(a)rockylinux.org<mailto:infrastructure@rockylinux.org> Vendor : Rocky URL : https://dun.github.io/munge/ Summary : Enables uid & gid authentication across a host cluster Description : MUNGE (MUNGE Uid 'N' Gid Emporium) is an authentication service for creating and validating credentials. It is designed to be highly scalable for use in an HPC cluster environment. It allows a process to authenticate the UID and GID of another local or remote process within a group of hosts having common users and groups. These hosts form a security realm that is defined by a shared cryptographic key. Clients within this security realm can create and validate credentials without the use of root privileges, reserved ports, or platform-specific methods. [root@node1 ~]# ========== [root@node1 ~]# rpm -qi slurm-slurmd Name : slurm-slurmd Version : 20.11.9 Release : 1.el8 Architecture: x86_64 Install Date: Sun Jan 26 22:15:34 2025 Group : Unspecified Size : 517922 License : GPLv2 and BSD Signature : RSA/SHA256, Fri May 6 00:03:49 2022, Key ID 21ea45ab2f86d6a1 Source RPM : slurm-20.11.9-1.el8.src.rpm Build Date : Thu May 5 23:33:30 2022 Build Host : buildvm-x86-18.iad2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://slurm.schedmd.com/ Bug URL : https://bugz.fedoraproject.org/slurm Summary : Slurm compute node daemon Description : Slurm compute node daemon. Used to launch jobs on compute nodes [root@node1 ~]# ======= slurmd2 ~]# rpm -qi slurm-slurmctld Name : slurm-slurmctld Version : 20.11.9 Release : 1.el8 Architecture: x86_64 Install Date: Wed 15 Jan 2025 02:11:48 AM UTC Group : Unspecified Size : 1097306 License : GPLv2 and BSD Signature : RSA/SHA256, Fri 06 May 2022 12:03:49 AM UTC, Key ID 21ea45ab2f86d6a1 Source RPM : slurm-20.11.9-1.el8.src.rpm Build Date : Thu 05 May 2022 11:33:30 PM UTC Build Host : buildvm-x86-18.iad2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://slurm.schedmd.com/ Bug URL : https://bugz.fedoraproject.org/slurm Summary : Slurm controller daemon Description : Slurm controller daemon. Used to manage the job queue, schedule jobs, and dispatch RPC messages to the slurmd processon the compute nodes to launch jobs. regards Steven ________________________________ From: Williams, Jenny Avis <jenny_williams(a)unc.edu<mailto:jenny_williams@unc.edu>> Sent: Friday, 31 January 2025 8:36 am To: Steven Jones <steven.jones(a)vuw.ac.nz<mailto:steven.jones@vuw.ac.nz>>; John Hearns <hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>> Cc: slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com> <slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com>> Subject: RE: [slurm-users] Re: RHEL8.10 V slurmctld You don't often get email from jenny_williams(a)unc.edu<mailto:jenny_williams@unc.edu>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> On both a compute node and the controller rpm -qi slurm-slurmctld rpm -qi slurm-slurmd check what the auth type is – for example, we still use munge, which in my compile is also the default auth type. : # strings `which slurmd` |egrep -i munge DEFAULT_AUTH_TYPE "auth/munge" DEFAULT_CRED_TYPE "cred/munge" #scontrol show config |egrep -i auth AuthAltTypes = (null) AuthAltParameters = (null) AuthInfo = (null) AuthType = auth/munge From: Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Sent: Thursday, January 30, 2025 2:07 PM To: John Hearns <hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>> Cc: slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com> Subject: [slurm-users] Re: RHEL8.10 V slurmctld Hi, Yes, even ssh works OK. [root@xxxunicobuildt1 warewulf]# ssh xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1> (xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1>) Password: Last login: Wed Jan 29 01:26:21 2025 from 130.195.87.12 [xxxjonesst@xxx.ac.nz@node1 ~]$ xxxjonesst@xxx.ac.nz@node1<mailto:xxxjonesst@xxx.ac.nz@node1> ~]$ whoami | id uid=1204805830(xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz>) gid=1204805830(xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz>) tail -f /var/log/secure ========= Jan 30 18:19:56 node1 sshd[15443]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser= rhost=130.195.87.12 user=xxxjonesst(a)xxx.ac.nz<mailto:user=xxxjonesst@xxx.ac.nz> Jan 30 18:19:56 node1 sshd[15440]: Accepted keyboard-interactive/pam for xxxjonesst(a)xxx.ac.nz<mailto:xxxjonesst@xxx.ac.nz> from 130.195.87.12 port 59402 ssh2 Would there be any relevant changes between RHEL8's slurm and RHEL9's slurm? [root@node1 ~]# rpm -qa |grep slurm slurm-libs-20.11.9-1.el8.x86_64 slurm-slurmd-20.11.9-1.el8.x86_64 slurm-20.11.9-1.el8.x86_64 [root@node1 ~]# I would have to go back and check but I do not think I hit this on RHEL9 what I did get was srun ver22 on the RHEL9 server didnt like srun ver20 on the rocky8 node. Can I compile / rpm build srun ver22 to run on rocky8? or is that part of slurmd? regards Steven ________________________________ From: John Hearns <hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>> Sent: Thursday, 30 January 2025 10:53 pm To: Steven Jones <steven.jones(a)vuw.ac.nz<mailto:steven.jones@vuw.ac.nz>> Cc: slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com> <slurm-users(a)schedmd.com<mailto:slurm-users@schedmd.com>> Subject: Re: [slurm-users] RHEL8.10 V slurmctld You don't often get email from hearnsj(a)gmail.com<mailto:hearnsj@gmail.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Have you run id on a computer node? On Wed, Jan 29, 2025, 6:47 PM Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> wrote: I am using Redhat's IdM/IPA for users Slurmctld is failing to run jobs and it is getting "invalid user id". "2025-01-28T21:48:50.271] sched: Allocate JobId=4 NodeList=node4 #CPUs=1 Partition=debug [2025-01-28T21:48:50.280] Killing non-startable batch JobId=4: Invalid user id" id on the slurm controller works fine. [xxxjoness@xxx.ac.nz@hpcunidrslurmd2 ~]$ id xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz> uid=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) gid=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) groups=1204805830(xxxjoness(a)xxx.ac.nz<mailto:xxxjoness@xxx.ac.nz>) 8><--- Any ideas please? because I am out..... I have tried RHEL9.5, this seemed to run but srun is version 22 and on rocky8 it is version20 so fails. regards Steven -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>

1 0

2025

2024

slurm-users ----- 2025 ----- April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users