- slurm-users - lists.schedmd.com

error in logs, does it mean anything?
by Steven Jones 06 Mar '25

06 Mar '25

[2025-03-06T00:26:06.002] error: _add_registered_cluster: trying to register a cluster (poc-cluster) with no remote port [2025-03-06T00:31:06.002] error: _add_registered_cluster: trying to register a cluster (poc-cluster) with no remote port [2025-03-06T00:36:06.001] error: _add_registered_cluster: trying to register a cluster (poc-cluster) with no remote port [2025-03-06T00:41:07.001] error: _add_registered_cluster: trying to register a cluster (poc-cluster) with no remote port regards Steven

1 0

Re: mariadb refusing access
by Steven Jones 05 Mar '25

05 Mar '25

Yes that is set, and my settings are, # slurmDBD info DbdAddr=localhost DbdHost=vuwunicoslurmd3.ods.vuw.ac.nz #DbdHost=localhost DbdPort=6819 SlurmUser=slurm #MessageTimeout=300 DebugLevel=verbose #DefaultQOS=normal,standby LogFile=/var/log/slurm/slurmdbd.log PidFile=/var/run/slurmdbd/slurmdbd.pid #PluginDir=/usr/lib/slurm #PrivateData=accounts,users,usage,jobs #TrackWCKey=yes # # Database info StorageType=accounting_storage/mysql #StorageHost=localhost StorageHost=localhost #StoragePort=1234 #StoragePort=3306 StoragePass=xxxxxxxx StorageUser=slurm StorageLoc=slurm_acct_db #ssj regards Steven ________________________________ From: Sarlo, Jeffrey S <JSarlo(a)Central.UH.EDU> Sent: Thursday, 6 March 2025 7:42 am To: Steven Jones <steven.jones(a)vuw.ac.nz> Subject: RE: [slurm-users] Re: mariadb refusing access You don't often get email from jsarlo(a)central.uh.edu. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Do you have the correct password listed in the slurmdbd.conf file? That was toward the bottom of the Wiki that Ole sent. Jeff From: Steven Jones via slurm-users <slurm-users(a)lists.schedmd.com> Sent: Wednesday, March 5, 2025 12:39 PM To: slurm-users(a)lists.schedmd.com; Ole Holm Nielsen <Ole.H.Nielsen(a)fysik.dtu.dk> Subject: [slurm-users] Re: mariadb refusing access Sorry, I dont follow you, Pass word works fine and grant looks OK. [root@vuwunicoslurmd3 ~]# mysql -u slurm -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 208 Server version: 10.11.10-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show grants; +--------------------------------------------------------------------------------------------------------------+ | Grants for slurm@localhost | +--------------------------------------------------------------------------------------------------------------+ | GRANT USAGE ON *.* TO `slurm`@`localhost` IDENTIFIED BY PASSWORD '*2F4EEC707189D8B5E329829B9303058EF7585569' | | GRANT ALL PRIVILEGES ON `slurm_acct_db`.* TO `slurm`@`localhost` | +--------------------------------------------------------------------------------------------------------------+ 2 rows in set (0.000 sec) MariaDB [(none)]> Isnt there something about password encryption / password hash? regards Steven ________________________________ From: Ole Holm Nielsen via slurm-users <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Sent: Wednesday, 5 March 2025 9:25 pm To: slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Subject: [slurm-users] Re: mariadb refusing access On 3/5/25 02:23, Steven Jones via slurm-users wrote: > In the logs I am seeing, > > root@vuwunicoslurmd3 mariadb]# tail -f mariadb.log > 2025-03-04 19:01:32 12565 [Warning] Access denied for user > 'slurm'@'localhost' (using password: YES) > 2025-03-04 19:06:19 12566 [Warning] Access denied for user > 'slurm'@'localhost' (using password: YES) > > However mysql -u slurm -p works just fine so it seems to be a config > error for slurmdbd It seems that you didn't select a suitable slurm user’s database password, see https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.fysi…<https://urldefense.com/v3/__https:/apc01.safelinks.protection.outlook.com/?…> IHTH, Ole -- slurm-users mailing list -- slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave(a)lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>

1 1

mariadb refusing access
by Steven Jones 05 Mar '25

05 Mar '25

In the logs I am seeing, root@vuwunicoslurmd3 mariadb]# tail -f mariadb.log 2025-03-04 19:01:32 12565 [Warning] Access denied for user 'slurm'@'localhost' (using password: YES) 2025-03-04 19:06:19 12566 [Warning] Access denied for user 'slurm'@'localhost' (using password: YES) However mysql -u slurm -p works just fine so it seems to be a config error for slurmdbd Any ideas please? regards Steven

4 6

Tracking costs - variable costs per partition
by Jeffrey A Dusenberry 05 Mar '25

05 Mar '25

Hello - We're in a similar situation as was described here: https://groups.google.com/g/slurm-users/c/eBDslkwoFio where we want to track (and control) costs on a fairly heterogenous system with different billing weights per partition. The solution proposed seems like it would work rather well, except our use of fairshare seems to interfere with the billing values we would want to use to limit usage based on credits granted. We have PriorityDecayHalfLife set on our system, so that billing value (GrpTRESRaw) seems to drop with time. Is there a way to implement something similar on an otherwise fairshare-based system? Thanks, Jeff

1 0

orted errors
by Berg, Stephen P CIV USN NRL DET SSC MS (USA) 04 Mar '25

04 Mar '25

I'm running a small-ish slurm grid, 87 nodes with various hardware. On a few occasions lately users submitting jobs will get an orted error and the job fails. Try again a few hours later or the next day and the same job runs just fine. Google-fu indicated it might be a DNS issue if for whatever reason a node couldn't figure out the address for other nodes in the job. So I populated the /etc/hosts on each node with a complete listing of all the nodes so there wouldn't be any reliance on DNS. And that very afternoon another job failed with orted. So it seems at least in my case DNS isn't the issue. What's the best way to troubleshoot this when orted fails but doesn't give any sort of error to indicate what the root cause of the failure might be? And I also can't predictably induce the failure, just have to wait until it randomly chokes.

2 1

Slurm versions 24.11.2 and 24.05.6 are now available
by Marshall Garey 04 Mar '25

04 Mar '25

We are pleased to announce the availability of Slurm versions 24.11.2 and 24.05.6. 24.11.2 fixes a variety of minor to major bugs. Fixed regressions include loading non-default QOS on pending jobs from pre-24.11 state, pending jobs displaying QOS=(null) when not explicitly requesting a QOS, running jobs that requested multiple partitions potentially having an incorrect partition when slurmctld is restarted, and burst_buffer.lua failing if slurm.conf is in a non-standard location. This release also fixes a few crashes in slurmctld: crashing when a job that can preempt requests --test-only, crasing when the scheduler evaluates a job on nodes with suspended jobs, and crashing due to a long-standing bug causing a job record without job_resrcs. 24.05.6 fixes sattach with auth/slurm, a slurmrestd crash when using data_parser/v0.0.40, a slurmctld crash when using job suspension, a performance regression for RPCs with large amounts of data, and some other moderate severity bugs. Downloads are available at https://www.schedmd.com/downloads.php . -- Marshall Garey Release Management, Support, and Development SchedMD LLC - Commercial Slurm Development and Support > * Changes in Slurm 24.11.2 > ========================== > -- Fix segfault when submitting --test-only jobs that can preempt. > -- Fix regression introduced in 23.11 that prevented the following > flags from being added to a reservation on an update: > DAILY, HOURLY, WEEKLY, WEEKDAY, and WEEKEND. > -- Fix crash and issues evaluating job's suitability for running in > nodes with already suspended job(s) there. > -- Slurmctld will ensure that healthy nodes are not reported as > UnavailableNodes in job reason codes. > -- Fix handling of jobs submitted to a current reservation with > flags OVERLAP,FLEX or OVERLAP,ANY_NODES when it overlaps nodes with a > future maintenance reservation. When a job submission had a time limit that > overlapped with the future maintenance reservation, it was rejected. Now > the job is accepted but stays pending with the reason "ReqNodeNotAvail, > Reserved for maintenance". > -- pam_slurm_adopt - avoid errors when explicitly setting > some arguments to the default value. > -- Fix qos preemption with PreemptMode=SUSPEND > -- slurmdbd - When changing a user's name update lineage > at the same time. > -- Fix regression in 24.11 in which burst_buffer.lua does not > inherit the SLURM_CONF environment variable from slurmctld and fails to run > if slurm.conf is in a non-standard location. > -- Fix memory leak in slurmctld if select/linear and the > PreemptParameters=reclaim_licenses options are both set in slurm.conf. > Regression in 24.11.1. > -- Fix running jobs, that requested multiple partitions, from > potentially being set to the wrong partition on restart. > -- switch/hpe_slingshot - Fix compatibility with newer cxi > drivers, specifically when specifying disable_rdzv_get. > -- Add ABORT_ON_FATAL environment variable to capture a backtrace > from any fatal() message. > -- Fix printing invalid address in rate limiting log statement. > -- sched/backfill - Fix node state PLANNED not being cleared from > fully allocated nodes during a backfill cycle. > -- select/cons_tres - Fix future planning of jobs with bf_licenses. > -- Prevent redundant "on_data returned rc: Rate limit exceeded, > please retry momentarily" error message from being printed in > slurmctld logs. > -- Fix loading non-default QOS on pending jobs from pre-24.11 state. > -- Fix pending jobs displaying QOS=(null) when not explicitly > requesting a QOS. > -- Fix segfault issue from job record with no job_resrcs > -- Fix failing sacctmgr delete/modify/show account operations > with where clauses. > -- Fix regression in 24.11 in which Slurm daemons started catching > several SIGTSTP, SIGTTIN and SIGUSR1 signals and ignored them, while before > they were not ignoring them. This also caused slurmctld to not being > able to shutdown after a SIGTSTP because slurmscriptd caught the signal > and stopped while slurmctld ignored it. Unify and fix these situations and > get back to the previous behavior for these signals. > -- Document that SIGQUIT is no longer ignored by slurmctld, > slurmdbd, and slurmd in 24.11. As of 24.11.0rc1, SIGQUIT is identical to > SIGINT and SIGTERM for these daemons, but this change was not documented. > -- Fix not considering nodes marked for reboot without ASAP > in the scheduler. > -- Remove the boot^ state on unexpected node reboot after > return to service. > -- Do not allow new jobs to start on a node which is being rebooted > with the flag nextstate=resume. > -- Prevent lower priority job running after cancelling an ASAP reboot. > -- Fix srun jobs starting on nextstate=resume rebooting nodes. > > * Changes in Slurm 24.05.6 > ========================== > -- data_parser/v0.0.40 - Prevent a segfault in the slurmrestd when > dumping data with v0.0.40+complex data parser. > -- Fix sattach when using auth/slurm. > -- scrun - Add support '--all' argument for kill subcommand. > -- Fix performance regression while packing larger RPCs. > -- Fix crash and issues evaluating job's suitability for running in > nodes with already suspended job(s) there. > -- Fixed a job requeuing issue that merged job entries into the > same SLUID when all nodes in a job failed simultaneously. > -- switch/hpe_slingshot - Fix compatibility with newer cxi > drivers, specifically when specifying disable_rdzv_get. > -- Add ABORT_ON_FATAL environment variable to capture a backtrace > from any fatal() message.

7 6

Re: how to set slurmdbd.conf if using two slurmdb node with HA database?
by taleintervenor＠sjtu.edu.cn 27 Feb '25

27 Feb '25

Do you mean the second configuration scheme? I think configuring `dbdhost=localhost` is the same as configuring ` DbdAddr =nodeC` and ` DbdAddr =nodeD` on the two nodes respectively. The key point is whether we should set the DbdBackupHost option and how it work? 发件人: Daniel Letai <dani(a)letai.org.il> 发送时间: 2025年2月19日 18:21 收件人: slurm-users(a)lists.schedmd.com 主题: [slurm-users] Re: how to set slurmdbd.conf if using two slurmdb node with HA database? I'm not sure it will work, didn't test it, but could you just do `dbdhost=localhost` to solve this? On 18/02/2025 11:59, hermes via slurm-users wrote: The deployment scenario is as follows: nodeA nodeB (slurmctld) (backup slurmctld) | \-------------------------------/ | | / \ | nodeC nodeD (slurmdbd) (backup slurmdbd) (mysql) <--multi master replica--> (mysql) Since the database is multi-master replicated, the slurmdbd should only talk to the mysql on its own node. In such case, how should we set the slurmdbd.conf? The conf file contains options “DbdAddr”, “DbdHost” and “DbdBackupHost”. Should they be consistent between nodeA-2 and nodeB-2? Such as: DbdAddr = nodeC | DbdAddr = nodeC DbdHost = nodeC | DbdHost = nodeC DbdBackupHost = nodeD | DbdBackupHost = nodeD StorageHost = nodeC | StorageHost = nodeD Or maybe just set different conf and don’t use the “DbdBackupHost” like: DbdAddr = nodeC | DbdAddr = nodeD DbdHost = nodeC | DbdHost = nodeD StorageHost = nodeC | StorageHost = nodeD I’m quite confused about the usage of DbdAddr and DbdHost. What is the difference between them and why only DbdHost has the backup one? Another confusing point is how DbdBackupHost work. I guess It is slurmctld that is responsible for selecting the available slurmdbd. Since the slurm.conf already contains “AccountingStorageHost” and “AccountingStorageBackupHost”, why we need set backupdbd again on slurmdbd side?

4 10

Limit CPUs per job (but not per user, partition or node)
by Herbert Fruchtl 26 Feb '25

26 Feb '25

We have a cluster with multi-core nodes (168) that can be shared by multiple jobs at the same time. How do I configure a partition such that it only accepts jobs requesting up to (say) 8 cores, but will run multiple jobs at the same time? The following is apparently not working: PartitionName=debug Nodes=node01 MaxTime=02:00:00 DefMemPerCPU=1000 MaxCPUsPerNode=8 Default=NO It allows one job using 8 cores, but a second one will not start because the limit is apparently for the partition as a whole. Thanks in advance, Herbert -- Herbert Fruchtl (he/him) Senior Scientific Computing Officer / HPC Administrator School of Chemistry, IT Services University of St Andrews -- The University of St Andrews is a charity registered in Scotland: No SC013532

3 2

Re: MinTRES in QoS and power saving
by Patryk Bełzak 26 Feb '25

26 Feb '25

Hi, there was this issue raised some time ago: https://www.mail-archive.com/slurm-users@lists.schedmd.com/msg10799.html We're experiencing exactly the same issue now with GPU nodes in power saving, some (but not all) jobs doesn't start because of that, and it's annoying users - badly. Anyone experienced something similiar? Or maybe Stefan has a solution? (we're on version 24.05.5) Best regards Patryk Belzak. -- HPC Administrator Wroclaw Centre for Networking and Supercomputing

1 0

Shard conf weirdness
by Reed Dier 24 Feb '25

24 Feb '25

Hoping someone can help me pin down the weirdness I’m experiencing. There are actually two issues, I’ve run into, the root issue, and then something odd when trying to work around the root issue. v23.11.10 - Ubuntu 22.04 - slurm-smd debs built <https://slurm.schedmd.com/archive/slurm-23.11.10/quickstart_admin.html#debu…> from the tarball. I have 2 slurmctld daemons, and 1 slurmdbd daemon. The slurm.conf is consistent across the cluster, all ctld,dbd,slurmd, have the same shasum hash. The backup slurmctld does not appear to like the gres configured on my gpu nodes, despite the primary slurmctld having no issues. When failing over with scontrol takeover, I get the following log messages on the secondary slurmctld, where it complains about a reported, but not configured typed gres/shard. > [2025-02-21T05:02:02.017] error: Setting node $HOST2 state to INVAL with reason:gres/shard type (p100) reported but not configured > [2025-02-21T05:02:02.018] drain_nodes: node $HOST2 state set to DRAIN > [2025-02-21T05:02:02.018] error: _slurm_rpc_node_registration node=$HOST2: Invalid argument > [2025-02-21T05:02:02.020] error: Setting node $HOST3 state to INVAL with reason:gres/shard type (p40) reported but not configured > [2025-02-21T05:02:02.020] drain_nodes: node $HOST3 state set to DRAIN > [2025-02-21T05:02:02.020] error: _slurm_rpc_node_registration node=$HOST3: Invalid argument > [2025-02-21T05:02:02.023] error: Setting node $HOST1 state to INVAL with reason:gres/shard type (t4) reported but not configured > [2025-02-21T05:02:02.020] drain_nodes: node $HOST1 state set to DRAIN > [2025-02-21T05:02:02.023] error: _slurm_rpc_node_registration node=$HOST1: Invalid argument And looking at those hosts in the slurm.conf, the shards are not typed, but generic. > NodeName=$HOST1 [SNIP] State=UNKNOWN Gres=gpu:t4:2,shard:8 > NodeName=$HOST2 [SNIP] State=UNKNOWN Gres=gpu:p100:2,shard:8 > NodeName=$HOST3 [SNIP] State=UNKNOWN Gres=gpu:p40:2,gpu:p100:2,shard:16 I should also point out at this point that my gres.conf is just AutoDetect=nvml. I am not explicitly mapping any devices. So this is problem 1, the slurmctld behaving differently between daemons, where the primary has zero issue with the configuration, but the secondary balking and draining the nodes consistently. Moving on to problem 2, which is me trying to solve the issue, and running into a different issue. I decided to try to add the typed gres/shards to both the AccountingStorageTRES, and the NodeName lists. > NodeName=$HOST1 [SNIP] State=UNKNOWN Gres=gpu:t4:2,shard:t4:8 > NodeName=$HOST2 [SNIP] State=UNKNOWN Gres=gpu:p100:2,shard:p100:8 > NodeName=$HOST3 [SNIP] State=UNKNOWN Gres=gpu:p40:2,gpu:p100:2,shard:p40:8,shard:p100:8 Now the secondary slurmctld is happy and no longer immediately drains the gpu nodes for the invalid gres. However, the node ($HOST3) with a mixed set of gpus is now complaining. > [2025-02-23T21:41:05.311] gpu/nvml: _get_system_gpu_list_nvml: 4 GPU system device(s) detected > [2025-02-23T21:41:05.311] fatal: _build_shared_list: bad configuration, multiple configurations without "File" No matter what I tried, any line of the NodeName with multiple shard:$type:$num generates the _build_shared_list error above. I tried re-ordering the list as gpu,shard,gpu,shard to no avail, I tried having untyped shards, with typed shards after that, but then it complained about too many shards (double the intended number, $untyped + $typed = too many) So my current solution is to have the slurm conf in sync everywhere BUT the host(s) with multiple gpu models in the same host, where I had to revert to untyped gres on those slurmd’s, but on the slurmctlds they are typed gres. Hopefully I’ve done a decent job of explaining the corner case, enough that someone can point me in the direction of figuring out whats going on, and what is the “correct” way of doing this. I tried increasing the slurmd and slurmctld logging to debug2, to nothing that stood out beyond what was already gathered. > [2025-02-23T21:46:44.895] debug: GRES[shard] Type:p100 Count:8 Cores(88):(null) Links:(null) Flags:HAS_TYPE File:(null) UniqueId:(null) > [2025-02-23T21:46:44.895] debug: GRES[shard] Type:p40 Count:8 Cores(88):(null) Links:(null) Flags:HAS_TYPE File:(null) UniqueId:(null) > [2025-02-23T21:46:44.895] fatal: _build_shared_list: bad configuration, multiple configurations without "File" Any ideas are greatly appreciated, Reed

1 0