- slurm-users - lists.schedmd.com

how to set slurmdbd.conf if using two slurmdb node with HA database?
by taleintervenor＠sjtu.edu.cn 19 Feb '25

19 Feb '25

The deployment scenario is as follows: nodeA nodeB (slurmctld) (backup slurmctld) | \-------------------------------/ | | / \ | nodeC nodeD (slurmdbd) (backup slurmdbd) (mysql) <--multi master replica--> (mysql) Since the database is multi-master replicated, the slurmdbd should only talk to the mysql on its own node. In such case, how should we set the slurmdbd.conf? The conf file contains options "DbdAddr", "DbdHost" and "DbdBackupHost". Should they be consistent between nodeA-2 and nodeB-2? Such as: DbdAddr = nodeC | DbdAddr = nodeC DbdHost = nodeC | DbdHost = nodeC DbdBackupHost = nodeD | DbdBackupHost = nodeD StorageHost = nodeC | StorageHost = nodeD Or maybe just set different conf and don't use the "DbdBackupHost" like: DbdAddr = nodeC | DbdAddr = nodeD DbdHost = nodeC | DbdHost = nodeD StorageHost = nodeC | StorageHost = nodeD I'm quite confused about the usage of DbdAddr and DbdHost. What is the difference between them and why only DbdHost has the backup one? Another confusing point is how DbdBackupHost work. I guess It is slurmctld that is responsible for selecting the available slurmdbd. Since the slurm.conf already contains "AccountingStorageHost" and "AccountingStorageBackupHost", why we need set backupdbd again on slurmdbd side?

2 1

Run only one time on a node
by John Hearns 19 Feb '25

19 Feb '25

I am running single node tests on a cluster. I can select named nodes using the -2 flag with sbatch. However - if I want to submit perhaps 20 test jobs is there any smart way to run only one time on a node? I know I could touch a file with the hostname and test for that file. I am just wondering if there is a smarter way to do this.\ I should point out that the tests take a few minutes to run, so if a node finishes a test run it could become idle and run the tests again.

3 2

Create filenames based on slurm hosts
by John Hearns 15 Feb '25

15 Feb '25

I am working on power logging of a GPU cluster I am working with. I am running jobs on multiple hosts. I wanst to create a file , one for each host, which has a unique filename containing the host name. Something like clush -w $SLURM_JOB_NODELIST "touch file$(hostname)" My foo is weak today. Help me Ole Wan Neilsen or any $Jedi

3 3

avoid using same GPU by the interactive job
by navin srivastava 13 Feb '25

13 Feb '25

hi, facing an issue in my environment where the batch job and the interactive job use the same gpu. Each server has 2 gpu. When 2 batch jobs are running it works fine and use the 2 different gpu's. but if one batch job is running and another job is submitted interactively then it uses the same GPU . Is there a way to avoid this? GresTypes=gpu NodeName=node[01-02] NodeAddr=node[01-02] CPUs=48 Boards=1 SocketsPerBoard=2 CoresPerSocket=24 ThreadsPerCore=1 TmpDisk=6000000 RealMemory=515634 Feature=A100 Gres=gpu:2 PartitionName=onprem Nodes=node[01-10] Default=YES MaxTime=21-00:00:00 DefaultTime=3-00:00:00 State=UP Shared=YES OverSubscribe=NO gres.conf: Name=gpu File=/dev/nvidia0 Name=gpu File=/dev/nvidia1 Any suggestions on this. Regards Navin

3 3

error no error
by Ricardo Román-Brenes 13 Feb '25

13 Feb '25

Hello. Could someone enlighten me as to what this error message is? Feb 13 10:02:00 gpu1 slurmd[573705]: slurmd: error: slurm_msg_sendto: address:port=192.168.9.1:36698 msg_type=8001: No error

1 0

Re: /etc/passwd sync?
by Mark W. Moorcroft 12 Feb '25

12 Feb '25

I made some progress without need for the /etc/passwd sync. Sbatch is working fine on multi-node jobs it appears. Now only salloc runs fail, which I guess is expected behavior without user account sync and ssh key setup. On bare metal OpenHPC, warewulf handles all that for me. So presumably I have MPI keys, but not user account sync and keys. Again, on OpenHPC they populated keys on account creation, and passwd was synced to the boot image. Can anyone tell me what this is about? slurmctld[46449]: slurmctld: error: failed to generate json for resume job/node list Mark Moorcroft Senior Linux Administrator Analytical Mechanics Associates

1 0

Re: /etc/passwd sync?
by Mark W. Moorcroft 12 Feb '25

12 Feb '25

The only service found in gov cloud with a search for “parallel” is Batch. And the awscli commands are absent there as well. In commercial cloud “parallel” yields many services. Mark Moorcroft Senior Linux Administrator Analytical Mechanics Associates e. mark.w.moorcroft(a)ama-inc.com<mailto:mark.w.moorcroft@ama-inc.com> w. ama-inc.com<https://ama-inc.com/> t. +1 408-290-5455 _____________________________________ Confidentiality Notice: This e-mail is intended only for the addressee named above. It contains information that is privileged, confidential or otherwise protected from use and disclosure. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, or dissemination of this transmission, or taking of any action in reliance on its contents, or other use is strictly prohibited. If you have received this transmission in error, please reply to the sender listed above immediately and permanently delete this message from your inbox. Thank you for your cooperation.

2 1

Force a refresh for sreport
by Jackson, Gary L. 10 Feb '25

10 Feb '25

As I understand it, slurmdbd will compile statistics for sreport once an hour. Is there any way I can force that to happen immediately? Restarting slurmdbd doesn’t seem to do anything. I don’t want to use this for anything operational. Just testing. -- Gary

1 0

jobs getting stuck in CG
by Ricardo Román-Brenes 10 Feb '25

10 Feb '25

Hello everyone. I have a cluster composed of 16 nodes, with 4 of them having GPUs with no particular configuration to manage them. The filesystem is gluster, authentication via slapd/munge. My problem is that very frequently, let's say at least a job daily, gets stuck in CG. I have no idea why this happens. Manually killing the slurmstep process releases the node but this is in no way a manageable solution. Has anyone experienced this (and fixed it?) Thank you. -Ricardo

4 5

MemSpecLimit Error
by Chase Schuette 07 Feb '25

07 Feb '25

After a Bright Computing/Base Command update. I'm encountering a slurm.conf error as seen below. I tried removing the MemSpecLimit parameter from the node names but the changes don't seem to be taking affect. Even after restarting slurmd across compute nodes and restarting slurmctld on the head node. I'm also suspicious of the RealMemory being set to zero. Any insight? Open to suggestions. Thanks ahead of time! sbatch: error: NodeNames=ch3lahpccn[001-032] MemSpecLimit=0 is invalid, reset to 0 sbatch: error: NodeNames=ch3lahpcgpu1 MemSpecLimit=0 is invalid, reset to 0 sbatch: error: Memory specification can not be satisfied sbatch: error: Batch job submission failed: Requested node configuration is not available Caterpillar: Confidential Green

2 1