- slurm-users - lists.schedmd.com

avoid using same GPU by the interactive job
by navin srivastava 13 Feb '25

13 Feb '25

hi, facing an issue in my environment where the batch job and the interactive job use the same gpu. Each server has 2 gpu. When 2 batch jobs are running it works fine and use the 2 different gpu's. but if one batch job is running and another job is submitted interactively then it uses the same GPU . Is there a way to avoid this? GresTypes=gpu NodeName=node[01-02] NodeAddr=node[01-02] CPUs=48 Boards=1 SocketsPerBoard=2 CoresPerSocket=24 ThreadsPerCore=1 TmpDisk=6000000 RealMemory=515634 Feature=A100 Gres=gpu:2 PartitionName=onprem Nodes=node[01-10] Default=YES MaxTime=21-00:00:00 DefaultTime=3-00:00:00 State=UP Shared=YES OverSubscribe=NO gres.conf: Name=gpu File=/dev/nvidia0 Name=gpu File=/dev/nvidia1 Any suggestions on this. Regards Navin

3 3

error no error
by Ricardo Román-Brenes 13 Feb '25

13 Feb '25

Hello. Could someone enlighten me as to what this error message is? Feb 13 10:02:00 gpu1 slurmd[573705]: slurmd: error: slurm_msg_sendto: address:port=192.168.9.1:36698 msg_type=8001: No error

1 0

Re: /etc/passwd sync?
by Mark W. Moorcroft 13 Feb '25

13 Feb '25

I made some progress without need for the /etc/passwd sync. Sbatch is working fine on multi-node jobs it appears. Now only salloc runs fail, which I guess is expected behavior without user account sync and ssh key setup. On bare metal OpenHPC, warewulf handles all that for me. So presumably I have MPI keys, but not user account sync and keys. Again, on OpenHPC they populated keys on account creation, and passwd was synced to the boot image. Can anyone tell me what this is about? slurmctld[46449]: slurmctld: error: failed to generate json for resume job/node list Mark Moorcroft Senior Linux Administrator Analytical Mechanics Associates

1 0

Re: /etc/passwd sync?
by Mark W. Moorcroft 12 Feb '25

12 Feb '25

The only service found in gov cloud with a search for “parallel” is Batch. And the awscli commands are absent there as well. In commercial cloud “parallel” yields many services. Mark Moorcroft Senior Linux Administrator Analytical Mechanics Associates e. mark.w.moorcroft(a)ama-inc.com<mailto:mark.w.moorcroft@ama-inc.com> w. ama-inc.com<https://ama-inc.com/> t. +1 408-290-5455 _____________________________________ Confidentiality Notice: This e-mail is intended only for the addressee named above. It contains information that is privileged, confidential or otherwise protected from use and disclosure. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, or dissemination of this transmission, or taking of any action in reliance on its contents, or other use is strictly prohibited. If you have received this transmission in error, please reply to the sender listed above immediately and permanently delete this message from your inbox. Thank you for your cooperation.

2 1

Force a refresh for sreport
by Jackson, Gary L. 10 Feb '25

10 Feb '25

As I understand it, slurmdbd will compile statistics for sreport once an hour. Is there any way I can force that to happen immediately? Restarting slurmdbd doesn’t seem to do anything. I don’t want to use this for anything operational. Just testing. -- Gary

1 0

jobs getting stuck in CG
by Ricardo Román-Brenes 10 Feb '25

10 Feb '25

Hello everyone. I have a cluster composed of 16 nodes, with 4 of them having GPUs with no particular configuration to manage them. The filesystem is gluster, authentication via slapd/munge. My problem is that very frequently, let's say at least a job daily, gets stuck in CG. I have no idea why this happens. Manually killing the slurmstep process releases the node but this is in no way a manageable solution. Has anyone experienced this (and fixed it?) Thank you. -Ricardo

4 5

MemSpecLimit Error
by Chase Schuette 07 Feb '25

07 Feb '25

After a Bright Computing/Base Command update. I'm encountering a slurm.conf error as seen below. I tried removing the MemSpecLimit parameter from the node names but the changes don't seem to be taking affect. Even after restarting slurmd across compute nodes and restarting slurmctld on the head node. I'm also suspicious of the RealMemory being set to zero. Any insight? Open to suggestions. Thanks ahead of time! sbatch: error: NodeNames=ch3lahpccn[001-032] MemSpecLimit=0 is invalid, reset to 0 sbatch: error: NodeNames=ch3lahpcgpu1 MemSpecLimit=0 is invalid, reset to 0 sbatch: error: Memory specification can not be satisfied sbatch: error: Batch job submission failed: Requested node configuration is not available Caterpillar: Confidential Green

2 1

Issue running slurm commands as normal account but work as root.
by Kent L. Hanson 05 Feb '25

05 Feb '25

Hello all! I have an odd question. On my headnode, also my login nodes, I can ping my outside network and inside network using any DNS name I choose. It is dual homed. When I run squeue, sinfo, or any slurm command as my regular account I get the below error. I can run srun, squeue, sinfo, or any slurm command and it works as expected. Does anyone know what is going on? I feel like this is a permissions issue on something. On the compute nodes I have no problem running slurm commands as my regular account or root. Everything works well on my compute nodes. Error: [user.name@headnode ~]$ squeue squeue: error: resolve_ctls_from_dns_srv: res_nsearch error: Hostname lookup failure squeue: error: fetch_config: DNS SRV lookup failed squeue: error: _establish_config_source: failed to fetch config squeue: fatal: Could not establish a configuration source Thanks! Kent Hanson

2 2

Submit multiple jobs odd info on squeue
by Vitorio Cargnini 04 Feb '25

04 Feb '25

Hi, I submit 2 or more jobs to test the scheduler and how it is handling multiple job requests and overall info. Now I submit two or more jobs I get the following odd output: [vitorio@dev35 verif_dev]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 2 sim run.sh vitorio PD 0:00 1 (InvalidAccount) 1 sim run.sh vitorio R 15:35 1 dev35 [vitorio@dev35 verif_dev]$ ^C What about this InvalidAccount ?? All subsequent jobs show up like that Regards, Vitorio

1 0

RHEL8.10 V slurmctld
by Steven Jones 04 Feb '25

04 Feb '25

I am using Redhat's IdM/IPA for users Slurmctld is failing to run jobs and it is getting "invalid user id". "2025-01-28T21:48:50.271] sched: Allocate JobId=4 NodeList=node4 #CPUs=1 Partition=debug [2025-01-28T21:48:50.280] Killing non-startable batch JobId=4: Invalid user id" id on the slurm controller works fine. [xxxjoness@xxx.ac.nz@hpcunidrslurmd2 ~]$ id xxxjoness(a)xxx.ac.nz uid=1204805830(xxxjoness(a)xxx.ac.nz) gid=1204805830(xxxjoness(a)xxx.ac.nz) groups=1204805830(xxxjoness(a)xxx.ac.nz) 8><--- Any ideas please? because I am out..... I have tried RHEL9.5, this seemed to run but srun is version 22 and on rocky8 it is version20 so fails. regards Steven

8 24

2025

2024

slurm-users ----- 2025 ----- May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users