- slurm-users - lists.schedmd.com

Re: sbatch: Node count specification invalid - when only specifying --ntasks
by George Leaver 10 Jun '24

10 Jun '24

Noam, Thanks for the suggestion but no luck: sbatch -p multinode -n 80 --ntasks-per-core=1 --wrap="..." sbatch: error: Batch job submission failed: Node count specification invalid sbatch -p multinode -n 2 -c 40 --ntasks-per-core=1 --wrap="..." sbatch: error: Batch job submission failed: Node count specification invalid sbatch -p multinode -N 2 -n 80 --ntasks-per-core=1 --wrap="..." Submitted batch job I guess that the MinNodes=2 in the partition def is now being enforced somewhat more strictly, or earlier in the submission process, before it can be determined that the request will satisfy the constraint. Regards, George -- George Leaver Research Infrastructure, IT Services, University of Manchester http://ri.itservices.manchester.ac.uk | @UoM_eResearch ________________________________________ From: Bernstein, Noam CIV USN NRL WASHINGTON DC (USA) <noam.bernstein.civ(a)us.navy.mil> Sent: 09 June 2024 19:33 To: George Leaver; slurm-users(a)lists.schedmd.com Subject: Re: sbatch: Node count specification invalid - when only specifying --ntasks It would be a shame to lose this capability. Have you tried adding `--ntasks-per-core` explicitly (but not number of nodes)? Noam

1 0

cpu distribution question
by Alan Stange 08 Jun '24

08 Jun '24

All, I have a very simple slurm cluster. It's just a single system with 2 sockets and 16 cores in each socket. I would like to be able to submit a simple task into this cluster, and to have the cpus assigned to that task allocated round robin across the two sockets. Everything I try is putting all the cpus for this single task on the same socket. I have not specified any CpuBind options in the slurm.conf file. For example, if I try $ srun -c 4 --pty bash I get a shell prompt on the system, and can run $ taskset -cp $$ pid 12345 current affinity list: 0,2,4,6 and I get this same set of cpus no matter what options I try (the cluster is idle with no tasks consuming slots). I've tried various srun command line options like: --hint=compute_bound --hint=memory_bound various --cpubind options -B 2:2 -m block:cyclic and block:fcyclic Note that if I try to allocation 17 cpus, then I do get the 17th cpu allocated on the 2nd socket. What magic incantation is needed to get an allocation where the cpus are chosen round robin across the sockets? Thank you! Alan

3 3

maxrss reported by sachet is wrong
by Feng Zhang 07 Jun '24

07 Jun '24

Hi All, I am having trouble calculating the real RSS memory usage by some kind of users' jobs. Which the sacct returned wrong numbers. Rocky Linux release 8.5, Slurm 21.08 (slurm.conf) ProctrackType=proctrack/cgroup JobAcctGatherType=jobacct_gather/linux The troubling jobs are like: 1. python spawn multithreading 96 threads; 2. Each thread uses SKlearn which again spawns 96 threads using openmp. Which is obviously over running the node, and I want to address it. The node has 300GB RAM, but the "sacct" (and seff) reports 1.2TB MaxRSS(also AveRSS). This does not look correct. I am suspecting that whether the SLurm+jobacct_gather/linux repeatedly sums up the memory used by all these threads, multiple counted the same thing many times. For the openMP part, maybe it is fine for slurm; while for python/multithreading, maybe it can not work well with Slurm for memory accounting? So, if this is the case, maybe 1.2TB/96= 12GB MaxRSS? I want to get the right MaxRSS to report to users. Thanks! Best, Feng

2 1

need to set From: address for slurm
by Vanhorn, Mike 07 Jun '24

07 Jun '24

All, When the slurm daemon is sending out emails, they are coming from “slurm(a)servername.subdomain.domain.edu<mailto:slurm@servername.subdomain.domain.edu>”. This has worked okay in the past, but due to a recent mail server change (over which I have no control whatsoever) this will no longer work. Now, the From: address is going to have to be something like “slurm-servername(a)domain.xn--edu-9o0a , or, at least something that ends in “(a)domain.xn--edu-9o0a (the subdomain being present will cause it to get rejected by the mail server. I am not seeing in the documentation how to change the “From:” address tha slurm uses. Is there a way to do this and I’m just missing it? --- Mike VanHorn Senior Computer Systems Administrator College of Engineering and Computer Science Wright State University 265 Russ Engineering Center 937-775-5157 michael.vanhorn(a)wright.edu

2 2

Re: Not being able to ssh to node with running job
by Ratnasamy, Fritz 06 Jun '24

06 Jun '24

So the squeue issue was resolved and was due to the partition being hidden. Unhiding it solves the problem. However, the ssh issue remains (looks like both were separate issues). The pam_slurm_adopt is working on all the other nodes but not on the new ones. Any idea how to solve this? Best, *Fritz Ratnasamy* Data Scientist Information Technology On Thu, Jun 6, 2024 at 2:11 PM Ratnasamy, Fritz via slurm-users < slurm-users(a)lists.schedmd.com> wrote: > As admin on the cluster, we do not observe any issue on our newly added > gpu nodes. > However, for regular users, they are not seeing their jobs running on > these gpu nodes when running squeue -u <username> ( it is > however showing as running status with sacct) and they are not able to ssh > to these newly added nodes when they have a running job on it. > I am not sure if these 2 are related (not being to ssh to mgpu node with > a running job on it and not listing a job with squeue for a user on the > same node). There are no issues reported on the other nodes. Anyone know > what is happening? > Best, > > *Fritz Ratnasamy* > > Data Scientist > > Information Technology > > > CAUTION: This email has originated outside of University email systems. > Please do not click links or open attachments unless you recognize the > sender and trust the contents as safe. > >

1 0

Re: JWKS File Update - Reread File
by Pedro Daniel Da Rocha Santos 06 Jun '24

06 Jun '24

Thanks for the reply. I was not aware of that change in 23.11. I am still using Slurm 23.02.7. Best regards, Pedro D.R. Santos Specialist, Ph.D., High Performance Computing Vestas Technology Centre Porto Vestas Wind Systems A/S Vestas will as part of your communication, interaction and business relationship with us collect and process personal data about you. You can read more about Vestas' collection and processing of your personal data and your rights as a data subject in our Privacy Policy<https://www.vestas.com/en/about/profile/privacy-policy>. Classification: Restricted

1 0

Not being able to ssh to node with running job
by Ratnasamy, Fritz 06 Jun '24

06 Jun '24

As admin on the cluster, we do not observe any issue on our newly added gpu nodes. However, for regular users, they are not seeing their jobs running on these gpu nodes when running squeue -u <username> ( it is however showing as running status with sacct) and they are not able to ssh to these newly added nodes when they have a running job on it. I am not sure if these 2 are related (not being to ssh to mgpu node with a running job on it and not listing a job with squeue for a user on the same node). There are no issues reported on the other nodes. Anyone know what is happening? Best, *Fritz Ratnasamy* Data Scientist Information Technology

1 0

JWKS File Update - Reread File
by Pedro Daniel Da Rocha Santos 06 Jun '24

06 Jun '24

Hello everyone, I am currently implementing a service to monitor for jwks key rotation. If a change is detected, the jwks file is updated. Is it enough to use "scontrol reconfigure" to reread the jwks file? Or does it require a full restart of the slurmctl daemon? Thanks in advance for the inputs. Best regards, Pedro D.R. Santos Specialist, Ph.D., High Performance Computing Vestas Technology Centre Porto Vestas Wind Systems A/S +351 924 241 907 pddro(a)vestas.com<mailto:pddro@vestas.com> http://www.vestas.com<http://www.vestas.com/> Company reg. name: Vestas Wind Systems A/S This e-mail is subject to our e-mail disclaimer statement. Please refer to www.vestas.com/legal/notice<http://www.vestas.com/legal/notice> If you have received this e-mail in error please contact the sender. Vestas will as part of your communication, interaction and business relationship with us collect and process personal data about you. You can read more about Vestas' collection and processing of your personal data and your rights as a data subject in our Privacy Policy<https://www.vestas.com/en/about/profile/privacy-policy>. Classification: Restricted

2 1

diagnosing why interactive/non-interactive job waits are so long with State=MIXED
by Robert Kudyba 05 Jun '24

05 Jun '24

At the moment we have 2 nodes that are having long wait times. Generally this is when the nodes are fully allocated. What would be the other reasons if there is still enough available memory and CPU available, that a job would take so long? Slurm version is 23.02.4 via Bright Computing. Note the compute nodes have hyperthreading enabled but that should be irrelevant. Is there a way to determine what else could be holding jobs up? srun --pty -t 0-01:00:00 --nodelist=node001 --gres=gpu:1 -A ourts -p short /bin/bash srun: job 672204 queued and waiting for resources scontrol show node node001 NodeName=m001 Arch=x86_64 CoresPerSocket=48 CPUAlloc=24 CPUEfctv=192 CPUTot=192 CPULoad=20.37 AvailableFeatures=location=local ActiveFeatures=location=local Gres=gpu:A6000:8 NodeAddr=node001 NodeHostName=node001 Version=23.02.4 OS=Linux 5.14.0-70.13.1.el9_0.x86_64 #1 SMP PREEMPT Thu Apr 14 12:42:38 EDT 2022 RealMemory=1031883 AllocMem=1028096 FreeMem=222528 Sockets=2 Boards=1 State=MIXED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=ours,short BootTime=2024-04-29T16:18:30 SlurmdStartTime=2024-05-18T16:48:11 LastBusyTime=2024-06-03T10:49:49 ResumeAfterTime=None CfgTRES=cpu=192,mem=1031883M,billing=192,gres/gpu=8 AllocTRES=cpu=24,mem=1004G,gres/gpu=2,gres/gpu:a6000=2 CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s grep 672204 /var/log/slurmctld [2024-06-04T15:50:35.627] sched: _slurm_rpc_allocate_resources JobId=672204 NodeList=(null) usec=852

3 5

Re: Problems with gres.conf
by Gestió Servidors 05 Jun '24

05 Jun '24

Hi, my GPU testing system (named “gpu-node”) is a simple computer with one socket and a processor " Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz". Executing "lscpu", I can see there are 4 cores per socket, 2 threads per core and 8 CPUs: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 26 Model name: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz My “gres.conf” file is: NodeName=gpu-node Name=gpu Type=GeForce-GTX-TITAN-X File=/dev/nvidia0 CPUs=0-1 NodeName=gpu-node Name=gpu Type=GeForce-GTX-TITAN-Black File=/dev/nvidia1 CPUs=2-3 Running “numactl -H” in “gpu-node” host, reports: available: 1 nodes (0) node 0 cpus: 0 1 2 3 4 5 6 7 node 0 size: 7809 MB node 0 free: 6597 MB node distances: node 0 0: 10 CPUs are assigned 0-1 for first GPU and 2-3 for second GPU. However, “lscpu” shows 8 CPUs… If I rewrite “gres.conf” in this way: NodeName=gpu-node Name=gpu Type=GeForce-GTX-TITAN-X File=/dev/nvidia0 CPUs=0-3 NodeName=gpu-node Name=gpu Type=GeForce-GTX-TITAN-Black File=/dev/nvidia1 CPUs=4-7 when I run “scontrol reconfigure”, slurmctld log reports this error message: [2024-06-05T11:42:18.558] error: _node_config_validate: gres/gpu: invalid GRES core specification (4-7) on node gpu-node So I think SLURM only can get physical cores and not threads, so my node only can serve 4 cores (in “lspcu”) but in gres.conf I need to write “CPUs”, not “Cores”… isn’t it? But if “numactl -H” shows 8 CPUs, why I can use this 8 CPUs in “gres.conf”? Sorry about this large email. Thanks.

1 0