- slurm-users - lists.schedmd.com

need to set From: address for slurm
by Vanhorn, Mike 07 Jun '24

07 Jun '24

All, When the slurm daemon is sending out emails, they are coming from “slurm(a)servername.subdomain.domain.edu<mailto:slurm@servername.subdomain.domain.edu>”. This has worked okay in the past, but due to a recent mail server change (over which I have no control whatsoever) this will no longer work. Now, the From: address is going to have to be something like “slurm-servername(a)domain.xn--edu-9o0a , or, at least something that ends in “(a)domain.xn--edu-9o0a (the subdomain being present … [View More]

2 2

Re: Not being able to ssh to node with running job
by Ratnasamy, Fritz 06 Jun '24

06 Jun '24

So the squeue issue was resolved and was due to the partition being hidden. Unhiding it solves the problem. However, the ssh issue remains (looks like both were separate issues). The pam_slurm_adopt is working on all the other nodes but not on the new ones. Any idea how to solve this? Best, *Fritz Ratnasamy* Data Scientist Information Technology On Thu, Jun 6, 2024 at 2:11 PM Ratnasamy, Fritz via slurm-users < slurm-users(a)lists.schedmd.com> wrote: > As admin on the cluster, … [View More]

1 0

Re: JWKS File Update - Reread File
by Pedro Daniel Da Rocha Santos 06 Jun '24

06 Jun '24

Thanks for the reply. I was not aware of that change in 23.11. I am still using Slurm 23.02.7. Best regards, Pedro D.R. Santos Specialist, Ph.D., High Performance Computing Vestas Technology Centre Porto Vestas Wind Systems A/S Vestas will as part of your communication, interaction and business relationship with us collect and process personal data about you. You can read more about Vestas' collection and processing of your personal data and your rights as a data subject in our Privacy … [View More]

1 0

Not being able to ssh to node with running job
by Ratnasamy, Fritz 06 Jun '24

06 Jun '24

As admin on the cluster, we do not observe any issue on our newly added gpu nodes. However, for regular users, they are not seeing their jobs running on these gpu nodes when running squeue -u <username> ( it is however showing as running status with sacct) and they are not able to ssh to these newly added nodes when they have a running job on it. I am not sure if these 2 are related (not being to ssh to mgpu node with a running job on it and not listing a job with squeue for a user on … [View More]

1 0

JWKS File Update - Reread File
by Pedro Daniel Da Rocha Santos 06 Jun '24

06 Jun '24

Hello everyone, I am currently implementing a service to monitor for jwks key rotation. If a change is detected, the jwks file is updated. Is it enough to use "scontrol reconfigure" to reread the jwks file? Or does it require a full restart of the slurmctl daemon? Thanks in advance for the inputs. Best regards, Pedro D.R. Santos Specialist, Ph.D., High Performance Computing Vestas Technology Centre Porto Vestas Wind Systems A/S +351 924 241 907 pddro(a)vestas.com<mailto:pddro@vestas.com… [View More]

2 1

diagnosing why interactive/non-interactive job waits are so long with State=MIXED
by Robert Kudyba 05 Jun '24

05 Jun '24

At the moment we have 2 nodes that are having long wait times. Generally this is when the nodes are fully allocated. What would be the other reasons if there is still enough available memory and CPU available, that a job would take so long? Slurm version is 23.02.4 via Bright Computing. Note the compute nodes have hyperthreading enabled but that should be irrelevant. Is there a way to determine what else could be holding jobs up? srun --pty -t 0-01:00:00 --nodelist=node001 --gres=gpu:1 -A … [View More]

3 5

Re: Problems with gres.conf
by Gestió Servidors 05 Jun '24

05 Jun '24

Hi, my GPU testing system (named “gpu-node”) is a simple computer with one socket and a processor " Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz". Executing "lscpu", I can see there are 4 cores per socket, 2 threads per core and 8 CPUs: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): … [View More]

1 0

Problems with gres.conf
by Gestió Servidors 04 Jun '24

04 Jun '24

Hello, I am trying to rewrite my gres.conf file. Before changes, this file was just like this: NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-1 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-2 AutoDetect=off Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Cores=12-23 NodeName=node-gpu-3 … [View More]AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-11 NodeName=node-gpu-4 AutoDetect=off Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Cores=0-7 # you can seee that nodes node-gpu-1 and node-gpu-2 have two GPUs each one, whereas nodes node-gpu-3 and node-gpu-4 have only one GPU each one And my slurmd.conf was this: [...] AccountingStorageTRES=gres/gpu GresTypes=gpu NodeName=node-gpu-1 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 CPUs=24 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 CPUs=8 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 [...] With this configuration, all seems works fine, except slurmctld.log reports: [...] error: _node_config_validate: gres/gpu: invalid GRES core specification (0-11) on node node-gpu-3 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-1 error: _node_config_validate: gres/gpu: invalid GRES core specification (12-23) on node node-gpu-2 error: _node_config_validate: gres/gpu: invalid GRES core specification (0-7) on node node-gpu-4 [...] However, even these errors, users can submit jobs and request GPUs resources. Now, I have tried to reconfigure gres.conf and slurmd.conf in this way: gres.conf: Name=gpu Type=GeForceRTX2070 File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia1 Name=gpu Type=GeForceGTX1080Ti File=/dev/nvidia0 Name=gpu Type=GeForceGTX1080 File=/dev/nvidia1 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 Name=gpu Type=GeForceRTX3080 File=/dev/nvidia0 # there is no NodeName attribute slurmd.conf: [...] NodeName=node-gpu-1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceRTX2070:1,gpu:GeForceGTX1080Ti:1 NodeName=node-gpu-2 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96000 TmpDisk=47000 Gres=gpu:GeForceGTX1080Ti:1,gpu:GeForceGTX1080:1 NodeName=node-gpu-3 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=23000 Gres=gpu:GeForceRTX3080:1 NodeName=node-gpu-4 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=7800 Gres=gpu:GeForceRTX3080:1 NodeName=node-worker-[0-22] SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=47000 # there is no CPUs attribute [...] With this new configuration, nodes with GPU start correctly slurmd.service daemon, but nodes without GPU (node-worker-[0-22]) can't start slurmd.service daemon and returns this error: [...] error: Waiting for gres.conf file /dev/nvidia0 fatal: can't stat gres.conf file /dev/nvidia0: No such file or directory [...] It seems SLURM is waiting that "node-workers" have also an nvidia GPU but not, theses nodes haven't GPU... So, where is my configuration error? I have read in https://slurm.schedmd.com/gres.conf.html about syntax and examples but it seems I'm doing some wrong. Thanks!! [View Less]

2 1

Tracking GPU utilization with sacct
by Robertson, Jordan 03 Jun '24

03 Jun '24

Hi all, I’m trying to pull (and understand) some GPU usage metrics for historical purposes, and dug into sacct’s TRES reporting a bit. We have AccountingStorageTRES=gres/gpu set in slurm.conf so we do see gres/gpuutil and gres/gpumem numbers available, but I’m struggling to find Slurm-side documentation that describes the units of these values. In looking at the code for gpu_nvml.c it seems the “nvmlDeviceGetProcessUtilization” function is being used and returns units in percentages, but I’m … [View More]

1 0

sbatch and --nodes
by Michael DiDomenico 31 May '24

31 May '24

its friday and i'm either doing something silly or have a misconfig somewhere, i can't figure out which when i run sbatch --nodes=1 --cpus-per-task=1 --array=1-100 --output test_%A_%a.txt --wrap 'uname -n' sbatch doesn't seem to be adhering to the --nodes param. when i look at my output files it's spreading them across more nodes. in the simple case above it's 50/50, but if i through a random sleep in, it'll be more. and if i expand the array it'll use even more nodes. i'm using con/tres … [View More]

2 1

2025

2024

slurm-users ----- 2025 ----- May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024 August 2024 July 2024 June 2024 May 2024 April 2024 March 2024 February 2024 January 2024

slurm-users