- slurm-users - lists.schedmd.com

Tried setting up GANG scheduling for timeslicing, but jobs are not alternating
by Francisco José Letterio 22 Jan '24

22 Jan '24

I'm trying to set up GANG scheduling with Slurm on my single-node server so the people at the lab can run experiments without blocking each other (so if say someone has to run some code that takes days to finish, other jobs that take less have the chance to run alternated with it and so they don't have to wait days until they can run) I followed the GANG scheduling slurm.conf setup tutorial on the slurm website, and to check if it's working properly I launched a bunch of jobs that print the current time and then sleep for a while. But when I check squeue, the jobs never alternate, they run sequentially one after the other. Why is this happening? Here's my slurm.conf file: # See the slurm.conf man page for more information. ClusterName=localcluster SlurmctldHost=localhost ProctrackType=proctrack/linuxproc ReturnToService=1 SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/lib/slurm/slurmd SlurmUser=slurm StateSaveLocation=/var/lib/slurm/slurmctld TaskPlugin=task/none # TIMERS InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 # SCHEDULING #Seteado para que cada laburo alterne cada 15 segundos SchedulerTimeSlice=15 SchedulerType=sched/builtin SelectType=select/linear SelectTypeParameters=CR_Memory PreemptMode=GANG # LOGGING AND ACCOUNTING JobCompType=jobcomp/none JobAcctGatherFrequency=30 SlurmctldDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurm/slurmd.log # COMPUTE NODES NodeName=lab04 CPUs=48 CoresPerSocket=12 ThreadsPerCore=2 State=UNKNOWN RealMemory=257249 PartitionName=LocalQ Nodes=ALL Default=YES MaxTime=INFINITE State=UP OverSubscribe=FORCE:6 DefMemPerNode=257249 MaxMemPerNode=257249 >From what I understand, SchedulerTimeSlice=15 means that jobs should alternate running every 15 seconds. This is the job I'm launching with sbatch (launching many copies of this job one after the other): #!/bin/bash #SBATCH -J test # Job name #SBATCH -o job.%j.out # Name of stdout output file (%j expands to %jobId) #SBATCH -N 1 # Total number of nodes requested echo "Test output from Slurm Testjob" date sleep 10 date sleep 10 date sleep 10 date sleep 10 I would expect jobs to print one or two dates, then the Slurm scheduler comes and lets another job run in the meantime, and then the final print(s) come with a delay way greater than 10secs However after launching many copies of this job this is what I see on squeue : Before the first job launched is done: JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 18 LocalQ test eiarussi PD 0:00 1 (Resources) 19 LocalQ test eiarussi PD 0:00 1 (Priority) 20 LocalQ test eiarussi PD 0:00 1 (Priority) 21 LocalQ test eiarussi PD 0:00 1 (Priority) 17 LocalQ test eiarussi R 0:31 1 lab04 After that job ends: JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 19 LocalQ test eiarussi PD 0:00 1 (Resources) 20 LocalQ test eiarussi PD 0:00 1 (Priority) 21 LocalQ test eiarussi PD 0:00 1 (Priority) 18 LocalQ test eiarussi R 0:02 1 lab04 Why did the first job run for 30+ seconds uninterrupted, instead of another job being allowed to run after 15 seconds? Am I misunderstanding how GANG scheduling works, or is the a problem in my conf file?

1 0

MIG-Slice: Unavailable GRES
by Dražen Jalšovec 21 Jan '24

21 Jan '24

Hi, We are testing the MIG deployment on our new slurm compute node with 4 x H100 GPUs. It looks like everything is configured correctly but we have a problem accessing mig devices. When I submit jobs requesting a mig gpu device #SBATCH --gres=gpu:H100_1g.10gb:1, the jobs get submitted to the node, but only 4 jobs get executed and all other jobs fail. I was able to "solve it" by adding code to submit to the mig device UUID. Am I missing something in the slurm/gres/cgroups configuration that would automatically assign correct CUDA_VISIBLE_DEVICES MIG UUID to the job? If I submit 10 jobs requesting --gres=gpu:H100_1g.10gb:1 8 jobs start running: #1 - CUDA_VISIBLE_DEVICES=0 - jobs 1-4 run on one of the mig devices configured on GPU-device ... #8 - CUDA_VISIBLE_DEVICES=7 - jobs 5-8 fail with the message that there are no available CUDA_VISIBLE_DEVICES ... #10 - CUDA_VISIBLE_DEVICES=? - after waiting for free mig resources jobs 9-10 runs or fails depending if GPU-device is free (migs are free) $ nvidia-smi --query-gpu=uuid --id=0 --format=csv,noheader GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3 $ nvidia-smi --query-gpu=uuid --id=7 --format=csv,noheader No devices were found The slurm assigns CUDA_VISIBLE_DEVICES numbers based on the requested resources: Request: gpu:H100_1g.10gb:1 - CUDA_VISIBLE_DEVICES=0 gpu:H100_1g.10gb:2 - CUDA_VISIBLE_DEVICES=0,1 ... Any job with H100_1g.10gb request is going to get device number between 0 and 7 gpu:H100_1g.10gb:8 - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 gpu:H100_2g.20gb:1 - CUDA_VISIBLE_DEVICES=8 Any job with H100_2g.20gb request is going to get device number between 8 and 10 ... gpu:H100_2g.10gb:3 - CUDA_VISIBLE_DEVICES=8,9,10 gpu:H100_3g.40gb:1 - CUDA_VISIBLE_DEVICES=11 gpu:H100_3g.40gb:2 - CUDA_VISIBLE_DEVICES=12 gpu:H100_7g.80gb:1 - CUDA_VISIBLE_DEVICES=13 I was able to "solve it" by adding this code to my slurm script: # Get the CUDA index using srun and awk CUDA_INDEX=$(srun env | grep CUDA_VISIBLE_DEVICES | awk -F '=' '{print $2}') ## GPU0 1g.10gb ## MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a ## MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664 ## MIG-d3ab0675-d318-5e53-b487-b50695cf2e00 ## MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0 ## MIG-85dd76da-c994-5830-adf0-467c66ae1b95 ## MIG-29a6d43b-882e-5b79-868a-15bb2c770b82 ## MIG-558debd2-dc13-5406-9256-73ef4f279737 ## GPU1 2g.20gb ## MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f ## MIG-0a804c0c-aa27-5993-97cb-cedb854735ce ## MIG-8af9af6d-8720-5763-81e2-83afc43eb42b ## GPU1 1g.10gb ## MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a ## GPU2 3g.40gb ## MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b ## MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc ## GPU3 7g.80gb ## MIG-ec043869-9176-577e-bac0-46c8411e4e37 # Define the list of UUIDs UUIDS=( "MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a" "MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664" "MIG-d3ab0675-d318-5e53-b487-b50695cf2e00" "MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0" "MIG-85dd76da-c994-5830-adf0-467c66ae1b95" "MIG-29a6d43b-882e-5b79-868a-15bb2c770b82" "MIG-558debd2-dc13-5406-9256-73ef4f279737" "MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a" "MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f" "MIG-0a804c0c-aa27-5993-97cb-cedb854735ce" "MIG-8af9af6d-8720-5763-81e2-83afc43eb42b" "MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b" "MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc" "MIG-ec043869-9176-577e-bac0-46c8411e4e37" ) # Assign the UUID based on the CUDA index SELECTED_UUID=${UUIDS[$CUDA_INDEX]} # Print the selected UUID echo $SELECTED_UUID export CUDA_VISIBLE_DEVICES=$SELECTED_UUID echo "CUDA_VISIBLE_DEVICES set to: "$SELECTED_UUID echo "Test: "$CUDA_VISIBLE_DEVICES CUDA_VISIBLE_DEVICES=$SELECTED_UUID python3.8 gpu_script.py Result: Script start MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a CUDA_VISIBLE_DEVICES set to: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a Matrix calculation on CUDA device completed successfully. Script end +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 7 0 28686 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 8 0 28744 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 9 0 28300 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 10 0 28512 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 11 0 28506 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 12 0 28516 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 13 0 28511 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 1 9 0 28552 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | +---------------------------------------------------------------------------------------+ This is the current setup: RockyLinux 9.2 slurm-22.05.9-1.el9.x86_64 CUDA Version: 12.1 Driver Version: 530.30.02 # nvidia-smi -L GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3) MIG 1g.10gb Device 0: (UUID: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a) MIG 1g.10gb Device 1: (UUID: MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664) MIG 1g.10gb Device 2: (UUID: MIG-d3ab0675-d318-5e53-b487-b50695cf2e00) MIG 1g.10gb Device 3: (UUID: MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0) MIG 1g.10gb Device 4: (UUID: MIG-85dd76da-c994-5830-adf0-467c66ae1b95) MIG 1g.10gb Device 5: (UUID: MIG-29a6d43b-882e-5b79-868a-15bb2c770b82) MIG 1g.10gb Device 6: (UUID: MIG-558debd2-dc13-5406-9256-73ef4f279737) GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1439a39c-948b-0657-98e9-aff8595a8729) MIG 2g.20gb Device 0: (UUID: MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f) MIG 2g.20gb Device 1: (UUID: MIG-0a804c0c-aa27-5993-97cb-cedb854735ce) MIG 2g.20gb Device 2: (UUID: MIG-8af9af6d-8720-5763-81e2-83afc43eb42b) MIG 1g.10gb Device 3: (UUID: MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a) GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-3b182fed-bfdf-bc11-d1bd-c41fd1468d2a) MIG 3g.40gb Device 0: (UUID: MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b) MIG 3g.40gb Device 1: (UUID: MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc) GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-7eba9d78-e908-3db8-2633-a269aeec395e) MIG 7g.80gb Device 0: (UUID: MIG-ec043869-9176-577e-bac0-46c8411e4e37) $ scontrol show node node16 NodeName=node16 Arch=x86_64 CoresPerSocket=1 CPUAlloc=1 CPUEfctv=112 CPUTot=112 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 NodeAddr=node16 NodeHostName=node16 Version=22.05.9 OS=Linux 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023 RealMemory=1030000 AllocMem=0 FreeMem=1011801 Sockets=112 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpu_H100 BootTime=2024-01-18T13:52:27 SlurmdStartTime=2024-01-19T11:25:44 LastBusyTime=2024-01-19T10:59:13 CfgTRES=cpu=112,mem=1030000M,billing=112,gres/gpu=14 AllocTRES=cpu=1 CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s $ scontrol show partition gpu_H100 PartitionName=gpu_H100 AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=node16 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=112 TotalNodes=1 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=112,mem=1030000M,node=1,billing=112,gres/gpu=14 slurm.conf: AccountingStorageTRES=gres/gpu,gres/gpu:A100,gres/gpu:H100,gres/gpu:H100_1g.10gb,gres/gpu:H100_2g.20gb,gres/gpu:H100_3g.40gb,gres/gpu:H100_7g.80gb GresTypes=gpu ... NodeName=node16 CPUs=112 RealMemory=1030000 Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 PartitionName=gpu_H100 Nodes=node16 Default=NO MaxTime=INFINITE State=UP gres.conf #AutoDetect=nvml # GPU 0 MIG 0 /proc/driver/nvidia/capabilities/gpu0/mig/gi7/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap66,/dev/nvidia-caps/nvidia-cap67 CPUs=0-55 # GPU 0 MIG 1 /proc/driver/nvidia/capabilities/gpu0/mig/gi8/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap75,/dev/nvidia-caps/nvidia-cap76 CPUs=0-55 # GPU 0 MIG 2 /proc/driver/nvidia/capabilities/gpu0/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap84,/dev/nvidia-caps/nvidia-cap85 CPUs=0-55 # GPU 0 MIG 3 /proc/driver/nvidia/capabilities/gpu0/mig/gi10/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap93,/dev/nvidia-caps/nvidia-cap94 CPUs=0-55 # GPU 0 MIG 4 /proc/driver/nvidia/capabilities/gpu0/mig/gi11/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap102,/dev/nvidia-caps/nvidia-cap103 CPUs=0-55 # GPU 0 MIG 5 /proc/driver/nvidia/capabilities/gpu0/mig/gi12/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap111,/dev/nvidia-caps/nvidia-cap112 CPUs=0-55 # GPU 0 MIG 6 /proc/driver/nvidia/capabilities/gpu0/mig/gi13/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap120,/dev/nvidia-caps/nvidia-cap121 CPUs=0-55 # GPU 1 MIG 0 /proc/driver/nvidia/capabilities/gpu1/mig/gi3/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap165,/dev/nvidia-caps/nvidia-cap166 CPUs=0-55 # GPU 1 MIG 1 /proc/driver/nvidia/capabilities/gpu1/mig/gi5/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap183,/dev/nvidia-caps/nvidia-cap184 CPUs=0-55 # GPU 1 MIG 2 /proc/driver/nvidia/capabilities/gpu1/mig/gi6/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap192,/dev/nvidia-caps/nvidia-cap193 CPUs=0-55 # GPU 1 MIG 3 /proc/driver/nvidia/capabilities/gpu1/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap219,/dev/nvidia-caps/nvidia-cap220 CPUs=0-55 # GPU 2 MIG 0 /proc/driver/nvidia/capabilities/gpu2/mig/gi1/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap282,/dev/nvidia-caps/nvidia-cap283 CPUs=0-55 # GPU 2 MIG 1 /proc/driver/nvidia/capabilities/gpu2/mig/gi2/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap291,/dev/nvidia-caps/nvidia-cap292 CPUs=0-55 # GPU 3 MIG 0 /proc/driver/nvidia/capabilities/gpu3/mig/gi0/access NodeName=node16 Name=gpu Type=H100_7g.80gb MultipleFiles=/dev/nvidia3,/dev/nvidia-caps/nvidia-cap408,/dev/nvidia-caps/nvidia-cap409 CPUs=0-55 [2024-01-19T14:31:13.701] debug: gres/gpu: init: loaded [2024-01-19T14:31:13.701] debug: gpu/generic: init: init: GPU Generic plugin loaded [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.702] Gres Name=gpu Type=H100_7g.80gb Count=1 Best, Drazen Jalsovec

1 0

Is there a way to map Azure AD users to NIS?
by Ben Wellborn 20 Jan '24

20 Jan '24

In the past, we would connect AD to NIS users using Centrify and adnisd, but I don't think adnisd can connect to Azure AD (HCI stack). Thanks, Ben

1 0

Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres)
by Jason Macklin 19 Jan '24

19 Jan '24

If you run "scontrol show jobid <jobid>" of your pending job with the "(Resources)" tag you may see more about what is unavailable to your job. Slurm default configs can cause an entire compute node of resources to be "allocated" to a running job regardless of whether it needs all of them or not so you may need to alter one or both of the following settings to allow more than one job to run on a single node at once. You'll find these in your slurm.conf. Don't forget to "scontrol reconf" and even potentially restart both "slurmctld" & "slurmd" on your nodes if you do end up making changes. SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory I hope this helps. Kind regards, Jason ---- Jason Macklin Manager Cyberinfrastructure, Research Cyberinfrastructure 860.837.2142 t | 860.202.7779 m jason.macklin(a)jax.org The Jackson Laboratory Maine | Connecticut | California | Shanghai www.jax.org<http://www.jax.org> The Jackson Laboratory: Leading the search for tomorrow's cures ________________________________ From: slurm-users <slurm-users-bounces(a)lists.schedmd.com> on behalf of slurm-users-request(a)lists.schedmd.com <slurm-users-request(a)lists.schedmd.com> Sent: Friday, January 19, 2024 9:24 AM To: slurm-users(a)lists.schedmd.com <slurm-users(a)lists.schedmd.com> Subject: [EXTERNAL]slurm-users Digest, Vol 75, Issue 31 Send slurm-users mailing list submissions to slurm-users(a)lists.schedmd.com To subscribe or unsubscribe via the World Wide Web, visit https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users or, via email, send a message with subject or body 'help' to slurm-users-request(a)lists.schedmd.com You can reach the person managing the list at slurm-users-owner(a)lists.schedmd.com When replying, please edit your Subject line so it is more specific than "Re: Contents of slurm-users digest..." Today's Topics: 1. Re: Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) (Marko Markoc) 2. Re: Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) (?mit Seren) ---------------------------------------------------------------------- Message: 1 Date: Fri, 19 Jan 2024 06:12:24 -0800 From: Marko Markoc <mmarkoc(a)pdx.edu> To: Slurm User Community List <slurm-users(a)lists.schedmd.com> Subject: Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) Message-ID: <CABnuMe4JTA0e6=VbO8D+To=8FGO+3Byv1dK_MC+OuRitzN5dXg(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" +1 on checking the memory allocation. Or add/check if you have any DefMemPerX set in your slurm.conf On Fri, Jan 19, 2024 at 12:33?AM mohammed shambakey <shambakey1(a)gmail.com> wrote: > Hi > > I'm not an expert, but is it possible that the currently running jobs is > consuming the whole node because it is allocated the whole memory of the > node (so the other 2 jobs had to wait until it finishes)? > Maybe if you try to restrict the required memory for each job? > > Regards > > On Thu, Jan 18, 2024 at 4:46?PM ?mit Seren <uemit.seren(a)gmail.com> wrote: > >> This line also has tobe changed: >> >> >> #SBATCH --gpus-per-node=4 ? #SBATCH --gpus-per-node=1 >> >> --gpus-per-node seems to be the new parameter that is replacing the --gres= >> one, so you can remove the ?gres line completely. >> >> >> >> Best >> >> ?mit >> >> >> >> *From: *slurm-users <slurm-users-bounces(a)lists.schedmd.com> on behalf of >> Kherfani, Hafedh (Professional Services, TC) <hafedh.kherfani(a)hpe.com> >> *Date: *Thursday, 18. January 2024 at 15:40 >> *To: *Slurm User Community List <slurm-users(a)lists.schedmd.com> >> *Subject: *Re: [slurm-users] Need help with running multiple >> instances/executions of a batch script in parallel (with NVIDIA HGX A100 >> GPU as a Gres) >> >> Hi Noam and Matthias, >> >> >> >> Thanks both for your answers. >> >> >> >> I changed the ?#SBATCH --gres=gpu:4? directive (in the batch script) with >> ?#SBATCH --gres=gpu:1? as you suggested, but it didn?t make a difference, >> as running this batch script 3 times will result in the first job to be in >> a running state, while the second and third jobs will still be in a pending >> state ? >> >> >> >> [slurmtest@c-a100-master test-batch-scripts]$ cat gpu-job.sh >> >> #!/bin/bash >> >> #SBATCH --job-name=gpu-job >> >> #SBATCH --partition=gpu >> >> #SBATCH --nodes=1 >> >> #SBATCH --gpus-per-node=4 >> >> #SBATCH --gres=gpu:1 # <<<< Changed from ?4? >> to ?1? >> >> #SBATCH --tasks-per-node=1 >> >> #SBATCH --output=gpu_job_output.%j >> >> #SBATCH --error=gpu_job_error.%j >> >> >> >> hostname >> >> date >> >> sleep 40 >> >> pwd >> >> >> >> [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh >> >> Submitted batch job *217* >> >> [slurmtest@c-a100-master test-batch-scripts]$ squeue >> >> JOBID PARTITION NAME USER ST TIME NODES >> NODELIST(REASON) >> >> 217 gpu gpu-job slurmtes R 0:02 1 >> c-a100-cn01 >> >> [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh >> >> Submitted batch job *218* >> >> [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh >> >> Submitted batch job *219* >> >> [slurmtest@c-a100-master test-batch-scripts]$ squeue >> >> JOBID PARTITION NAME USER ST TIME NODES >> NODELIST(REASON) >> >> 219 gpu gpu-job slurmtes *PD* 0:00 1 >> (Priority) >> >> 218 gpu gpu-job slurmtes *PD* 0:00 1 >> (Resources) >> >> 217 gpu gpu-job slurmtes *R* 0:07 1 >> c-a100-cn01 >> >> >> >> Basically I?m seeking for some help/hints on how to tell Slurm, from the >> batch script for example: ?I want only 1 or 2 GPUs to be used/consumed by >> the job?, and then I run the batch script/job a couple of times with sbatch >> command, and confirm that we can indeed have multiple jobs using a GPU and >> running in parallel, at the same time. >> >> >> >> Makes sense ? >> >> >> >> >> >> Best regards, >> >> >> >> *Hafedh * >> >> >> >> *From:* slurm-users <slurm-users-bounces(a)lists.schedmd.com> *On Behalf >> Of *Bernstein, Noam CIV USN NRL (6393) Washington DC (USA) >> *Sent:* jeudi 18 janvier 2024 2:30 PM >> *To:* Slurm User Community List <slurm-users(a)lists.schedmd.com> >> *Subject:* Re: [slurm-users] Need help with running multiple >> instances/executions of a batch script in parallel (with NVIDIA HGX A100 >> GPU as a Gres) >> >> >> >> On Jan 18, 2024, at 7:31 AM, Matthias Loose <m.loose(a)mindcode.de> wrote: >> >> >> >> Hi Hafedh, >> >> Im no expert in the GPU side of SLURM, but looking at you current >> configuration to me its working as intended at the moment. You have defined >> 4 GPUs and start multiple jobs each consuming 4 GPUs each. So the jobs wait >> for the ressource the be free again. >> >> I think what you need to look into is the MPS plugin, which seems to do >> what you are trying to achieve: >> https://slurm.schedmd.com/gres.html#MPS_Management >> >> >> >> I agree with the first paragraph. How many GPUs are you expecting each >> job to use? I'd have assumed, based on the original text, that each job is >> supposed to use 1 GPU, and the 4 jobs were supposed to be running >> side-by-side on the one node you have (with 4 GPUs). If so, you need to >> tell each job to request only 1 GPU, and currently each one is requesting 4. >> >> >> >> If your jobs are actually supposed to be using 4 GPUs each, I still don't >> see any advantage to MPS (at least in what is my usual GPU usage pattern): >> all the jobs will take longer to finish, because they are sharing the fixed >> resource. If they take turns, at least the first ones finish as fast as >> they can, and the last one will finish no later than it would have if they >> were all time-sharing the GPUs. I guess NVIDIA had something in mind when >> they developed MPS, so I guess our pattern may not be typical (or at least >> not universal), and in that case the MPS plugin may well be what you need. >> > > > -- > Mohammed >

1 0

slurmctld/slurmdbd (code=exited, status=217/USER)
by Miriam Olmi 19 Jan '24

19 Jan '24

Hi all, I am having some issue with the new version of slurm 23.11.0-1. I had already installed and configured slurm 23.02.3-1 on my cluster and all the services were active and running properly. After I install with the same procedure the new version of slurm I have that the slurmctld and slurmdbd daemons fail to start all with the same error: (code=exited, status=217/USER) And investigating the problem with the command journalctl -xe I find: slurmctld.service: Failed to determine user credentials: No such process slurmctld.service: Failed at step USER spawning /usr/sbin/slurmctld: No such process I had a look at the slurmctld.service file for both the slurm versions and I found the following differences in the [Service] section. >From the slurmctld.service file of slurm 23.02.3-1: [Service] Type=simple EnvironmentFile=-/etc/sysconfig/slurmctld EnvironmentFile=-/etc/default/slurmctld ExecStart=/usr/sbin/slurmctld -D -s $SLURMCTLD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID LimitNOFILE=65536 TasksMax=infinity >From the slurmctld.service file of slurm 23.11.0-1: [Service] Type=notify EnvironmentFile=-/etc/sysconfig/slurmctld EnvironmentFile=-/etc/default/slurmctld User=slurm Group=slurm ExecStart=/usr/sbin/slurmctld --systemd $SLURMCTLD_OPTIONS ExecReload=/bin/kill -HUP $MAINPID LimitNOFILE=65536 TasksMax=infinity I think the presence of the new lines regarding the slurm user might be the problem but I am not sure and I have no idea how to solve it. Can anyone halp me? Thanks in advance, Miriam

2 1

Jobs exiting together
by Alexander Silva 19 Jan '24

19 Jan '24

Recently, i have built an hpc cluster with slurm as workload. The test jobs with quatum chemistry codes have worked fine. However, production jobs with lammps have shown an unexpected behavior when the first job completed, normally or not, cause the termination of the others in the same compute node. Initially, I thought that was due to mpi malfunction, but this behavior is algo observed for serial lammps code. The lammps group said to me that behavior could be generated by slurm. My question to you is about what parameter in slurm.conf could be responsible for the termination of the other jobs. I am using an epilogue script that work normally in another cluster. Thanks.

1 0

Running slurm job on requested nvidia mig device
by Dražen Jalšovec 18 Jan '24

18 Jan '24

Hi, We are testing the MIG deployment on our new slurm compute node with 4 x H100 GPUs. It looks like everything is configured correctly but we have a problem accessing mig devices. When I submit jobs requesting a mig gpu device #SBATCH --gres=gpu:H100_1g.10gb:1, the jobs get submitted to the node, but only 4 jobs get executed and all other jobs fail. I was able to "solve it" by adding code to submit to the mig device UUID. Am I missing something in the slurm/gres/cgroups configuration that would automatically assign correct CUDA_VISIBLE_DEVICES MIG UUID to the job? If I submit 10 jobs requesting --gres=gpu:H100_1g.10gb:1 8 jobs start running: #1 - CUDA_VISIBLE_DEVICES=0 - jobs 1-4 run on one of the mig devices configured on GPU-device ... #8 - CUDA_VISIBLE_DEVICES=7 - jobs 5-8 fail with the message that there are no available CUDA_VISIBLE_DEVICES ... #10 - CUDA_VISIBLE_DEVICES=? - after waiting for free mig resources jobs 9-10 runs or fails depending if GPU-device is free (migs are free) $ nvidia-smi --query-gpu=uuid --id=0 --format=csv,noheader GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3 $ nvidia-smi --query-gpu=uuid --id=7 --format=csv,noheader No devices were found The slurm assigns CUDA_VISIBLE_DEVICES numbers based on the requested resources: Request: gpu:H100_1g.10gb:1 - CUDA_VISIBLE_DEVICES=0 gpu:H100_1g.10gb:2 - CUDA_VISIBLE_DEVICES=0,1 ... Any job with H100_1g.10gb request is going to get device number between 0 and 7 gpu:H100_1g.10gb:8 - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 gpu:H100_2g.20gb:1 - CUDA_VISIBLE_DEVICES=8 Any job with H100_2g.20gb request is going to get device number between 8 and 10 ... gpu:H100_2g.10gb:3 - CUDA_VISIBLE_DEVICES=8,9,10 gpu:H100_3g.40gb:1 - CUDA_VISIBLE_DEVICES=11 gpu:H100_3g.40gb:2 - CUDA_VISIBLE_DEVICES=12 gpu:H100_7g.80gb:1 - CUDA_VISIBLE_DEVICES=13 I was able to "solve it" by adding this code to my slurm script: # Get the CUDA index using srun and awk CUDA_INDEX=$(srun env | grep CUDA_VISIBLE_DEVICES | awk -F '=' '{print $2}') ## GPU0 1g.10gb ## MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a ## MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664 ## MIG-d3ab0675-d318-5e53-b487-b50695cf2e00 ## MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0 ## MIG-85dd76da-c994-5830-adf0-467c66ae1b95 ## MIG-29a6d43b-882e-5b79-868a-15bb2c770b82 ## MIG-558debd2-dc13-5406-9256-73ef4f279737 ## GPU1 2g.20gb ## MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f ## MIG-0a804c0c-aa27-5993-97cb-cedb854735ce ## MIG-8af9af6d-8720-5763-81e2-83afc43eb42b ## GPU1 1g.10gb ## MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a ## GPU2 3g.40gb ## MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b ## MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc ## GPU3 7g.80gb ## MIG-ec043869-9176-577e-bac0-46c8411e4e37 # Define the list of UUIDs UUIDS=( "MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a" "MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664" "MIG-d3ab0675-d318-5e53-b487-b50695cf2e00" "MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0" "MIG-85dd76da-c994-5830-adf0-467c66ae1b95" "MIG-29a6d43b-882e-5b79-868a-15bb2c770b82" "MIG-558debd2-dc13-5406-9256-73ef4f279737" "MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a" "MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f" "MIG-0a804c0c-aa27-5993-97cb-cedb854735ce" "MIG-8af9af6d-8720-5763-81e2-83afc43eb42b" "MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b" "MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc" "MIG-ec043869-9176-577e-bac0-46c8411e4e37" ) # Assign the UUID based on the CUDA index SELECTED_UUID=${UUIDS[$CUDA_INDEX]} # Print the selected UUID echo $SELECTED_UUID export CUDA_VISIBLE_DEVICES=$SELECTED_UUID echo "CUDA_VISIBLE_DEVICES set to: "$SELECTED_UUID echo "Test: "$CUDA_VISIBLE_DEVICES CUDA_VISIBLE_DEVICES=$SELECTED_UUID python3.8 gpu_script.py Result: Script start MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a CUDA_VISIBLE_DEVICES set to: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a Matrix calculation on CUDA device completed successfully. Script end +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 7 0 28686 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 8 0 28744 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 9 0 28300 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 10 0 28512 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 11 0 28506 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 12 0 28516 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 0 13 0 28511 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | | 1 9 0 28552 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB | +---------------------------------------------------------------------------------------+ This is the current setup: RockyLinux 9.2 slurm-22.05.9-1.el9.x86_64 CUDA Version: 12.1 Driver Version: 530.30.02 # nvidia-smi -L GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3) MIG 1g.10gb Device 0: (UUID: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a) MIG 1g.10gb Device 1: (UUID: MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664) MIG 1g.10gb Device 2: (UUID: MIG-d3ab0675-d318-5e53-b487-b50695cf2e00) MIG 1g.10gb Device 3: (UUID: MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0) MIG 1g.10gb Device 4: (UUID: MIG-85dd76da-c994-5830-adf0-467c66ae1b95) MIG 1g.10gb Device 5: (UUID: MIG-29a6d43b-882e-5b79-868a-15bb2c770b82) MIG 1g.10gb Device 6: (UUID: MIG-558debd2-dc13-5406-9256-73ef4f279737) GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1439a39c-948b-0657-98e9-aff8595a8729) MIG 2g.20gb Device 0: (UUID: MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f) MIG 2g.20gb Device 1: (UUID: MIG-0a804c0c-aa27-5993-97cb-cedb854735ce) MIG 2g.20gb Device 2: (UUID: MIG-8af9af6d-8720-5763-81e2-83afc43eb42b) MIG 1g.10gb Device 3: (UUID: MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a) GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-3b182fed-bfdf-bc11-d1bd-c41fd1468d2a) MIG 3g.40gb Device 0: (UUID: MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b) MIG 3g.40gb Device 1: (UUID: MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc) GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-7eba9d78-e908-3db8-2633-a269aeec395e) MIG 7g.80gb Device 0: (UUID: MIG-ec043869-9176-577e-bac0-46c8411e4e37) $ scontrol show node node16 NodeName=node16 Arch=x86_64 CoresPerSocket=1 CPUAlloc=1 CPUEfctv=112 CPUTot=112 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 NodeAddr=node16 NodeHostName=node16 Version=22.05.9 OS=Linux 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023 RealMemory=1030000 AllocMem=0 FreeMem=1011801 Sockets=112 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpu_H100 BootTime=2024-01-18T13:52:27 SlurmdStartTime=2024-01-19T11:25:44 LastBusyTime=2024-01-19T10:59:13 CfgTRES=cpu=112,mem=1030000M,billing=112,gres/gpu=14 AllocTRES=cpu=1 CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s $ scontrol show partition gpu_H100 PartitionName=gpu_H100 AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO QoS=N/A DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=node16 PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO OverTimeLimit=NONE PreemptMode=OFF State=UP TotalCPUs=112 TotalNodes=1 SelectTypeParameters=NONE JobDefaults=(null) DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED TRES=cpu=112,mem=1030000M,node=1,billing=112,gres/gpu=14 slurm.conf: AccountingStorageTRES=gres/gpu,gres/gpu:A100,gres/gpu:H100,gres/gpu:H100_1g.10gb,gres/gpu:H100_2g.20gb,gres/gpu:H100_3g.40gb,gres/gpu:H100_7g.80gb GresTypes=gpu ... NodeName=node16 CPUs=112 RealMemory=1030000 Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1 PartitionName=gpu_H100 Nodes=node16 Default=NO MaxTime=INFINITE State=UP gres.conf #AutoDetect=nvml # GPU 0 MIG 0 /proc/driver/nvidia/capabilities/gpu0/mig/gi7/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap66,/dev/nvidia-caps/nvidia-cap67 CPUs=0-55 # GPU 0 MIG 1 /proc/driver/nvidia/capabilities/gpu0/mig/gi8/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap75,/dev/nvidia-caps/nvidia-cap76 CPUs=0-55 # GPU 0 MIG 2 /proc/driver/nvidia/capabilities/gpu0/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap84,/dev/nvidia-caps/nvidia-cap85 CPUs=0-55 # GPU 0 MIG 3 /proc/driver/nvidia/capabilities/gpu0/mig/gi10/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap93,/dev/nvidia-caps/nvidia-cap94 CPUs=0-55 # GPU 0 MIG 4 /proc/driver/nvidia/capabilities/gpu0/mig/gi11/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap102,/dev/nvidia-caps/nvidia-cap103 CPUs=0-55 # GPU 0 MIG 5 /proc/driver/nvidia/capabilities/gpu0/mig/gi12/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap111,/dev/nvidia-caps/nvidia-cap112 CPUs=0-55 # GPU 0 MIG 6 /proc/driver/nvidia/capabilities/gpu0/mig/gi13/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap120,/dev/nvidia-caps/nvidia-cap121 CPUs=0-55 # GPU 1 MIG 0 /proc/driver/nvidia/capabilities/gpu1/mig/gi3/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap165,/dev/nvidia-caps/nvidia-cap166 CPUs=0-55 # GPU 1 MIG 1 /proc/driver/nvidia/capabilities/gpu1/mig/gi5/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap183,/dev/nvidia-caps/nvidia-cap184 CPUs=0-55 # GPU 1 MIG 2 /proc/driver/nvidia/capabilities/gpu1/mig/gi6/access NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap192,/dev/nvidia-caps/nvidia-cap193 CPUs=0-55 # GPU 1 MIG 3 /proc/driver/nvidia/capabilities/gpu1/mig/gi9/access NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap219,/dev/nvidia-caps/nvidia-cap220 CPUs=0-55 # GPU 2 MIG 0 /proc/driver/nvidia/capabilities/gpu2/mig/gi1/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap282,/dev/nvidia-caps/nvidia-cap283 CPUs=0-55 # GPU 2 MIG 1 /proc/driver/nvidia/capabilities/gpu2/mig/gi2/access NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap291,/dev/nvidia-caps/nvidia-cap292 CPUs=0-55 # GPU 3 MIG 0 /proc/driver/nvidia/capabilities/gpu3/mig/gi0/access NodeName=node16 Name=gpu Type=H100_7g.80gb MultipleFiles=/dev/nvidia3,/dev/nvidia-caps/nvidia-cap408,/dev/nvidia-caps/nvidia-cap409 CPUs=0-55 [2024-01-19T14:31:13.701] debug: gres/gpu: init: loaded [2024-01-19T14:31:13.701] debug: gpu/generic: init: init: GPU Generic plugin loaded [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1 [2024-01-19T14:31:13.702] Gres Name=gpu Type=H100_7g.80gb Count=1 Best, Drazen Jalsovec

1 0

Potential Side Effects of larger MessageTimeout value
by Herc Silverstein 18 Jan '24

18 Jan '24

Hi, What are potential bad side effects of using a large/larger MessageTimeout? And is there a value at which this setting is too large (long)? Thanks, Herc

1 0

error
by Felix 18 Jan '24

18 Jan '24

Hello I started a new AMD node, and the error is as follows: "CPU frequency setting not configured for this node" extended looks like this: [2024-01-18T18:28:06.682] CPU frequency setting not configured for this node [2024-01-18T18:28:06.691] slurmd started on Thu, 18 Jan 2024 18:28:06 +0200 [2024-01-18T18:28:06.691] CPUs=128 Boards=1 Sockets=1 Cores=64 Threads=2 Memory=256786 TmpDisk=875797 Uptime=4569 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null) In the configuration file I have the following: NodeName=awn-1[04] NodeAddr=192.168.4.[111] CPUs=128 RealMemory=256000 Sockets=1 CoresPerSocket=64 ThreadsPerCore=2 Feature=HyperThread Could you please help me? Thank you Felix -- Dr. Eng. Farcas Felix National Institute of Research and Development of Isotopic and Molecular Technology, IT - Department - Cluj-Napoca, Romania Mobile: +40742195323

2 1

Re: [slurm-users] [BULK] slurm-users Digest, Vol 75, Issue 26
by Jason Macklin 18 Jan '24

18 Jan '24

If you run "scontrol show jobid <jobid>" of your pending job with the "(Resources)" tag you may see more about what is unavailable to your job. Slurm default configs can cause an entire compute node of resources to be "allocated" to a running job regardless of whether it needs all of them or not so you may need to alter one or both of the following settings to allow more than one job to run on a single node at once. You'll find these in your slurm.conf. Don't forget to "scontrol reconf" and even potentially restart both "slurmctld" & "slurmd" on your nodes if you do end up making changes. SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory I hope this helps. Kind regards, Jason ---- Jason Macklin Manager Cyberinfrastructure, Research Cyberinfrastructure 860.837.2142 t | 860.202.7779 m jason.macklin(a)jax.org The Jackson Laboratory Maine | Connecticut | California | Shanghai www.jax.org<http://www.jax.org> The Jackson Laboratory: Leading the search for tomorrow's cures ________________________________ From: slurm-users <slurm-users-bounces(a)lists.schedmd.com> on behalf of slurm-users-request(a)lists.schedmd.com <slurm-users-request(a)lists.schedmd.com> Sent: Thursday, January 18, 2024 9:46 AM To: slurm-users(a)lists.schedmd.com <slurm-users(a)lists.schedmd.com> Subject: [BULK] slurm-users Digest, Vol 75, Issue 26 Send slurm-users mailing list submissions to slurm-users(a)lists.schedmd.com To subscribe or unsubscribe via the World Wide Web, visit https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users or, via email, send a message with subject or body 'help' to slurm-users-request(a)lists.schedmd.com You can reach the person managing the list at slurm-users-owner(a)lists.schedmd.com When replying, please edit your Subject line so it is more specific than "Re: Contents of slurm-users digest..." Today's Topics: 1. Re: Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) (Baer, Troy) ---------------------------------------------------------------------- Message: 1 Date: Thu, 18 Jan 2024 14:46:48 +0000 From: "Baer, Troy" <troy(a)osc.edu> To: Slurm User Community List <slurm-users(a)lists.schedmd.com> Subject: Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) Message-ID: <CH0PR01MB6924127AF471DED69151805BCF712(a)CH0PR01MB6924.prod.exchangelabs.com> Content-Type: text/plain; charset="utf-8" Hi Hafedh, Your job script has the sbatch directive ??gpus-per-node=4? set. I suspect that if you look at what?s allocated to the running job by doing ?scontrol show job <jobid>? and looking at the TRES field, it?s been allocated 4 GPUs instead of one. Regards, --Troy From: slurm-users <slurm-users-bounces(a)lists.schedmd.com> On Behalf Of Kherfani, Hafedh (Professional Services, TC) Sent: Thursday, January 18, 2024 9:38 AM To: Slurm User Community List <slurm-users(a)lists.schedmd.com> Subject: Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) Hi Noam and Matthias, Thanks both for your answers. I changed the ?#SBATCH --gres=gpu:?4? directive (in the batch script) with ?#SBATCH --gres=gpu:?1? as you suggested, but it didn?t make a difference, as running Hi Noam and Matthias, Thanks both for your answers. I changed the ?#SBATCH --gres=gpu:4? directive (in the batch script) with ?#SBATCH --gres=gpu:1? as you suggested, but it didn?t make a difference, as running this batch script 3 times will result in the first job to be in a running state, while the second and third jobs will still be in a pending state ? [slurmtest@c-a100-master test-batch-scripts]$ cat gpu-job.sh #!/bin/bash #SBATCH --job-name=gpu-job #SBATCH --partition=gpu #SBATCH --nodes=1 #SBATCH --gpus-per-node=4 #SBATCH --gres=gpu:1 # <<<< Changed from ?4? to ?1? #SBATCH --tasks-per-node=1 #SBATCH --output=gpu_job_output.%j #SBATCH --error=gpu_job_error.%j hostname date sleep 40 pwd [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 217 [slurmtest@c-a100-master test-batch-scripts]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 217 gpu gpu-job slurmtes R 0:02 1 c-a100-cn01 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 218 [slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh Submitted batch job 219 [slurmtest@c-a100-master test-batch-scripts]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 219 gpu gpu-job slurmtes PD 0:00 1 (Priority) 218 gpu gpu-job slurmtes PD 0:00 1 (Resources) 217 gpu gpu-job slurmtes R 0:07 1 c-a100-cn01 Basically I?m seeking for some help/hints on how to tell Slurm, from the batch script for example: ?I want only 1 or 2 GPUs to be used/consumed by the job?, and then I run the batch script/job a couple of times with sbatch command, and confirm that we can indeed have multiple jobs using a GPU and running in parallel, at the same time. Makes sense ? Best regards, Hafedh From: slurm-users <slurm-users-bounces(a)lists.schedmd.com<mailto:slurm-users-bounces@lists.schedmd.com>> On Behalf Of Bernstein, Noam CIV USN NRL (6393) Washington DC (USA) Sent: jeudi 18 janvier 2024 2:30 PM To: Slurm User Community List <slurm-users(a)lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>> Subject: Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres) On Jan 18, 2024, at 7:31 AM, Matthias Loose <m.loose(a)mindcode.de<mailto:m.loose@mindcode.de>> wrote: Hi Hafedh, Im no expert in the GPU side of SLURM, but looking at you current configuration to me its working as intended at the moment. You have defined 4 GPUs and start multiple jobs each consuming 4 GPUs each. So the jobs wait for the ressource the be free again. I think what you need to look into is the MPS plugin, which seems to do what you are trying to achieve: https://slurm.schedmd.com/gres.html#MPS_Management<https://urldefense.com/v3/__https:/slurm.schedmd.com/gres.html*MPS_Manageme…> I agree with the first paragraph. How many GPUs are you expecting each job to use? I'd have assumed, based on the original text, that each job is supposed to use 1 GPU, and the 4 jobs were supposed to be running side-by-side on the one node you have (with 4 GPUs). If so, you need to tell each job to request only 1 GPU, and currently each one is requesting 4. If your jobs are actually supposed to be using 4 GPUs each, I still don't see any advantage to MPS (at least in what is my usual GPU usage pattern): all the jobs will take longer to finish, because they are sharing the fixed resource. If they take turns, at least the first ones finish as fast as they can, and the last one will finish no later than it would have if they were all time-sharing the GPUs. I guess NVIDIA had something in mind when they developed MPS, so I guess our pattern may not be typical (or at least not universal), and in that case the MPS plugin may well be what you need.

1 0

slurm.conf
by LEROY Christine 208562 18 Jan '24

18 Jan '24

Hello all, Is there an env variable in SLURM to tell where the slurm.conf is? We would like to have on the same client node, 2 type of possible submissions to address 2 different cluster. Thanks in advance, Christine

4 3

What happens if GPU GRES exceeding number of GPUs per node
by Purwanto, Wirawan 18 Jan '24

18 Jan '24

Hi, In my HPC center, I found a SLURM job that was submitted with --gres=gpu:6 whereas the cluster has only four GPUs per node each. It is a parallel job. Here are some relevant field printout: AllocCPUS 30 AllocGRES gpu:6 AllocTRES billing=30,cpu=30,gres/gpu=6,node=3 CPUTime 1-01:23:00 CPUTimeRAW 91380 Elapsed 00:50:46 JobID 20073 JobIDRaw 20073 JobName simple_cuda NCPUS 30 NGPUS 6.0 What happened in this case? This job was asking for 3 nodes, 10 core per node. When the user specified “--gres=gpu:6”, does this mean six GPUs for the entire job, or six GPUs per node? Per the description in https://slurm.schedmd.com/gres.html#Running_Jobs, it says: gres is “Generic resources required per node”. So it is illogical to request six GPUs per node. So what happened? Did SLURM quietly ignore the request and grant just one, or grant the max number (4)? Because apparently the job ran without error. Wirawan Purwanto Computational Scientist, HPC Group Information Technology Services Old Dominion University Norfolk, VA 23529

2 1

Strict GrpTRESMins limit
by Kamil Wilczek 17 Jan '24

17 Jan '24

Dear All, I tried to implement a strict limit on the GrpTRESMins for each user. The effect I'm trying to achieve is that after the limit of GPU minutes is reached, no new jobs can be run. No decay, no automatic resource replenishment. After the limit on GPU minutes is reached, each user should ask for more minutes. But despite exceeding the limits users *can* run new jobs. * When I'm adding a user to the cluster I set: sacctmgr --immediate add user name=... ... QOS=2gpu2d GrpTRESMins=gres/gpu=20000 * In the "slurm.conf" ("safe" means limits and associations are automatically set). Storage is MariaDB with SlurmDBD: GresTypes=gpu AccountingStorageTRES=gres/gpu AccountingStorageEnforce=qos,safe # This disables GPU minutes replenishing. PriorityDecayHalfLife=0 PriorityUsageResetPeriod=NONE But when I look at a user's account info and usage, you can see that the limits are not enforced. Account User Partition QOS GrpTRESMins ---------- ---------------- ------------ ------------ -------------------- redacted redacted a6000 2gpu2d gres/gpu=10000 -------------------------------------------------------------------------------- Top 1 Users 2024-01-05T00:00:00 - 2024-01-17T19:59:59 (1108800 secs) Usage reported in TRES Minutes -------------------------------------------------------------------------------- Login Used TRES Name ------------ -------- ---------------- redacted 184311 gres/gpu redacted 1558558 cpu Could someone explain, where could the problem be? Am I missing something? Apparently yes :) Kind regards -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

1 0

Suspend/Resume request limit
by 김종록 17 Jan '24

17 Jan '24

2 1

preemptable queue
by Davide DelVento 12 Jan '24

12 Jan '24

I would like to add a preemptable queue to our cluster. Actually I already have. We simply want jobs submitted to that queue be preempted if there are no resources available for jobs in other (high priority) queues. Conceptually very simple, no conditionals, no choices, just what I wrote. However it does not work as desired. This is the relevant part: grep -i Preemp /opt/slurm/slurm.conf #PreemptType = preempt/partition_prio PartitionName=regular DefMemPerCPU=4580 Default=True Nodes=node[01-12] State=UP PreemptMode=off PriorityTier=200 PartitionName=All DefMemPerCPU=4580 Nodes=node[01-36] State=UP PreemptMode=off PriorityTier=500 PartitionName=lowpriority DefMemPerCPU=4580 Nodes=node[01-36] State=UP PreemptMode=cancel PriorityTier=100 That PreemptType setting (now commented) fully breaks slurm, everything refuses to run with errors like $ squeue squeue: error: PreemptType and PreemptMode values incompatible squeue: fatal: Unable to process configuration file If I understand correctly the documentation at https://slurm.schedmd.com/preempt.html that is because preemption cannot cancel jobs based on partition priority, which (if true) is really unfortunate. I understand that allowing cross-partition time-slicing could be tricky and so I understand why that isn't allowed, but cancelling? Anyway, I have to questions: 1) is that correct and so should I avoid using either partition priority or cancelling? 2) is there an easy way to trick slurm into requeing and then have those jobs cancelled instead? 3) I guess the cleanest option would be to implement QoS, but I've never done it and we don't really need it for anything else other than this. The documentation looks complicated, but is it? The great Ole's website is unavailable at the moment... Thanks!!

2 4

Re: [slurm-users] sacct --name --status filtering
by Drucker, Daniel 11 Jan '24

11 Jan '24

Yes, that makes sense. Thank you! The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

sacct --name --status filtering
by Drucker, Daniel 10 Jan '24

10 Jan '24

What am I misunderstanding about how sacct filtering works here? I would have expected the second command to show the exact same results as the first. [root@mickey ddrucker]# sacct --starttime $(date -d "7 days ago" +"%Y-%m-%d") -X --format JobID,JobName,State,Elapsed --name zsh JobID JobName State Elapsed ------------ ---------- ---------- ---------- 257713 zsh COMPLETED 00:01:02 257714 zsh COMPLETED 00:04:01 257715 zsh COMPLETED 00:03:01 257716 zsh COMPLETED 00:03:01 [root@mickey ddrucker]# sacct --starttime $(date -d "7 days ago" +"%Y-%m-%d") -X --format JobID,JobName,State,Elapsed --name zsh --state COMPLETED JobID JobName State Elapsed ------------ ---------- ---------- ---------- [root@mickey ddrucker]# sinfo --version slurm 21.08.8-2 -- Daniel M. Drucker, Ph.D. Director of IT, MGB Imaging at Belmont McLean Hospital, a Harvard Medical School Affiliate The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

3 2

Re: [slurm-users] sacct --name --status filtering
by Drucker, Daniel 10 Jan '24

10 Jan '24

> All I can say is that this has to do with --starttime and that you have to read the manual really carefully about how they interact, including when you have --endtime set. It’s a bit fiddly and annoying, IMO, and I can never quite remember how it works. Oh, I think I understand. --starttime actually behaves differently when --state is present: If states are given with the '-s' option then only jobs in this state at this time will be returned. So is there a way to do what I want? I want to see jobs which - started later than 7 days ago - whose state is COMPLETED Surely that's possible without resorting to grep? Daniel The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

1 0

Cleanup of old clusters in database
by Jeffrey R. Lang 10 Jan '24

10 Jan '24

We have shuttered two clusters and need to remove them from the database. To do this, do we remove the table spaces associated with the cluster names from the Slurm database? Thanks, Jeff

1 0

Re: [slurm-users] DBD_SEND_MULT_MSG - invalid uid error (Timony, Mick)
by Craig Stark 10 Jan '24

10 Jan '24

All good ideas Mick – - I've restarted slurmd on all nodes – no effect - Ran this on all nodes: #!/bin/bash uname -n id slurm id 59999 scontrol show config | grep SlurmUser All show slurm being that 59999 user. - The firewalld already has the internal network interface being used set to the trusted zone I get a bit more info out of setting the slurmctld to debug level, but I'm not sure what to make of it TBH. I'm not sure what "_handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474)" is trying to tell me. Jan 10 10:48:26 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:31 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:35 kirby slurmctld[461138]: slurmctld: debug: accounting_storage/slurmdbd: _handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474): DBD_SEND_MULT_MSG message from invalid uid Jan 10 10:48:36 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:41 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:46 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:46 kirby slurmctld[461138]: slurmctld: debug: sched/backfill: _attempt_backfill: beginning Jan 10 10:48:46 kirby slurmctld[461138]: slurmctld: debug: sched/backfill: _attempt_backfill: no jobs to backfill Jan 10 10:48:51 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:48:53 kirby slurmctld[461138]: slurmctld: debug: accounting_storage/slurmdbd: _handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474): DBD_SEND_MULT_MSG message from invalid uid Jan 10 10:48:56 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:01 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:06 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:11 kirby slurmctld[461138]: slurmctld: debug: accounting_storage/slurmdbd: _handle_mult_rc_ret: PERSIST_RC is 2002 from DBD_SEND_MULT_MSG(1474): DBD_SEND_MULT_MSG message from invalid uid Jan 10 10:49:11 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:16 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid Jan 10 10:49:17 kirby slurmctld[461138]: slurmctld: debug: sched: Running job scheduler for full queue. Jan 10 10:49:21 kirby slurmctld[461138]: slurmctld: error: DBD_SEND_MULT_JOB_START message from invalid uid A bit more info / another possible clue. While "sacctmgr list Account" or "sacctmgr list user" shows expected account groups and users, "sreport user top start=12/1/23" and "sreport cluster utilization start=12/1/23" both report empty tables. Craig Stark, Ph.D. Professor, Department of Neurobiology and Behavior Director, Facility for Imaging and Brain Research (FIBRE) Director, Campus Center for Neuroimaging (CCNI) School of Biological Sciences, University of California, Irvine cestark(a)uci.edu<mailto:cestark@uci.edu>

1 0

Beginner admin question: Prioritization within a partition based on time limit
by Kenneth Chiu 09 Jan '24

09 Jan '24

I'm just learning about slurm. I understand that different different partitions can be prioritized separately, and can have different max time limits. I was wondering whether or not there was a way to have a finer-grained prioritization based on the time limit specified by a job, within a single partition. Or perhaps this is already happening by default? Would the backfill scheduler be best for this?

2 1

Re: [slurm-users] DBD_SEND_MULT_MSG - invalid uid error
by Craig Stark 08 Jan '24

08 Jan '24

This ticket with SchedMD implies it's a munged issue: https://urldefense.com/v3/__https://bugs.schedmd.com/show_bug.cgi?id=1293__… Is the munge daemon running on all systems? If it is, are all servers running a network time daemon such chronyd or ntpd and the time is in sync on all hosts? Thanks Mick, munge is seemingly running on all systems (systemctl status munge). I do get a warning about the munge file changing on disk, but I'm pretty sure that's from warewulf sync'ing files every minute. A sha256sum on the munge.key file on the compute nodes and host node says they're the same, so I think I can put that aside. The management node runs chrony and the compute nodes sync to the management node. [root@kirby uber]# chronyc tracking Reference ID : 4A06A849 (t2.time.gq1.yahoo.com) Stratum : 3 Ref time (UTC) : Mon Jan 08 22:26:44 2024 System time : 0.000032525 seconds slow of NTP time Last offset : -0.000021390 seconds RMS offset : 0.000055729 seconds Frequency : 38.797 ppm slow Residual freq : +0.001 ppm Skew : 0.018 ppm Root delay : 0.033342984 seconds Root dispersion : 0.000524800 seconds Update interval : 256.8 seconds Leap status : Normal vs [root@sonic01 ~]# chronyc tracking Reference ID : C0A80102 (warewulf) Stratum : 4 Ref time (UTC) : Mon Jan 08 22:31:02 2024 System time : 0.000000120 seconds slow of NTP time Last offset : -0.000000092 seconds RMS offset : 0.000014737 seconds Frequency : 47.495 ppm slow Residual freq : +0.000 ppm Skew : 0.066 ppm Root delay : 0.033458963 seconds Root dispersion : 0.000283949 seconds Update interval : 64.2 seconds Leap status : Normal So, the compute node is talking to the host and the host is talking to generic NTP sources. "date" shows the same time on the compute nodes

1 0

Re: [slurm-users] DBD_SEND_MULT_MSG - invalid uid error
by Craig Stark 08 Jan '24

08 Jan '24

3rd time trying to get this to come through to the list - hopefully this time works. I've been running SLURM for several years now, but in setting it up on a new cluster, I'm hitting a recurring issue. I'm using a MariaDB and configured it just as I had in my several-year-ago setup and in the docs. There's a "slurm" user (59999) on the OS (Rocky 9), that's on all the nodes, and I've added the slurm@localhost as instructed (grant all on slurm_acct_db.* TO 'slurm'@'localhost' identified by 'PASSWORD'). But, I keep getting things like this: ``` Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: DBD_SEND_MULT_MSG message from invalid uid 59999 Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: Processing last message from connection 7(192.168.1.2) uid(59999) Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: CONN:7 DBD_REGISTER_CTLD message from invalid uid 59999 Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: CONN:7 Security violation, DBD_REGISTER_CTLD Dec 22 14:22:07 kirby slurmdbd[14518]: slurmdbd: error: Processing last message from connection 7(192.168.1.2) uid(59999) ``` I'm a total SQL noob, but can at least verify that the user is in there: MariaDB [(none)]> SELECT User, Host, Password FROM mysql.user; +-------------+-----------+-------------------------------------------+ | User | Host | Password | +-------------+-----------+-------------------------------------------+ | mariadb.sys | localhost | | | root | localhost | invalid | | mysql | localhost | invalid | | slurm | localhost | *D6665ECF4F3CB12BCA836117F7727B6D0B78D644 | +-------------+-----------+-------------------------------------------+ 4 rows in set (0.002 sec) Any thoughts as to where I might look to fix this? Craig

2 1

Multifactor fair-share with single account
by Kamil Wilczek 08 Jan '24

08 Jan '24

Dear All, I have a question regarding the fair-share factor of the multifactor priority algorithm. My current understanding is that the fair-share makes sure that different *accounts* have a fair share of the computational power. But what if my organisation structure is flat and I have only one account where all my user reside. Is fair-share algorithm working in this situation -- does it take into account users (associations) from this single account, and tries to assing a fair-factor to each user? Or each user from this account have the same fair-factor at each iteration? And what if I have, say 3 accounts, but I do not wan't to calculate fair-share between accounts, but between all associations from all 3 accounts? In other words, is there a fair-share factor for users/associations instead of accounts? Kind regards -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

4 6

Tool for profiling resource usage by slurm jobs
by Nicolas Granger 08 Jan '24

08 Jan '24

Hi all, Happy new year everyone! I've been looking for a simple tool that reports how much resources are actually consumed by a job to help my colleagues and I adjust job requirements. I could not find such a tool, or the ones mentioned on this ML were not easy to install and use, so I have written a new one: https://github.com/CEA-LIST/sprofile It's a simple python script which parses cgroup and nvml data from the nvidia driver. It reports duration, cpu load, peak RAM, GPU load and peak GPU memory like so: |-- sprofile report (node03) -- Time: 0:00:03 / 1:00:00 CPU load: 2.0 / 4.0 RAM peak mem: 7G / 8G GPU load: 0.2 / 2.0 GPU peak mem: 7G / 40G| The requirements are to use the slurm cgroup plugin and to enable accounting on the GPU (nvidia-smi --accounting-mode=1). I hope you find this useful and let me know I you find bugs or want to contribute. Regards, Nicolas Granger

1 0