Hi,We are testing the MIG deployment on our new slurm compute node with 4 x H100 GPUs. It looks like everything is configured correctly but we have a problem accessing mig devices. When I submit jobs requesting a mig gpu device #SBATCH --gres=gpu:H100_1g.10gb:1, the jobs get submitted to the node, but only 4 jobs get executed and all other jobs fail. I was able to "solve it" by adding code to submit to the mig device UUID. Am I missing something in the slurm/gres/cgroups configuration that would automatically assign correct CUDA_VISIBLE_DEVICES MIG UUID to the job?
If I submit 10 jobs requesting --gres=gpu:H100_1g.10gb:1
8 jobs start running:
#1 - CUDA_VISIBLE_DEVICES=0 - jobs 1-4 run on one of the mig devices configured on GPU-device
...
#8 - CUDA_VISIBLE_DEVICES=7 - jobs 5-8 fail with the message that there are no available CUDA_VISIBLE_DEVICES
...
#10 - CUDA_VISIBLE_DEVICES=? - after waiting for free mig resources jobs 9-10 runs or fails depending if GPU-device is free (migs are free)
$ nvidia-smi --query-gpu=uuid --id=0 --format=csv,noheader
GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3
$ nvidia-smi --query-gpu=uuid --id=7 --format=csv,noheader
No devices were found
The slurm assigns CUDA_VISIBLE_DEVICES numbers based on the requested resources:
Request:
gpu:H100_1g.10gb:1 - CUDA_VISIBLE_DEVICES=0
gpu:H100_1g.10gb:2 - CUDA_VISIBLE_DEVICES=0,1
...
Any job with H100_1g.10gb request is going to get device number between 0 and 7
gpu:H100_1g.10gb:8 - CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
gpu:H100_2g.20gb:1 - CUDA_VISIBLE_DEVICES=8
Any job with H100_2g.20gb request is going to get device number between 8 and 10
...
gpu:H100_2g.10gb:3 - CUDA_VISIBLE_DEVICES=8,9,10
gpu:H100_3g.40gb:1 - CUDA_VISIBLE_DEVICES=11
gpu:H100_3g.40gb:2 - CUDA_VISIBLE_DEVICES=12
gpu:H100_7g.80gb:1 - CUDA_VISIBLE_DEVICES=13
I was able to "solve it" by adding this code to my slurm script:
# Get the CUDA index using srun and awk
CUDA_INDEX=$(srun env | grep CUDA_VISIBLE_DEVICES | awk -F '=' '{print $2}')
## GPU0 1g.10gb
## MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a
## MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664
## MIG-d3ab0675-d318-5e53-b487-b50695cf2e00
## MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0
## MIG-85dd76da-c994-5830-adf0-467c66ae1b95
## MIG-29a6d43b-882e-5b79-868a-15bb2c770b82
## MIG-558debd2-dc13-5406-9256-73ef4f279737
## GPU1 2g.20gb
## MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f
## MIG-0a804c0c-aa27-5993-97cb-cedb854735ce
## MIG-8af9af6d-8720-5763-81e2-83afc43eb42b
## GPU1 1g.10gb
## MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a
## GPU2 3g.40gb
## MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b
## MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc
## GPU3 7g.80gb
## MIG-ec043869-9176-577e-bac0-46c8411e4e37
# Define the list of UUIDs
UUIDS=(
"MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a"
"MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664"
"MIG-d3ab0675-d318-5e53-b487-b50695cf2e00"
"MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0"
"MIG-85dd76da-c994-5830-adf0-467c66ae1b95"
"MIG-29a6d43b-882e-5b79-868a-15bb2c770b82"
"MIG-558debd2-dc13-5406-9256-73ef4f279737"
"MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a"
"MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f"
"MIG-0a804c0c-aa27-5993-97cb-cedb854735ce"
"MIG-8af9af6d-8720-5763-81e2-83afc43eb42b"
"MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b"
"MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc"
"MIG-ec043869-9176-577e-bac0-46c8411e4e37"
)
# Assign the UUID based on the CUDA index
SELECTED_UUID=${UUIDS[$CUDA_INDEX]}
# Print the selected UUID
echo $SELECTED_UUID
export CUDA_VISIBLE_DEVICES=$SELECTED_UUID
echo "CUDA_VISIBLE_DEVICES set to: "$SELECTED_UUID
echo "Test: "$CUDA_VISIBLE_DEVICES
CUDA_VISIBLE_DEVICES=$SELECTED_UUID python3.8 gpu_script.py
Result:
Script start
MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a
CUDA_VISIBLE_DEVICES set to: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a
Matrix calculation on CUDA device completed successfully.
Script end
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 7 0 28686 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB |
| 0 8 0 28744 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB |
| 0 9 0 28300 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB |
| 0 10 0 28512 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB |
| 0 11 0 28506 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB |
| 0 12 0 28516 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB |
| 0 13 0 28511 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB |
| 1 9 0 28552 C ...gpu-cuda12.1-python38/bin/python3.8 866MiB |
+---------------------------------------------------------------------------------------+
This is the current setup:
RockyLinux 9.2
slurm-22.05.9-1.el9.x86_64
CUDA Version: 12.1 Driver Version: 530.30.02
# nvidia-smi -L
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-6c7f34eb-7b0b-9b09-4a2a-a009fb1ba6d3)
MIG 1g.10gb Device 0: (UUID: MIG-fbb82c87-989b-5e2c-98e5-a57919c5cb1a)
MIG 1g.10gb Device 1: (UUID: MIG-d1679b4c-c8e7-5d78-bd63-07334ecad664)
MIG 1g.10gb Device 2: (UUID: MIG-d3ab0675-d318-5e53-b487-b50695cf2e00)
MIG 1g.10gb Device 3: (UUID: MIG-c1384014-7937-5942-ba2a-900bd8a4c4b0)
MIG 1g.10gb Device 4: (UUID: MIG-85dd76da-c994-5830-adf0-467c66ae1b95)
MIG 1g.10gb Device 5: (UUID: MIG-29a6d43b-882e-5b79-868a-15bb2c770b82)
MIG 1g.10gb Device 6: (UUID: MIG-558debd2-dc13-5406-9256-73ef4f279737)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-1439a39c-948b-0657-98e9-aff8595a8729)
MIG 2g.20gb Device 0: (UUID: MIG-a78cd536-25b7-53cb-941a-a2db5eb0375f)
MIG 2g.20gb Device 1: (UUID: MIG-0a804c0c-aa27-5993-97cb-cedb854735ce)
MIG 2g.20gb Device 2: (UUID: MIG-8af9af6d-8720-5763-81e2-83afc43eb42b)
MIG 1g.10gb Device 3: (UUID: MIG-68d47441-f5fb-5a3b-ab5b-a0f3449c5b2a)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-3b182fed-bfdf-bc11-d1bd-c41fd1468d2a)
MIG 3g.40gb Device 0: (UUID: MIG-2ed972aa-6640-5116-9271-3d61c0dc9f3b)
MIG 3g.40gb Device 1: (UUID: MIG-3bdad788-60a5-5bef-85b9-fbdb36dc71fc)
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-7eba9d78-e908-3db8-2633-a269aeec395e)
MIG 7g.80gb Device 0: (UUID: MIG-ec043869-9176-577e-bac0-46c8411e4e37)
$ scontrol show node node16
NodeName=node16 Arch=x86_64 CoresPerSocket=1
CPUAlloc=1 CPUEfctv=112 CPUTot=112 CPULoad=0.00
AvailableFeatures=(null)
ActiveFeatures=(null)
Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1
NodeAddr=node16 NodeHostName=node16 Version=22.05.9
OS=Linux 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023
RealMemory=1030000 AllocMem=0 FreeMem=1011801 Sockets=112 Boards=1
State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
Partitions=gpu_H100
BootTime=2024-01-18T13:52:27 SlurmdStartTime=2024-01-19T11:25:44
LastBusyTime=2024-01-19T10:59:13
CfgTRES=cpu=112,mem=1030000M,billing=112,gres/gpu=14
AllocTRES=cpu=1
CapWatts=n/a
CurrentWatts=0 AveWatts=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
$ scontrol show partition gpu_H100
PartitionName=gpu_H100
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=NO QoS=N/A
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED
Nodes=node16
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=112 TotalNodes=1 SelectTypeParameters=NONE
JobDefaults=(null)
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
TRES=cpu=112,mem=1030000M,node=1,billing=112,gres/gpu=14
slurm.conf:
AccountingStorageTRES=gres/gpu,gres/gpu:A100,gres/gpu:H100,gres/gpu:H100_1g.10gb,gres/gpu:H100_2g.20gb,gres/gpu:H100_3g.40gb,gres/gpu:H100_7g.80gb
GresTypes=gpu
...
NodeName=node16 CPUs=112 RealMemory=1030000 Gres=gpu:H100_1g.10gb:8,gpu:H100_2g.20gb:3,gpu:H100_3g.40gb:2,gpu:H100_7g.80gb:1
PartitionName=gpu_H100 Nodes=node16 Default=NO MaxTime=INFINITE State=UP
gres.conf
#AutoDetect=nvml
# GPU 0 MIG 0 /proc/driver/nvidia/capabilities/gpu0/mig/gi7/access
NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap66,/dev/nvidia-caps/nvidia-cap67 CPUs=0-55
# GPU 0 MIG 1 /proc/driver/nvidia/capabilities/gpu0/mig/gi8/access
NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap75,/dev/nvidia-caps/nvidia-cap76 CPUs=0-55
# GPU 0 MIG 2 /proc/driver/nvidia/capabilities/gpu0/mig/gi9/access
NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap84,/dev/nvidia-caps/nvidia-cap85 CPUs=0-55
# GPU 0 MIG 3 /proc/driver/nvidia/capabilities/gpu0/mig/gi10/access
NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap93,/dev/nvidia-caps/nvidia-cap94 CPUs=0-55
# GPU 0 MIG 4 /proc/driver/nvidia/capabilities/gpu0/mig/gi11/access
NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap102,/dev/nvidia-caps/nvidia-cap103 CPUs=0-55
# GPU 0 MIG 5 /proc/driver/nvidia/capabilities/gpu0/mig/gi12/access
NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap111,/dev/nvidia-caps/nvidia-cap112 CPUs=0-55
# GPU 0 MIG 6 /proc/driver/nvidia/capabilities/gpu0/mig/gi13/access
NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap120,/dev/nvidia-caps/nvidia-cap121 CPUs=0-55
# GPU 1 MIG 0 /proc/driver/nvidia/capabilities/gpu1/mig/gi3/access
NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap165,/dev/nvidia-caps/nvidia-cap166 CPUs=0-55
# GPU 1 MIG 1 /proc/driver/nvidia/capabilities/gpu1/mig/gi5/access
NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap183,/dev/nvidia-caps/nvidia-cap184 CPUs=0-55
# GPU 1 MIG 2 /proc/driver/nvidia/capabilities/gpu1/mig/gi6/access
NodeName=node16 Name=gpu Type=H100_2g.20gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap192,/dev/nvidia-caps/nvidia-cap193 CPUs=0-55
# GPU 1 MIG 3 /proc/driver/nvidia/capabilities/gpu1/mig/gi9/access
NodeName=node16 Name=gpu Type=H100_1g.10gb MultipleFiles=/dev/nvidia1,/dev/nvidia-caps/nvidia-cap219,/dev/nvidia-caps/nvidia-cap220 CPUs=0-55
# GPU 2 MIG 0 /proc/driver/nvidia/capabilities/gpu2/mig/gi1/access
NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap282,/dev/nvidia-caps/nvidia-cap283 CPUs=0-55
# GPU 2 MIG 1 /proc/driver/nvidia/capabilities/gpu2/mig/gi2/access
NodeName=node16 Name=gpu Type=H100_3g.40gb MultipleFiles=/dev/nvidia2,/dev/nvidia-caps/nvidia-cap291,/dev/nvidia-caps/nvidia-cap292 CPUs=0-55
# GPU 3 MIG 0 /proc/driver/nvidia/capabilities/gpu3/mig/gi0/access
NodeName=node16 Name=gpu Type=H100_7g.80gb MultipleFiles=/dev/nvidia3,/dev/nvidia-caps/nvidia-cap408,/dev/nvidia-caps/nvidia-cap409 CPUs=0-55
[2024-01-19T14:31:13.701] debug: gres/gpu: init: loaded
[2024-01-19T14:31:13.701] debug: gpu/generic: init: init: GPU Generic plugin loaded
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_1g.10gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_2g.20gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1
[2024-01-19T14:31:13.701] Gres Name=gpu Type=H100_3g.40gb Count=1
[2024-01-19T14:31:13.702] Gres Name=gpu Type=H100_7g.80gb Count=1
Best,
Drazen Jalsovec