Slurm 22.05.8

Ubuntu 20.04

cgroups v1 (ProctrackType=proctrack/cgroup)

It seems to be partly a corner case with a couple of caveats.

This host has 2 different GPU types in the same host.

All GPUs are configured with 6 shards/gpu.

NodeName=gpu03 CPUs=88 RealMemory=768000 Sockets=2 CoresPerSocket=22 ThreadsPerCore=2 State=UNKNOWN Feature=avx512 Gres=gpu:rtx:2,gpu:p40:6,shard:48

My gres.conf is auto-nvml, with nothing else in the file.

AutoDetect=nvml

Schedule and select from slurm.conf

SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_CPU_Memory

GresTypes=gpu,shard

Here’s a copy of slurmd’s log starting up with debug logging: https://pastebin.com/fbriTAZD

I checked there to make sure that it was allocating an even 6 shards per GPU as intended, and it did.

The only thing odd about the rtx cards is that they have an NVLink bridge between them, but I wouldn’t expect that to have any impact with respect to shards.

So far it only appears to manifest if anything is using the named gpu (--gres gpu:rtx:1), but not as a general gres (--gres gpu:1).

It also doesn’t occur with any count of named p40 gres either, only with the rtx gres.

The below was as “high” as I was able to capture it oversubscribed in very quick testing.

You can see there are 2 rtx and 43 shards allocated, which should be an impossible number, if we correctly assume an rtx==6 shards, and with 48 total shards, that should reduce the available pool of shards to 36 shards.

NodeName=gpu03 Arch=x86_64 CoresPerSocket=22
CPUAlloc=88 CPUEfctv=88 CPUTot=88 CPULoad=0.27
AvailableFeatures=avx512
ActiveFeatures=avx512
Gres=gpu:p40:6(S:0),gpu:rtx:2(S:0),shard:p40:36(S:0),shard:rtx:12(S:0)
NodeAddr=gpu03 NodeHostName=gpu03 Version=22.05.8
OS=Linux 5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023
RealMemory=768000 AllocMem=45056 FreeMem=563310 Sockets=2 Boards=1
State=ALLOCATED+RESERVED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
Partitions=gpu,gpu-prod
BootTime=2024-02-02T14:17:07 SlurmdStartTime=2024-02-14T16:28:17
LastBusyTime=2024-02-14T16:38:03
CfgTRES=cpu=88,mem=750G,billing=88,gres/gpu=8,gres/gpu:p40=6,gres/gpu:rtx=2,gres/shard=48
AllocTRES=cpu=88,mem=44G,gres/gpu=2,gres/gpu:rtx=2,gres/shard=43
CapWatts=n/a
CurrentWatts=0 AveWatts=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

The above is 1 rtx:2 jobs, and 43 shard:1 jobs, which is as many as I can seem to push through with the given cpu cores on the system, I expect if there were 98-100 cores I could completely oversubscribe it.

If I do (1) rtx:2 + (24) shard:2 it actually limits it correctly to 36 shards, which is odd because the case that caused us to discover this was when 2 rtx:1 jobs were running for multiple days, and somehow shards got scheduled on those gpus.

AllocTRES=cpu=40,mem=20G,gres/gpu=2,gres/gpu:rtx=2,gres/shard=36

If I flip it around to shards before rtx, then I’m unable to oversubscribe it.