I seem to have run into an edge case where I’m able to oversubscribe a specific subset of GPUs on one host in particular.

Slurm 22.05.8 Ubuntu 20.04 cgroups v1 (ProctrackType=proctrack/cgroup)

It seems to be partly a corner case with a couple of caveats. This host has 2 different GPU types in the same host. All GPUs are configured with 6 shards/gpu.

NodeName=gpu03 CPUs=88 RealMemory=768000 Sockets=2 CoresPerSocket=22 ThreadsPerCore=2 State=UNKNOWN Feature=avx512 Gres=gpu:rtx:2,gpu:p40:6,shard:48

My gres.conf is auto-nvml, with nothing else in the file.

AutoDetect=nvml

Schedule and select from slurm.conf

SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_CPU_Memory GresTypes=gpu,shard

Here’s a copy of slurmd’s log starting up with debug logging: https://pastebin.com/fbriTAZD https://pastebin.com/fbriTAZD I checked there to make sure that it was allocating an even 6 shards per GPU as intended, and it did. The only thing odd about the rtx cards is that they have an NVLink bridge between them, but I wouldn’t expect that to have any impact with respect to shards.

So far it only appears to manifest if anything is using the named gpu (--gres gpu:rtx:1), but not as a general gres (--gres gpu:1). It also doesn’t occur with any count of named p40 gres either, only with the rtx gres.

The below was as “high” as I was able to capture it oversubscribed in very quick testing. You can see there are 2 rtx and 43 shards allocated, which should be an impossible number, if we correctly assume an rtx==6 shards, and with 48 total shards, that should reduce the available pool of shards to 36 shards.

NodeName=gpu03 Arch=x86_64 CoresPerSocket=22 CPUAlloc=88 CPUEfctv=88 CPUTot=88 CPULoad=0.27 AvailableFeatures=avx512 ActiveFeatures=avx512 Gres=gpu:p40:6(S:0),gpu:rtx:2(S:0),shard:p40:36(S:0),shard:rtx:12(S:0) NodeAddr=gpu03 NodeHostName=gpu03 Version=22.05.8 OS=Linux 5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023 RealMemory=768000 AllocMem=45056 FreeMem=563310 Sockets=2 Boards=1 State=ALLOCATED+RESERVED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpu,gpu-prod BootTime=2024-02-02T14:17:07 SlurmdStartTime=2024-02-14T16:28:17 LastBusyTime=2024-02-14T16:38:03 CfgTRES=cpu=88,mem=750G,billing=88,gres/gpu=8,gres/gpu:p40=6,gres/gpu:rtx=2,gres/shard=48 AllocTRES=cpu=88,mem=44G,gres/gpu=2,gres/gpu:rtx=2,gres/shard=43 CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

The above is 1 rtx:2 jobs, and 43 shard:1 jobs, which is as many as I can seem to push through with the given cpu cores on the system, I expect if there were 98-100 cores I could completely oversubscribe it.

If I do (1) rtx:2 + (24) shard:2 it actually limits it correctly to 36 shards, which is odd because the case that caused us to discover this was when 2 rtx:1 jobs were running for multiple days, and somehow shards got scheduled on those gpus.

AllocTRES=cpu=40,mem=20G,gres/gpu=2,gres/gpu:rtx=2,gres/shard=36

If I flip it around to shards before rtx, then I’m unable to oversubscribe it. If I add any P40 jobs before the rtx and then shard jobs, also unable to oversubscribe.

This sounded familiar to Bug 16484 https://bugs.schedmd.com/show_bug.cgi?id=16484 although they claimed it was resolved by using auto-nvml which would preclude this node.

Here’s my very simple bash script that just srun's a sleep 20 so that it runs long enough to observe it.

#!/bin/bash #rtx 2 for((i=1;i<=1;++i)) ; do srun --gres gpu:rtx:2 --reservation=shardtest -J "RTX $i" sleep 20 & done sleep 1

#shard 48 for((i=1;i<=43;++i)) ; do srun --gres shard:1 --reservation=shardtest -J "shard $i" sleep 20 & done sleep 5 ; scontrol show node gpu03

Curious if anyone has any ideas beyond try a newer release?

Thanks, Reed