We have a node with 8 H100 GPUs that are split into MIG instances. We are using cgroups. This seems to work fine. Users can do something like

sbatch --gres="gpu:1g.10gb:1"...

and the job starts on the node with the gpus and cuda visible devices and the pytorch debug shows that the cgroup only gives them the gpu they asked for.

In the accounting database, jobs in the job table always have the "gres_used" column be empty. I'd expect to see "gpu:1g.10gb:1" appearing for the job above.

I have this set in slurm.conf

AccountingStorageTRES=gres/gpu

How can I see what gres was requested with the job ? At the moment I only see something like this in AllocTres

billing=1,cpu=1,gres/gpu=1,mem=8G,node=1

and can't see any way to see what the specific MIG gpu asked for was. This is related to the email from Richard Lefebvre dated 7th June 2023 entitled "Billing/accounting for MIGs is not working". As far as I can see this got no replies.

We are running slurm version 23.11.6.

Regards,