[slurm-users] Usage gathering for GPUs

[Christopher Samuel]

On 5/24/23 11:39 am, Fulton, Ben wrote:

> Hi,

Hi Ben,

> The release notes for 23.02 say “Added usage gathering for gpu/nvml 
> (Nvidia) and gpu/rsmi (AMD) plugins”.
> 
> How would I go about enabling this?

I can only comment on the nvidia side (as those are the GPUs we have) 
but for that you need Slurm built with NVML support and running with 
"Autodetect=NVML" in gres.conf and then that information is stored in 
slurmdbd as part of the TRES usage data.

For example to grab a job step for a test code I ran the other day:

csamuel at perlmutter:login01:~> sacct -j 9285567.0 -Pno TRESUsageInAve | 
tr , \\n | fgrep gpu
gres/gpumem=493120K
gres/gpuutil=76

Hope that helps!

All the best,
Chris
-- 
Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA