[slurm-users] Usage gathering for GPUs

Christopher Samuel chris at csamuel.org
Wed May 24 19:45:48 UTC 2023

On 5/24/23 11:39 am, Fulton, Ben wrote:

> Hi,

Hi Ben,

> The release notes for 23.02 say “Added usage gathering for gpu/nvml 
> (Nvidia) and gpu/rsmi (AMD) plugins”.
> How would I go about enabling this?

I can only comment on the nvidia side (as those are the GPUs we have) 
but for that you need Slurm built with NVML support and running with 
"Autodetect=NVML" in gres.conf and then that information is stored in 
slurmdbd as part of the TRES usage data.

For example to grab a job step for a test code I ran the other day:

csamuel at perlmutter:login01:~> sacct -j 9285567.0 -Pno TRESUsageInAve | 
tr , \\n | fgrep gpu

Hope that helps!

All the best,
Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA

More information about the slurm-users mailing list