[slurm-users] [EXTERNAL] Re: gpu utilization of a reserved node

[Greg Wickham]

Hi Purvesh,

With some caveats, you can do:

$ sacct -N <nodename> -X -S <start date> -E <end date> -P format=jobid,alloctres

And then post process the results with a scripting language.

The caveats? . . The -X above is returning the job allocation, which in your case it appears to be everything you need. However for a job or step that spans multiple nodes Slurm doesn’t save in the database what specific resources were allocated on each node.

“scontrol show job <jobid> -d” does display the node specific resource allocations, but this information is discarded during summarisation to Slurmdbd.

   -Greg

From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of Purvesh Parmar <purveshp0507 at gmail.com>
Date: Thursday, 5 May 2022 at 5:10 am
To: slurm-users at lists.schedmd.com <slurm-users at lists.schedmd.com>
Subject: [EXTERNAL] Re: [slurm-users] gpu utilization of a reserved node
Hi,

We have a node given to a group that has all the 2 GPUs in dedicated mode by setting reservation on the node  for 6 months. We want to find out GPU hours  weekly utilization of that particular reserved node. The node is not in to seperate partition.
Below command does not help in showing the allocated gpu hours and also does not show for a week duration.
sreport reservation utilization name=rizwan_res start=2022-03-28T10:00:00 end=2022-04-03T10:00:00

Please help.

Regards,
Purvesh

On Sat, 30 Apr 2022 at 15:57, Purvesh Parmar <purveshp0507 at gmail.com<mailto:purveshp0507 at gmail.com>> wrote:
Hello,

We have a node given to a group that has 2 GPUs in dedicated mode by setting reservation for 6 months. We want to find out GPU hours utilization weekly utilization of that particular reserved node. The node is not in to seperate partition.
Below command does not help in showing the allocated gpu hours and alos does not show for a week duration.
sreport reservation utilization name=rizwan_res start=2022-03-28T10:00:00 end=2022-04-03T10:00:00

Please help.

Regards,
Purvesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220507/116ea16b/attachment-0001.htm>