Hello, We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff? It will be great to know how much GPU resources the users request and compare with how much GPU resources they use. Kind regards. [image: Vicomtech] <https://www.vicomtech.org> Josu Lazkano Lete Systems Manager Infrastructures and General Services jlazkano@vicomtech.org +(34) 943 30 92 30 The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it. Avoid printing this message if it is not strictly necessary.
Well you do not say which type of GPU you use... If you use AMD this may be useful https://github.com/ROCm/device-metrics-exporter On Tue, 2 Sept 2025 at 14:41, Josu Lazkano Lete via slurm-users < slurm-users@lists.schedmd.com> wrote:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use.
Kind regards.
[image: Vicomtech] <https://www.vicomtech.org>
Josu Lazkano Lete Systems Manager Infrastructures and General Services jlazkano@vicomtech.org +(34) 943 30 92 30
The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it. Avoid printing this message if it is not strictly necessary.
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Hi, Josu Lazkano Lete via slurm-users <slurm-users@lists.schedmd.com> writes:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use.
Various sites have produced their own versions of 'seff'-like programs. We currently use https://github.com/PrincetonUniversity/jobstats which reports CPU, memory and GPU utility as well as providing suggestions to users about the amount of resources users should request for similar future jobs. Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin
Brown also uses jobstats, with prometheus. On Tue, Sep 2, 2025 at 10:04 AM Loris Bennett via slurm-users < slurm-users@lists.schedmd.com> wrote:
Hi,
Josu Lazkano Lete via slurm-users <slurm-users@lists.schedmd.com> writes:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use.
Various sites have produced their own versions of 'seff'-like programs. We currently use
https://github.com/PrincetonUniversity/jobstats
which reports CPU, memory and GPU utility as well as providing suggestions to users about the amount of resources users should request for similar future jobs.
Cheers,
Loris
-- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Thanks for all your replies, we will check them. We use Nvidia GPUs. I will inform you about the updates. Best regards. [image: Vicomtech] <https://www.vicomtech.org> Josu Lazkano Lete Systems Manager Infrastructures and General Services jlazkano@vicomtech.org +(34) 943 30 92 30 The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it. Avoid printing this message if it is not strictly necessary. El mar, 2 sept 2025 a las 16:35, Fulcomer, Samuel via slurm-users (< slurm-users@lists.schedmd.com>) escribió:
Brown also uses jobstats, with prometheus.
On Tue, Sep 2, 2025 at 10:04 AM Loris Bennett via slurm-users < slurm-users@lists.schedmd.com> wrote:
Hi,
Josu Lazkano Lete via slurm-users <slurm-users@lists.schedmd.com> writes:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use.
Various sites have produced their own versions of 'seff'-like programs. We currently use
https://github.com/PrincetonUniversity/jobstats
which reports CPU, memory and GPU utility as well as providing suggestions to users about the amount of resources users should request for similar future jobs.
Cheers,
Loris
-- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
On 9/2/25 10:01 AM, Loris Bennett via slurm-users wrote:
Hi,
Josu Lazkano Lete via slurm-users <slurm-users@lists.schedmd.com> writes:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use. Various sites have produced their own versions of 'seff'-like programs. We currently use
https://github.com/PrincetonUniversity/jobstats
which reports CPU, memory and GPU utility as well as providing suggestions to users about the amount of resources users should request for similar future jobs.
Cheers,
Loris
Both seff and jobstats were created by the same group of people at Princeton University. Prentice
On 9/10/25 11:37 AM, Prentice Bisbal wrote:
On 9/2/25 10:01 AM, Loris Bennett via slurm-users wrote:
Hi,
Josu Lazkano Lete via slurm-users <slurm-users@lists.schedmd.com> writes:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use. Various sites have produced their own versions of 'seff'-like programs. We currently use
https://github.com/PrincetonUniversity/jobstats
which reports CPU, memory and GPU utility as well as providing suggestions to users about the amount of resources users should request for similar future jobs.
Cheers,
Loris
Both seff and jobstats were created by the same group of people at Princeton University.
Prentice
I just e-mailed one of the developers/maintainers of seff and jobstats. seff is being deprecated because it uses the Slurm Perl API which SchedMD is moving away from. For their own in-house use, jobstats has largely replaced seff, but moving to jobstats requires setting up additional machinery which seff doesn't require, so jobstats is not a 1:1 replacement for seff. Unfortunately, it doesn't look like GPU support will be added to seff. Prentice
That's sad that seff is being deprecated due to dropping the perl api. It would be great if it was reimplemented via the C API, REST API, or via command parsing using python. As I understand it OSC has implemented a seff-gpu along the same lines of Princeton JobStats (they require the same collectors) but the source code is not public. It would be great to merge that into the general job stats platform. Suffice it to say seff is a really useful command and it would be a shame to see it die, it should in fact be extended to include GPUs. -Paul Edmon- On 9/10/25 12:01 PM, Prentice Bisbal via slurm-users wrote:
On 9/10/25 11:37 AM, Prentice Bisbal wrote:
On 9/2/25 10:01 AM, Loris Bennett via slurm-users wrote:
Hi,
Josu Lazkano Lete via slurm-users <slurm-users@lists.schedmd.com> writes:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use. Various sites have produced their own versions of 'seff'-like programs. We currently use
https://github.com/PrincetonUniversity/jobstats
which reports CPU, memory and GPU utility as well as providing suggestions to users about the amount of resources users should request for similar future jobs.
Cheers,
Loris
Both seff and jobstats were created by the same group of people at Princeton University.
Prentice
I just e-mailed one of the developers/maintainers of seff and jobstats. seff is being deprecated because it uses the Slurm Perl API which SchedMD is moving away from. For their own in-house use, jobstats has largely replaced seff, but moving to jobstats requires setting up additional machinery which seff doesn't require, so jobstats is not a 1:1 replacement for seff.
Unfortunately, it doesn't look like GPU support will be added to seff.
Prentice
That's sad that seff is being deprecated due to dropping the perl api. Agreed, but at least the source code is available, so some intrepid spirit can look at the existing code and port it do a different API. When I wore a younger man's clothes, I probably would have foolishly volunteered to do something like this.
Prentice On 9/10/25 12:08 PM, Paul Edmon via slurm-users wrote:
That's sad that seff is being deprecated due to dropping the perl api. It would be great if it was reimplemented via the C API, REST API, or via command parsing using python.
As I understand it OSC has implemented a seff-gpu along the same lines of Princeton JobStats (they require the same collectors) but the source code is not public. It would be great to merge that into the general job stats platform.
Suffice it to say seff is a really useful command and it would be a shame to see it die, it should in fact be extended to include GPUs.
-Paul Edmon-
On 9/10/25 12:01 PM, Prentice Bisbal via slurm-users wrote:
On 9/10/25 11:37 AM, Prentice Bisbal wrote:
On 9/2/25 10:01 AM, Loris Bennett via slurm-users wrote:
Hi,
Josu Lazkano Lete via slurm-users <slurm-users@lists.schedmd.com> writes:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use. Various sites have produced their own versions of 'seff'-like programs. We currently use
https://github.com/PrincetonUniversity/jobstats
which reports CPU, memory and GPU utility as well as providing suggestions to users about the amount of resources users should request for similar future jobs.
Cheers,
Loris
Both seff and jobstats were created by the same group of people at Princeton University.
Prentice
I just e-mailed one of the developers/maintainers of seff and jobstats. seff is being deprecated because it uses the Slurm Perl API which SchedMD is moving away from. For their own in-house use, jobstats has largely replaced seff, but moving to jobstats requires setting up additional machinery which seff doesn't require, so jobstats is not a 1:1 replacement for seff.
Unfortunately, it doesn't look like GPU support will be added to seff.
Prentice
-- Prentice Bisbal HPC Systems Engineer III Computational & Information Systems Laboratory (CISL) NSF National Center for Atmospheric Research (NSF NCAR) https://www.cisl.ucar.edu https://ncar.ucar.edu
On Tue, 2025-09-02 at 15:38 +0200, Josu Lazkano Lete via slurm-users wrote:
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use.
Hi Josu, I have forked sps and added support for Nvidia and AMD GPUs https://github.com/mhagdorn/sps Regards magnus -- Dr. Magnus Hagdorn Charité – Universitätsmedizin Berlin Geschäftsbereich IT | Scientific Computing Campus Charité Mitte BALTIC - Invalidenstraße 120/121 10115 Berlin magnus.hagdorn@charite.de https://www.charite.de HPC Helpdesk: sc-hpc-helpdesk@charite.de
participants (7)
-
Fulcomer, Samuel -
Hagdorn, Magnus Karl Moritz -
John Hearns -
Josu Lazkano Lete -
Loris Bennett -
Paul Edmon -
Prentice Bisbal