Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use.
Kind regards.
[image: Vicomtech] https://www.vicomtech.org
Josu Lazkano Lete Systems Manager Infrastructures and General Services jlazkano@vicomtech.org +(34) 943 30 92 30
The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it. Avoid printing this message if it is not strictly necessary.
Well you do not say which type of GPU you use...
If you use AMD this may be useful https://github.com/ROCm/device-metrics-exporter
On Tue, 2 Sept 2025 at 14:41, Josu Lazkano Lete via slurm-users < slurm-users@lists.schedmd.com> wrote:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use.
Kind regards.
[image: Vicomtech] https://www.vicomtech.org
Josu Lazkano Lete Systems Manager Infrastructures and General Services jlazkano@vicomtech.org +(34) 943 30 92 30
The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it. Avoid printing this message if it is not strictly necessary.
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Hi,
Josu Lazkano Lete via slurm-users slurm-users@lists.schedmd.com writes:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and compare with how much GPU resources they use.
Various sites have produced their own versions of 'seff'-like programs. We currently use
https://github.com/PrincetonUniversity/jobstats
which reports CPU, memory and GPU utility as well as providing suggestions to users about the amount of resources users should request for similar future jobs.
Cheers,
Loris
Brown also uses jobstats, with prometheus.
On Tue, Sep 2, 2025 at 10:04 AM Loris Bennett via slurm-users < slurm-users@lists.schedmd.com> wrote:
Hi,
Josu Lazkano Lete via slurm-users slurm-users@lists.schedmd.com writes:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it possible
to add GPU info in the seff?
It will be great to know how much GPU resources the users request and
compare with how much GPU resources they use.
Various sites have produced their own versions of 'seff'-like programs. We currently use
https://github.com/PrincetonUniversity/jobstats
which reports CPU, memory and GPU utility as well as providing suggestions to users about the amount of resources users should request for similar future jobs.
Cheers,
Loris
-- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Thanks for all your replies, we will check them.
We use Nvidia GPUs.
I will inform you about the updates.
Best regards.
[image: Vicomtech] https://www.vicomtech.org
Josu Lazkano Lete Systems Manager Infrastructures and General Services jlazkano@vicomtech.org +(34) 943 30 92 30
The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it. Avoid printing this message if it is not strictly necessary.
El mar, 2 sept 2025 a las 16:35, Fulcomer, Samuel via slurm-users (< slurm-users@lists.schedmd.com>) escribió:
Brown also uses jobstats, with prometheus.
On Tue, Sep 2, 2025 at 10:04 AM Loris Bennett via slurm-users < slurm-users@lists.schedmd.com> wrote:
Hi,
Josu Lazkano Lete via slurm-users slurm-users@lists.schedmd.com writes:
Hello,
We are looking to optimize the GPU jobs of our HPC users, is it
possible to add GPU info in the seff?
It will be great to know how much GPU resources the users request and
compare with how much GPU resources they use.
Various sites have produced their own versions of 'seff'-like programs. We currently use
https://github.com/PrincetonUniversity/jobstats
which reports CPU, memory and GPU utility as well as providing suggestions to users about the amount of resources users should request for similar future jobs.
Cheers,
Loris
-- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-
Fulcomer, Samuel -
John Hearns -
Josu Lazkano Lete -
Loris Bennett