<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:10.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<style type="text/css">.style1 {font-family: "Times New Roman";}</style></head><body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="p1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Ahhhh, that was it. The failure state was persisting after the problem was fixed. This loop put all my nodes back into idle state.<br>
<br>
</span><span class="s1">for H in {01..08}; do scontrol update NodeName=sjc01enadsapp$H State=UNDRAIN; done</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks, David.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">-jimk<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of slurm-users-request@lists.schedmd.com <slurm-users-request@lists.schedmd.com><br>
<b>Date: </b>Tuesday, May 3, 2022 at 5:20 PM<br>
<b>To: </b>slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>[EXTERNAL] slurm-users Digest, Vol 55, Issue 5<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">WARNING: This e-mail is sent from outside the organization DO NOT CLICK on any links or open attachments unless you trust the sender<br>
<br>
Send slurm-users mailing list submissions to<br>
slurm-users@lists.schedmd.com<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users">https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users</a><br>
or, via email, send a message with subject or body 'help' to<br>
slurm-users-request@lists.schedmd.com<br>
<br>
You can reach the person managing the list at<br>
slurm-users-owner@lists.schedmd.com<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of slurm-users digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: gres/gpu count lower than reported (David Henkemeyer)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 3 May 2022 14:05:45 -0700<br>
From: David Henkemeyer <david.henkemeyer@gmail.com><br>
To: Slurm User Community List <slurm-users@lists.schedmd.com><br>
Subject: Re: [slurm-users] gres/gpu count lower than reported<br>
Message-ID:<br>
<CABjsmAH+z=xA9_QxvRyg-1uSj_YPWzamCbE5tybJcsm6zfsN0g@mail.gmail.com><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
I have found that the "reason" field doesn't get updated after you correct<br>
the issue. For me, its only when I move the node back to the idle state,<br>
that the reason field is then reset. So, assuming /dev/nvidia[0-3] is<br>
correct (I've never seen otherwise with nvidia GPUs), then try taking them<br>
back into the idle state. Also, keep an eye on the slurmctld and slurmd<br>
logs. They usually are quite helpful to highlight what the issue is.<br>
<br>
David<br>
<br>
On Tue, May 3, 2022 at 11:50 AM Jim Kavitsky <JimKavitsky@lucidmotors.com><br>
wrote:<br>
<br>
> Hello Fellow Slurm Admins,<br>
><br>
><br>
><br>
> I have a new Slurm installation that was working and running basic test<br>
> jobs until I added gpu support. My worker nodes are now all in drain state,<br>
> with gres/gpu count reported lower than configured (0 < 4)<br>
><br>
><br>
><br>
> This is in spite of the fact that nvidia-smi reports all four A100?s as<br>
> active on each node. I have spent a good chunk of a week googling around<br>
> for the solution to this, and trying variants of the gpu config<br>
> lines/restarting daemons without any luck.<br>
><br>
><br>
><br>
> The relevant lines from my current config files are below. The head node<br>
> and all workers have the same gres.conf and slurm.conf files. Can anyone<br>
> suggest anything else I should be looking at or adding? I?m guessing that<br>
> this is a problem that many have faced, and any guidance would be greatly<br>
> appreciated.<br>
><br>
><br>
><br>
> root@sjc01enadsapp00:/etc/slurm-llnl# grep gpu slurm.conf<br>
><br>
> GresTypes=*gpu*<br>
><br>
> NodeName=sjc01enadsapp0[1-8] RealMemory=2063731 Sockets=2<br>
> CoresPerSocket=16 ThreadsPerCore=2 Gres=*gpu*:4 State=UNKNOWN<br>
><br>
><br>
><br>
> root@sjc01enadsapp00:/etc/slurm-llnl# cat gres.conf<br>
><br>
> NodeName=sjc01enadsapp0[1-8] Name=gpu File=/dev/nvidia[0-3]<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> root@sjc01enadsapp00:~# sinfo -N -o "%.20N %.15C %.10t %.10m %.15P %.15G<br>
> %.75E"<br>
><br>
> NODELIST CPUS(A/I/O/T) STATE MEMORY PARTITION<br>
> GRES<br>
> REASON<br>
><br>
> sjc01enadsapp01 0/0/64/64 drain 2063731 Primary*<br>
> gpu:4 gres/gpu count reported lower than<br>
> configured (0 < 4)<br>
><br>
> sjc01enadsapp02 0/0/64/64 drain 2063731 Primary*<br>
> gpu:4 gres/gpu count reported lower than<br>
> configured (0 < 4)<br>
><br>
> sjc01enadsapp03 0/0/64/64 drain 2063731 Primary*<br>
> gpu:4 gres/gpu count reported lower than<br>
> configured (0 < 4)<br>
><br>
> sjc01enadsapp04 0/0/64/64 drain 2063731 Primary*<br>
> gpu:4 gres/gpu count reported lower than<br>
> configured (0 < 4)<br>
><br>
> sjc01enadsapp05 0/0/64/64 drain 2063731 Primary*<br>
> gpu:4 gres/gpu count reported lower than<br>
> configured (0 < 4)<br>
><br>
> sjc01enadsapp06 0/0/64/64 drain 2063731 Primary*<br>
> gpu:4 gres/gpu count reported lower than<br>
> configured (0 < 4)<br>
><br>
> sjc01enadsapp07 0/0/64/64 drain 2063731 Primary*<br>
> gpu:4 gres/gpu count reported lower than<br>
> configured (0 < 4)<br>
><br>
> sjc01enadsapp08 0/0/64/64 drain 2063731 Primary*<br>
> gpu:4 gres/gpu count reported lower than<br>
> configured (0 < 4)<br>
><br>
><br>
><br>
><br>
><br>
> root@sjc01enadsapp07:~# nvidia-smi<br>
><br>
> Tue May 3 18:41:34 2022<br>
><br>
><br>
> +-----------------------------------------------------------------------------+<br>
><br>
> | NVIDIA-SMI 470.103.01 Driver Version: 470.103.01 CUDA Version: 11.4<br>
> |<br>
><br>
><br>
> |-------------------------------+----------------------+----------------------+<br>
><br>
> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.<br>
> ECC |<br>
><br>
> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute<br>
> M. |<br>
><br>
> | | | MIG<br>
> M. |<br>
><br>
><br>
> |===============================+======================+======================|<br>
><br>
> | 0 NVIDIA A100-PCI... On | 00000000:17:00.0 Off |<br>
> 0 |<br>
><br>
> | N/A 42C P0 49W / 250W | 4MiB / 40536MiB | 0%<br>
> Default |<br>
><br>
> | | |<br>
> Disabled |<br>
><br>
><br>
> +-------------------------------+----------------------+----------------------+<br>
><br>
> | 1 NVIDIA A100-PCI... On | 00000000:65:00.0 Off |<br>
> 0 |<br>
><br>
> | N/A 41C P0 48W / 250W | 4MiB / 40536MiB | 0%<br>
> Default |<br>
><br>
> | | |<br>
> Disabled |<br>
><br>
><br>
> +-------------------------------+----------------------+----------------------+<br>
><br>
> | 2 NVIDIA A100-PCI... On | 00000000:CA:00.0 Off |<br>
> 0 |<br>
><br>
> | N/A 35C P0 44W / 250W | 4MiB / 40536MiB | 0%<br>
> Default |<br>
><br>
> | | |<br>
> Disabled |<br>
><br>
><br>
> +-------------------------------+----------------------+----------------------+<br>
><br>
> | 3 NVIDIA A100-PCI... On | 00000000:E3:00.0 Off |<br>
> 0 |<br>
><br>
> | N/A 38C P0 45W / 250W | 4MiB / 40536MiB | 0%<br>
> Default |<br>
><br>
> | | |<br>
> Disabled |<br>
><br>
><br>
> +-------------------------------+----------------------+----------------------+<br>
><br>
><br>
><br>
><br>
><br>
> +-----------------------------------------------------------------------------+<br>
><br>
> | Processes:<br>
> |<br>
><br>
> | GPU GI CI PID Type Process name GPU<br>
> Memory |<br>
><br>
> | ID ID Usage<br>
> |<br>
><br>
><br>
> |=============================================================================|<br>
><br>
> | 0 N/A N/A 2179 G /usr/lib/xorg/Xorg<br>
> 4MiB |<br>
><br>
> | 1 N/A N/A 2179 G /usr/lib/xorg/Xorg<br>
> 4MiB |<br>
><br>
> | 2 N/A N/A 2179 G /usr/lib/xorg/Xorg<br>
> 4MiB |<br>
><br>
> | 3 N/A N/A 2179 G /usr/lib/xorg/Xorg<br>
> 4MiB |<br>
><br>
><br>
> +-----------------------------------------------------------------------------+<br>
><br>
><br>
><br>
><br>
> This message and any attachments are Confidential Information, for the<br>
> exclusive use of the addressee and may be legally privileged. Any receipt<br>
> by anyone other than the intended addressee does not constitute a loss of<br>
> the confidential or privileged nature of the communication. Any other<br>
> distribution, use or reproduction is unauthorized and prohibited. If you<br>
> are not the intended recipient, please contact the sender by return<br>
> electronic mail and delete all copies of this communication<br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.schedmd.com/pipermail/slurm-users/attachments/20220503/935f4bf0/attachment.htm">http://lists.schedmd.com/pipermail/slurm-users/attachments/20220503/935f4bf0/attachment.htm</a>><br>
<br>
End of slurm-users Digest, Vol 55, Issue 5<br>
******************************************<o:p></o:p></span></p>
</div>
<br><br><p style="font-family: Verdana; font-size:10pt; color:#666666;"></p><p style="font-family: Helvetica; font-size:13.333px; color:#666666;">This message and any attachments are Confidential Information, for the exclusive use of the addressee and may be legally privileged. Any receipt by anyone other than the intended addressee does not constitute a loss of the confidential or privileged nature of the communication. Any other distribution, use or reproduction is unauthorized and prohibited. If you are not the intended recipient, please contact the sender by return electronic mail and delete all copies of this communication</p></body></html>