<div dir="ltr"><font face="arial, sans-serif">Hi,</font><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">I recently upgraded Slurm from 19.05.5 to 20.02.5 and I noticed that I am no longer able to use the scontrol update command on GRES unless the new count is 0. </font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">For example, if I have a GRES named "test" defined and in slurm.conf </font></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace">NodeName=node1 <span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">Gres=test:2</span></font></blockquote>





<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Initially, I have:</font></div><div>





<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="monospace">$ sinfo -n node1 -O gres -h</font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="monospace">$ test:2</font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="arial, sans-serif"><br></font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="arial, sans-serif">After scontrol update, count remains the same:</font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace">$ sudo scontrol update NodeName=node1 Gres=test:1</font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="monospace">$ sinfo -n node1 -O gres -h</font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="monospace">$ test:2</font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="arial, sans-serif"><br></font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="arial, sans-serif">Only updating to 0 seems to be working</font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace">$ sudo scontrol update NodeName=node1 Gres=test:0</font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="monospace">$ sinfo -n node1 -O gres -h</font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="monospace"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span></font></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="monospace">$ </font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="arial, sans-serif"><br></font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="arial, sans-serif">It does not show any error or output but nothing is updated.</font></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="arial, sans-serif">Non-zero Gres update was working for me on Slurm 19.05.5 and I did not see anything related in the release note.</font></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="arial, sans-serif"><br></font></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="arial, sans-serif">I only saw that both <a href="https://slurm.schedmd.com/archive/slurm-19.05.5/scontrol.html">19.05.5</a> and <a href="https://slurm.schedmd.com/scontrol.html">20.02.5</a> scontrol documentation mentions this:</font></p><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font face="arial, sans-serif"><span style="color:rgb(70,84,92)">Modification of GRES count associated with specific files (e.g. GPUs) is not allowed other than to set their count on a node to zero.</span></font></blockquote><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="arial, sans-serif">However, in my slurm.conf, Gres "test" is not specified with a file.</font></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="arial, sans-serif"><br></font></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="arial, sans-serif">Is this the expected behavior of Slurm 20.02.5? Is there any possible workaround (maybe explicitly clear out the 'file' field)?</font></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="arial, sans-serif"><br></font></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="arial, sans-serif">Any help or comments would be greatly appreciated!</font></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="arial, sans-serif"><br></font></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="arial, sans-serif">Best,</font></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="arial, sans-serif" style="">Chen</font></p></div></div>