<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Some additional information after enabling debug3 on slurmctld daemon:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Logs show that there are enough usable nodes for the job:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-11<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-12<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-13<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-14<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-15<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-16<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-17<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-18<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-19<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-20<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-21<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-22<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-23<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-24<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-25<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-26<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-27<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-28<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-29<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-30<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-31<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-32<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-33<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-34<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-35<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-36<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-37<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-38<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-39<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-40<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-41<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-42<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-43<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-44<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-45<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-46<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-47<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-48<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-49<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-50<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-51<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-52<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-53<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-54<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-55<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-56<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-57<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-58<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-59<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-60<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-61<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-62<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-63<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-64<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-65<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-66<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-67<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-68<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-69<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-70<o:p></o:p></p>
<p class="MsoNormal">[2021-04-01T10:39:14.400] debug2: found 1 usable nodes from config containing node-71<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But then the following line is in the log as well:<o:p></o:p></p>
<p class="MsoNormal">debug3: select_nodes: JobId=67171529 required nodes not avail<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Sajesh-<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Sajesh Singh<br>
<b>Sent:</b> Thursday, March 25, 2021 9:02 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Limit on number of nodes user able to request<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="line-height:12.0pt;background:#FFEB9C"><b><span style="font-size:10.0pt;color:black">EXTERNAL SENDER</span></b><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">No nodes in downed or drained state. These are nodes that are dynamically brought up and down via the powersave plugin. When the are taken offline due to being idle I believe the state is set to FUTURE by the powersave plugin.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">-Sajesh-<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com">slurm-users-bounces@lists.schedmd.com</a>>
<b>On Behalf Of </b>Brian Andrus<br>
<b>Sent:</b> Wednesday, March 24, 2021 11:02 PM<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> Re: [slurm-users] Limit on number of nodes user able to request<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="line-height:12.0pt;background:#FFEB9C"><b><span style="font-size:10.0pt;color:black">EXTERNAL SENDER</span></b><span style="font-size:10.0pt;color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p>Do 'sinfo -R' and see if you have any down or drained nodes.<o:p></o:p></p>
<p>Brian Andrus<o:p></o:p></p>
<div>
<p class="MsoNormal">On 3/24/2021 6:31 PM, Sajesh Singh wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Slurm 20.02<o:p></o:p></p>
<p class="MsoNormal">CentOS 8<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I just recently noticed a strange behavior when using the powersave plugin for bursting to AWS. I have a queue configured with 60 nodes, but if I submit a job to use all of the nodes I get the error:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">(Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">If I lower the job to request 50 nodes it gets submitted and runs with no problems. I do not have and associations or QOS limits in place that would limit the user. Any ideas as to what could be causing this limit of 50 nodes to be imposed?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">-Sajesh-<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</body>
</html>