<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000">Hi, Matt --</div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000">There is the 'exclude' flag, '-x' or '--exclude', that can take a list of nodes to exclude from consideration. So it's selecting from a range, but by denoting the nodes to exclude from the range. The nodes may be specified on the command line, or denoted in a file that is referenced. For example,</div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000"><p dir="ltr" class="gmail-CDt4Ke gmail-zfr3Q"><span style="font-family:monospace">#SBATCH --exclude ./exclude-avx.list</span> </p><p dir="ltr" class="gmail-CDt4Ke gmail-zfr3Q">This approach is recommended to our researchers for running Gaussian compiled with support for AVX2 instructions, because we do not define features to identify nodes by instruction-set. <br></p><p class="gmail-CDt4Ke gmail-zfr3Q">Cheers,</p><p class="gmail-CDt4Ke gmail-zfr3Q">~ Em<br></p><p dir="ltr" class="gmail-CDt4Ke gmail-zfr3Q"><br></p><p dir="ltr" class="gmail-CDt4Ke gmail-zfr3Q"><br></p></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;color:#000000"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 14, 2021 at 12:13 PM Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <<a href="mailto:matthew.thompson@nasa.gov" target="_blank">matthew.thompson@nasa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Max,<br>
<br>
Well, that was just an example. I also was doing the same things with like 125 node runs. Obviously, that's beyond a rack and your chances of hitting a down node increases! I guess I figured if I can under-specify a nodelist, maybe I could over-specify it!<br>
<br>
It's possible that using the topology bits could do it, but I'm also just a basic end-user with not too much knowledge of the system setup. This was the first time I even tried --nodelist (with help from the admins) so I was approaching it naively as you saw. <br>
<br>
Or, I suppose, is there a flag that one can pass to sbatch that gives the user a warning? That is: <br>
<br>
Dear user, the allocation requested contains a node in a downed state. <br>
This allocation will be PENDING for a while. You might want to rethink this.<br>
<br>
I guess SLURM knows all the downed nodes, so maybe? (But then again, maybe sbatch would get magnitudes slower if it had to query a database of all nodes and do checks to see this...)<br>
<br>
Matt<br>
<br>
PS: Or I guess I could stare at Ole's cool SLURM tools and figure out a way to have my own "job checker". Find the downed nodes, parse 'scontrol show job 1234', and display possible/impossible jobs :D<br>
<br>
-- <br>
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst<br>
NASA GSFC, Global Modeling and Assimilation Office<br>
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771<br>
Phone: 301-614-6712 Fax: 301-614-6246<br>
<a href="http://science.gsfc.nasa.gov/sed/bio/matthew.thompson" rel="noreferrer" target="_blank">http://science.gsfc.nasa.gov/sed/bio/matthew.thompson</a><br>
<br>
On 7/14/21, 1:42 PM, "slurm-users on behalf of Max Voit" <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a> on behalf of <a href="mailto:max.voit_mlsl@with-eyes.net" target="_blank">max.voit_mlsl@with-eyes.net</a>> wrote:<br>
<br>
On Wed, 14 Jul 2021 17:04:45 +0000<br>
"Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]"<br>
<<a href="mailto:matthew.thompson@nasa.gov" target="_blank">matthew.thompson@nasa.gov</a>> wrote:<br>
<br>
> Namely, I needed say, 20 nodes on a cluster on the same rack<br>
> ...<br>
> So, my question is, is there a way to say, "Please give me X nodes<br>
> inside this specific range of nodes?"<br>
<br>
Is the requirement actually the nodes being in the same rack, or rather<br>
being connected to the same switch? For the latter: If you specify a<br>
topology.conf file you can use --switch=... , c.f.<br>
<a href="https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Ftopology.html&data=04%7C01%7Cmatthew.thompson%40nasa.gov%7C3769db7ffe274ecca56a08d946eeb9cc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637618813398627878%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=nT%2FVHMkQeLZwnQq3ynNNDbvDjlmJ9e%2F6%2FyYVS89%2BMhc%3D&reserved=0" rel="noreferrer" target="_blank">https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Ftopology.html&data=04%7C01%7Cmatthew.thompson%40nasa.gov%7C3769db7ffe274ecca56a08d946eeb9cc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637618813398627878%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=nT%2FVHMkQeLZwnQq3ynNNDbvDjlmJ9e%2F6%2FyYVS89%2BMhc%3D&reserved=0</a><br>
<br>
Best,<br>
Max<br>
<br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><span style="font-family:verdana,sans-serif"><span style="font-size:small">E.M. (Em) Dragowsky, Ph.D.</span><br></span></div><div><span style="font-family:verdana,sans-serif"><font size="2">Research Computing -- UTech</font></span></div><div><span style="font-family:verdana,sans-serif"><font size="2">Case Western Reserve University<br></font></span></div><div><font size="2" face="comic sans ms, sans-serif"><span style="font-family:verdana,sans-serif">(216) 368-0082</span><br></font></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>