<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>This should be fine assuming you don't mind the mismatch in CPU
speeds. Unless the codes are super sensitive to topology things
should be okay as modern IB is wicked fast.</p>
<p><br>
</p>
<p>In our environment here we have a variety of different hardware
types all networked together on the same IB fabric. That said we
create partitions for different hardware types and we don't have a
queue that schedules across both, though we do have a backfill
serial queue that underlies everything. All of that though is
scheduled via a single scheduler with a single topology.conf
governing it all. We also have all our internode IP comms going
over our IB fabric and it works fine.</p>
<p><br>
</p>
<p>-Paul Edmon-<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 12/7/2021 11:05 AM, David Baker
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:LO4P265MB4264A7ED27C34CB32D55D298FE6E9@LO4P265MB4264.GBRP265.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Hello,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
These days we have now enabled topology aware scheduling on our
Slurm cluster. One part of the cluster consists of two racks of
AMD compute nodes. These racks are, now, treated as separate
entities by Slurm. Soon, we may add another set of AMD nodes
with slightly difference CPU specs to the existing nodes. We'll
aim to balance the new nodes across the racks re cooling/heating
requirements. The new nodes will be controlled by a new
partition.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Does anyone know if it is possible to regard the two racks as a
single entity (by connecting the InfiniBand switches together),
and so schedule jobs across the resources in the racks with no
loss efficiency. I would be grateful for your comments and
ideas, please. The alternative is to put all the new nodes in a
completely new rack, but that does mean that we'll have purchase
some new Ethernet and IB switches. We are not happy, by the way,
to have node/switch connections across racks. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Best regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
David</div>
</blockquote>
</body>
</html>