<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Hi Rob, <br>
</p>
<p><br>
</p>
<p>Yes, those questions make sense. From what I understand, MIG
should essentially split the GPU so that they behave as separate
cards. Hence two different users should be able to use two
different MIG instances at the same time and also a single job
could use all 14 instances. The result you observed suggests that
MIG is a feature of the driver i.e lspci shows one device but
nvidia-smi shows 7 devices.<br>
</p>
<p><br>
</p>
<p>I haven't played around with this myself in slurm but would be
interested to know the answers. <br>
</p>
<p><br>
</p>
<p>Laurence <br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 15/11/2022 17:46, Groner, Rob wrote:<br>
</div>
<blockquote type="cite" cite="mid:BL0PR02MB4499E04AB1879240E2F8916E80049@BL0PR02MB4499.namprd02.prod.outlook.com">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
We have successfully used the nvidia-smi tool to take the 2
A100's in a node and split them into multiple GPU devices. In
one case, we split the 2 GPUS into 7 MIG devices each, so 14 in
that node total, and in the other case, we split the 2 GPUs into
2 MIG devices each, so 4 total in the node.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
From our limited testing so far, and from the "sinfo" output, it
appears that slurm might be considering all of the MIG devices
on the node to be in the same socket (even though the MIG
devices come from two separate graphics cards in the node). The
sinfo output says (S:0) after the 14 devices are shown,
indicating they're in socket 0. That seems to be preventing 2
different users from using MIG devices at the same time. Am I
wrong that having 14 MIG gres devices show up in slurm should
mean that, in theory, 14 different users could use one at the
same time?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
Even IF that doesn't work....if I have 14 devices spread across
2 physical GPU cards, can one user utilize all 14 for a single
job? I would hope that slurm would treat each of the MIG
devices as its own separate card, which would mean 14 different
jobs could run at the same time using their own particular MIG,
right?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
Do those questions make sense to anyone? <span id="🙂">🙂</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
<span>Rob</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);" class="elementToProof">
<span><br>
</span></div>
</blockquote>
</body>
</html>