<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
No, I can't submit more than 7 individual jobs and have them all run, the jobs after the first 7 will go to pending until the first 7 finish.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
And it's not a limit (at least, not of "7"), because here's the same problem but with a node configured for 2x3g.20gb per card (2 cards, so, 4 total MIG gpus in the node)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted0 ContentPasted1" style="margin: 0px; color: rgb(0, 0, 0);">[rug262@testsch (RC) slurm] sinfo -o "%20N %10c %10m %25f %40G "</span><br class="ContentPasted1">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted0 ContentPasted1" style="margin: 0px; color: rgb(0, 0, 0);">NODELIST CPUS MEMORY AVAIL_FEATURES GRES </span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;" class="ContentPasted1">t-gc-1201 48 358400 3gc20gb gpu:nvidia_a100_3g.20gb:4(S:0)</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
So, there are 4 of them on that node</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0 ContentPasted2">
[rug262@testsch (RC) slurm] sbatch --gpus=1 --cpus-per-task=2 --partition=debug --nodelist=t-gc-1201 --wrap="sleep 100"
<div class="ContentPasted0"><br>
</div>
<div class="ContentPasted0">I submit 3 jobs, each asking for 1 gpu from that node</div>
<div class="ContentPasted0"><br>
</div>
<div class="ContentPasted0"><br>
</div>
<div class="ContentPasted0">[rug262@testsch (RC) slurm] squeue</div>
<div class="ContentPasted0"> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)</div>
<div class="ContentPasted0"> 5049 debug wrap rug262 PD 0:00 1 (Resources)</div>
<div class="ContentPasted0"> 5048 debug wrap rug262 R 0:09 1 t-gc-1201</div>
<div class="ContentPasted0"> 5047 <span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">
debug</span> wrap rug262 R 0:31 1 t-gc-1201</div>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
The first 2 go fine, but any after that go to pending, even though there should be 4 available (according to sinfo output)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
Rob</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div id="signature_bookmark"></div>
<div id="appendonsend"></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Yair Yarom <irush@cs.huji.ac.il><br>
<b>Sent:</b> Thursday, November 17, 2022 8:19 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] NVIDIA MIG question</font>
<div> </div>
</div>
<div>
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="border:0; display:table; width:100%; table-layout:fixed; border-collapse:seperate; float:none">
<tbody style="display:block">
<tr>
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="padding: 7px 2px; background-color: rgb(166, 166, 166);">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" color="#212121" style="width: 100%; padding: 7px 5px 7px 15px; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; overflow-wrap: break-word; background-color: rgb(234, 234, 234); color: rgb(33, 33, 33);">
<div>You don't often get email from irush@cs.huji.ac.il. <a href="https://aka.ms/LearnAboutSenderIdentification" data-auth="NotApplicable">
Learn why this is important</a></div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" color="#212121" style="width: 75px; padding: 7px 5px; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; overflow-wrap: break-word; background-color: rgb(234, 234, 234); color: rgb(33, 33, 33);">
</td>
</tr>
</tbody>
</table>
<div>
<div dir="ltr">
<div>Can you request more than 7 single gpu jobs on the same node?<br>
</div>
<div>It could be that there's another limit you've encountered (e.g. memory or cpu), or some other limit (in the account, partition, or qos)<br>
</div>
<div><br>
</div>
<div>On our setup we're limiting jobs to 1 gpu per job (via partition qos), however we can use up all the MIGs with single gpu jobs.<br>
</div>
<br>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Wed, 16 Nov 2022 at 23:48, Groner, Rob <<a href="mailto:rug262@psu.edu" data-auth="NotApplicable">rug262@psu.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg-5570338526575159845">
<div dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
That does help, thanks for the extra info.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
If I have two separate GPU cards in the node, and I setup 7 MIGs on each card, for a total of 14 MIG "gpus" in the node...then, SHOULD I be able to salloc requesting, say 10 GPUs (7 from 1 card, 3 from the other)? Because I can't.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I can request up to 7 just fine. When I request more than that, it adds in other nodes to try to give me that, even though there are theoretically 14 on the one node. When I ask for 8, it gives me 7 from t-gc-1202 and then 1 from t-gc-1201. When I ask for
10, then it fails because it can't give me 10 without using 2 cards in one node.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
[<span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline;">rug262</span>@testsch ~ ]# sinfo -o "%20N %10c %10m %25f %50G "
<div>NODELIST CPUS MEMORY AVAIL_FEATURES GRES
</div>
<div>t-gc-1201 48 358400 3gc20gb gpu:nvidia_a100_3g.20gb:4(S:0)
</div>
<div>t-gc-1202 48 358400 1gc5gb gpu:nvidia_a100_1g.5gb:14(S:0)
</div>
<div><br>
</div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
[rug262@testsch (RC) ~] salloc --gpus=10 --account=1gc5gb --partition=sla-prio
<div><span>salloc: Job allocation 5015 has been revoked.</span><br>
</div>
<div>salloc: error: Job submit/allocate failed: Requested node configuration is not available</div>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Rob</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div id="x_m_-5570338526575159845appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-5570338526575159845divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" data-auth="NotApplicable">slurm-users-bounces@lists.schedmd.com</a>>
on behalf of Yair Yarom <<a href="mailto:irush@cs.huji.ac.il" data-auth="NotApplicable">irush@cs.huji.ac.il</a>><br>
<b>Sent:</b> Wednesday, November 16, 2022 3:48 AM<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" data-auth="NotApplicable">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> Re: [slurm-users] NVIDIA MIG question</font>
<div> </div>
</div>
<div>
<table width="100%" cellspacing="0" cellpadding="0" border="0" align="left" style="border:0px none; display:table; width:100%; table-layout:fixed; float:none">
<tbody style="display:block">
<tr>
<td cellpadding="7px 2px 7px 2px" width="1px" valign="middle" bgcolor="#A6A6A6" style="padding: 7px 2px; background-color: rgb(166, 166, 166);">
</td>
<td cellpadding="7px 5px 7px 15px" color="#212121" width="100%" valign="middle" bgcolor="#EAEAEA" style="width: 100%; padding: 7px 5px 7px 15px; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; background-color: rgb(234, 234, 234); color: rgb(33, 33, 33);">
<div>You don't often get email from <a href="mailto:irush@cs.huji.ac.il" data-auth="NotApplicable">
irush@cs.huji.ac.il</a>. <a href="https://aka.ms/LearnAboutSenderIdentification" data-auth="NotApplicable">
Learn why this is important</a></div>
</td>
<td cellpadding="7px 5px 7px 5px" color="#212121" width="75px" valign="middle" bgcolor="#EAEAEA" align="left" style="width: 75px; padding: 7px 5px; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; background-color: rgb(234, 234, 234); color: rgb(33, 33, 33);">
</td>
</tr>
</tbody>
</table>
<div>
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
<div>From what we observed, Slurm sees the MIGs each as a distinct gres/gpu. So you can have 14 jobs each using a different MIG.<br>
</div>
<div>However (unless something has changed in the past year), due to nvidia limitations, a single process can't access more than one MIG simultaneously (this is unrelated to Slurm). So while you can have a user request a Slurm job with 2 gpus (MIGs), they'll
have to run two distinct processes within that job in order to utilize those two MIGs.</div>
<div><br>
</div>
<div>HTH,</div>
<div><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Tue, 15 Nov 2022 at 23:42, Laurence <<a href="mailto:laurence.field@cern.ch" data-auth="NotApplicable">laurence.field@cern.ch</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<p>Hi Rob, <br>
</p>
<p><br>
</p>
<p>Yes, those questions make sense. From what I understand, MIG should essentially split the GPU so that they behave as separate cards. Hence two different users should be able to use two different MIG instances at the same time and also a single job could
use all 14 instances. The result you observed suggests that MIG is a feature of the driver i.e lspci shows one device but nvidia-smi shows 7 devices.<br>
</p>
<p><br>
</p>
<p>I haven't played around with this myself in slurm but would be interested to know the answers.
<br>
</p>
<p><br>
</p>
<p>Laurence <br>
</p>
<p><br>
</p>
<div>On 15/11/2022 17:46, Groner, Rob wrote:<br>
</div>
<blockquote type="cite">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
We have successfully used the nvidia-smi tool to take the 2 A100's in a node and split them into multiple GPU devices. In one case, we split the 2 GPUS into 7 MIG devices each, so 14 in that node total, and in the other case, we split the 2 GPUs into 2 MIG
devices each, so 4 total in the node.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
From our limited testing so far, and from the "sinfo" output, it appears that slurm might be considering all of the MIG devices on the node to be in the same socket (even though the MIG devices come from two separate graphics cards in the node). The sinfo
output says (S:0) after the 14 devices are shown, indicating they're in socket 0. That seems to be preventing 2 different users from using MIG devices at the same time. Am I wrong that having 14 MIG gres devices show up in slurm should mean that, in theory,
14 different users could use one at the same time?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Even IF that doesn't work....if I have 14 devices spread across 2 physical GPU cards, can one user utilize all 14 for a single job? I would hope that slurm would treat each of the MIG devices as its own separate card, which would mean 14 different jobs could
run at the same time using their own particular MIG, right?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Do those questions make sense to anyone? <span id="x_m_-5570338526575159845x_m_2940866027957411465🙂">
🙂</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span>Rob</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span><br>
</span></div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<pre style="font-family:monospace"> <span style="color: rgb(133, 12, 27);">/|</span> |
<span style="color: rgb(133, 12, 27);">\/</span> | <span style="color: rgb(51, 88, 104); font-weight: bold;">Yair Yarom </span><span style="color: rgb(51, 88, 104);">| System Group (DevOps)</span>
<span style="color: rgb(92, 181, 149);">[]</span> | <span style="color: rgb(51, 88, 104); font-weight: bold;">The Rachel and Selim Benin School</span>
<span style="color: rgb(92, 181, 149);">[]</span> <span style="color: rgb(133, 12, 27);">/\</span> | <span style="color: rgb(51, 88, 104); font-weight: bold;">of Computer Science and Engineering</span>
<span style="color: rgb(92, 181, 149);">[]</span><span style="color: rgb(0, 161, 146);">//</span><span style="color: rgb(133, 12, 27);">\</span><span style="color: rgb(133, 12, 27);">\</span><span style="color: rgb(49, 154, 184);">/</span> | <span style="color: rgb(51, 88, 104);">The Hebrew University of Jerusalem</span>
<span style="color: rgb(92, 181, 149);">[</span><span style="color: rgb(1, 84, 76);">/</span><span style="color: rgb(0, 161, 146);">/</span> <span style="color: rgb(41, 16, 22);">\</span><span style="color: rgb(41, 16, 22);">\</span> | <span style="color: rgb(51, 88, 104);">T +972-2-5494522 | F +972-2-5494522</span>
<span style="color: rgb(1, 84, 76);">//</span> <span style="color: rgb(21, 122, 134);">\</span> | <span style="color: rgb(51, 88, 104);"><a href="mailto:irush@cs.huji.ac.il" data-auth="NotApplicable">irush@cs.huji.ac.il</a></span>
<span style="color: rgb(127, 130, 103);">/</span><span style="color: rgb(1, 84, 76);">/</span> |
</pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<pre style="font-family:monospace"> <span style="color: rgb(133, 12, 27);">/|</span> |
<span style="color: rgb(133, 12, 27);">\/</span> | <span style="color: rgb(51, 88, 104); font-weight: bold;">Yair Yarom </span><span style="color: rgb(51, 88, 104);">| System Group (DevOps)</span>
<span style="color: rgb(92, 181, 149);">[]</span> | <span style="color: rgb(51, 88, 104); font-weight: bold;">The Rachel and Selim Benin School</span>
<span style="color: rgb(92, 181, 149);">[]</span> <span style="color: rgb(133, 12, 27);">/\</span> | <span style="color: rgb(51, 88, 104); font-weight: bold;">of Computer Science and Engineering</span>
<span style="color: rgb(92, 181, 149);">[]</span><span style="color: rgb(0, 161, 146);">//</span><span style="color: rgb(133, 12, 27);">\</span><span style="color: rgb(133, 12, 27);">\</span><span style="color: rgb(49, 154, 184);">/</span> | <span style="color: rgb(51, 88, 104);">The Hebrew University of Jerusalem</span>
<span style="color: rgb(92, 181, 149);">[</span><span style="color: rgb(1, 84, 76);">/</span><span style="color: rgb(0, 161, 146);">/</span> <span style="color: rgb(41, 16, 22);">\</span><span style="color: rgb(41, 16, 22);">\</span> | <span style="color: rgb(51, 88, 104);">T +972-2-5494522 | F +972-2-5494522</span>
<span style="color: rgb(1, 84, 76);">//</span> <span style="color: rgb(21, 122, 134);">\</span> | <span style="color: rgb(51, 88, 104);"><a href="mailto:irush@cs.huji.ac.il" data-auth="NotApplicable">irush@cs.huji.ac.il</a></span>
<span style="color: rgb(127, 130, 103);">/</span><span style="color: rgb(1, 84, 76);">/</span> |
</pre>
</div>
</div>
</div>
</div>
</div>
</body>
</html>