<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Hi Tim,</div>
<div class="">we have MIG configured and integrated with Slurm using the slurm-mig-discovery tools:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://gitlab.com/nvidia/hpc/slurm-mig-discovery" class="">https://gitlab.com/nvidia/hpc/slurm-mig-discovery</a></div>
<div class=""><br class="">
</div>
<div class="">The mig-parted tool is great for setting up MIG itself: </div>
<div class=""><br class="">
</div>
<div class=""><a href="https://github.com/NVIDIA/mig-parted" class="">https://github.com/NVIDIA/mig-parted</a></div>
<div class=""><br class="">
</div>
<div class="">Once setup MIG instances work fine with Slurm although the output from nvidia-smi is a little different as one sees both GPUs - the “visible device” is the MIG instance::</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""><font face="Courier" class="">$ salloc -p interactive -n 1 -c 8 --gres=gpu:1 </font></div>
<div class=""><font face="Courier" class="">salloc: Granted job allocation 5235</font></div>
<div class=""><font face="Courier" class="">salloc: Waiting for resource configuration</font></div>
<div class=""><font face="Courier" class="">salloc: Nodes gpu001 are ready for job</font></div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""><font face="Courier" class="">$ env | grep CUDA</font></div>
<div class=""><font face="Courier" class="">CUDA_VISIBLE_DEVICES=0</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class="">$ nvidia-smi -L</font></div>
<div class=""><font face="Courier" class="">GPU 0: A100-PCIE-40GB (UUID: GPU-c1976541-7b00-3f9f-f557-a17f45b879e9)</font></div>
<div class=""><font face="Courier" class=""> MIG 3g.20gb Device 0: (UUID: MIG-GPU-c1976541-7b00-3f9f-f557-a17f45b879e9/1/0)</font></div>
<div class=""><font face="Courier" class="">GPU 1: A100-PCIE-40GB (UUID: GPU-83f9ff5b-09c3-8de1-b3eb-adaadb1cda9f)</font></div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">The caveats are that MIG and the slurm integration is rather static for the moment so it’s not really possible to dynamically change the profiles. </div>
<div class=""><br class="">
</div>
<div class="">The other slight issue is that all combinations of MIG instances waste some compute or memory capacity. We have divided each A100 into two 3g.20gb devices so all the memory is used but 1/7 of the compute capacity is lost.</div>
<div class=""><br class="">
</div>
<div class="">Thanks</div>
<br class="">
<div class="">Ewan Roche<br class="">
<br class="">
Division Calcul et Soutien à la Recherche<br class="">
UNIL | Université de Lausanne<br class="">
<br class="">
</div>
<br class="">
<blockquote type="cite" class="">On 21 Apr 2021, at 09:14, Timothy Carr <<a href="mailto:timothy.carr@uct.ac.za" class="">timothy.carr@uct.ac.za</a>> wrote:<br class="">
<br class="">
Dear Community, <br class="">
<br class="">
Trust everyone is well and keeping safe? <br class="">
<br class="">
We are considering the purchase of nodes with the Nvidia A100 GPUs and enabling the MIG feature which allows for the creation of instance resource profiles. The creation of these profiles seems to be straightforward as per the documentation. Have any of you
had the opportunity to implement the A100 MIG with SLURM and have you found any caveats you are willing to share? <br class="">
<br class="">
Kind Regards <br class="">
<br class="">
--<br class="">
Tim<br class="">
<br class="">
<br class="">
<br class="">
Disclaimer - University of Cape Town This email is subject to UCT policies and email disclaimer published on our website at <a href="http://www.uct.ac.za/main/email-disclaimer" class="">http://www.uct.ac.za/main/email-disclaimer</a> or obtainable from +27 21
650 9111. If this email is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via https://csirt.uct.ac.za/page/report-an-incident.php.<br class="">
</blockquote>
<br class="">
</body>
</html>