<HTML><HEAD>
<META content="text/html; charset=utf-8" http-equiv=content-type></HEAD>
<BODY dir=ltr>
<DIV dir=ltr>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: 'Calibri'; COLOR: #000000">
<DIV>Hi Zhang</DIV>
<DIV> </DIV>
<DIV>Thanks for the quick reply. </DIV>
<DIV> </DIV>
<DIV>Could you please guide me on specifying MIG partitions in gres.conf and in
slurm.conf</DIV>
<DIV> </DIV>
<DIV>My MIG is as below:</DIV>
<DIV> </DIV>
<DIV>root@rl-dgxs-r21-l2:~# sudo nvidia-smi mig -lgi</DIV>
<DIV>+----------------------------------------------------------------+</DIV>
<DIV>| GPU
instances:
|</DIV>
<DIV>| GPU
Name
Profile Instance Placement |</DIV>
<DIV>|
ID
ID
Start:Size |</DIV>
<DIV>|========================================|</DIV>
<DIV>| 0 MIG
1g.10gb
19
9
2:1 |</DIV>
<DIV>+----------------------------------------------------------------+</DIV>
<DIV>| 0 MIG
1g.10gb
19
10
3:1 |</DIV>
<DIV>+----------------------------------------------------------------+</DIV>
<DIV>| 0 MIG
2g.20gb
14
3
0:2 |</DIV>
<DIV>+----------------------------------------------------------------+</DIV>
<DIV>| 0 MIG
3g.40gb
9
2
4:4 |</DIV>
<DIV>+----------------------------------------------------------------+</DIV>
<DIV> </DIV>
<DIV>root@rl-dgxs-r21-l2:~# nvidia-smi -L</DIV>
<DIV>GPU 0: NVIDIA A100-SXM4-80GB (UUID:
GPU-a044d304-28b2-c3f1-42ea-b9440d868231)</DIV>
<DIV> MIG 3g.40gb Device 0: (UUID:
MIG-d4514f04-e287-50e9-b3c4-c19fddbb9aa2)</DIV>
<DIV> MIG 2g.20gb Device 1: (UUID:
MIG-4f393220-5308-51f7-bd7a-322306593545)</DIV>
<DIV> MIG 1g.10gb Device 2: (UUID:
MIG-4d988c3e-160a-52f3-a3e1-8eeccfee4585)</DIV>
<DIV> MIG 1g.10gb Device 3: (UUID:
MIG-4ff411c0-c0e2-5b86-a3a4-e76a6b6491cb)</DIV>
<DIV>GPU 1: NVIDIA A100-SXM4-80GB (UUID:
GPU-6aecec97-f63e-4815-3c20-503c4e82fa57)</DIV>
<DIV>GPU 2: NVIDIA A100-SXM4-80GB (UUID:
GPU-2ab474df-49c4-531a-6580-cd44d9982d0a)</DIV>
<DIV>GPU 3: NVIDIA DGX Display (UUID:
GPU-aa49c52b-640c-39b2-1cee-a0120d0b5fa7)</DIV>
<DIV>GPU 4: NVIDIA A100-SXM4-80GB (UUID:
GPU-c47adbbe-7ef8-246c-f700-010542e60ad0)</DIV>
<DIV> </DIV>
<DIV>Any suggestions on using MIG partitions in my slurm jobs?</DIV>
<DIV> </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: 'Calibri'; COLOR: #000000">With Warm
Regards<BR>Ravi <BR><BR></DIV>
<DIV
style='FONT-SIZE: small; TEXT-DECORATION: none; FONT-FAMILY: "Calibri"; FONT-WEIGHT: normal; COLOR: #000000; FONT-STYLE: normal; DISPLAY: inline'>
<DIV style="FONT: 10pt tahoma">
<DIV> </DIV>
<DIV style="BACKGROUND: #f5f5f5">
<DIV style="font-color: black"><B>From:</B> <A
title=szhang@ngs.gen-info.osaka-u.ac.jp>Shunran Zhang</A> </DIV>
<DIV><B>Sent:</B> Thursday, November 30, 2023 9:50 PM</DIV>
<DIV><B>To:</B> <A title=ravibhatk@gmail.com>Ravi Konila</A> ; <A
title=slurm-users@lists.schedmd.com>Slurm User Community List</A> </DIV>
<DIV><B>Subject:</B> Re: [slurm-users] Autodetect of nvml is not working
in</DIV></DIV></DIV>
<DIV> </DIV></DIV>
<DIV
style='FONT-SIZE: small; TEXT-DECORATION: none; FONT-FAMILY: "Calibri"; FONT-WEIGHT: normal; COLOR: #000000; FONT-STYLE: normal; DISPLAY: inline'>
<DIV dir=ltr>Hi Ravi </DIV>
<DIV dir=ltr> </DIV>
<DIV dir=ltr>Unfortunately if the NVML flag is off on compile time ( when the
maintainer build the apt package for you to install ), that part of code would
not be in your binary code. </DIV>
<DIV dir=ltr> </DIV>
<DIV dir=ltr>Recompile yourself following the official documentation or find
some repository that builds slurm with NVML are your only options. </DIV>
<DIV dir=ltr> </DIV>
<DIV dir=ltr>Good luck</DIV>
<DIV dir=ltr>S. Zhang</DIV>
<DIV dir=ltr><BR>
<BLOCKQUOTE type="cite">Ravi Konila <ravibhatk@gmail.com>於2023年12月1日
00:51寫道：<BR><BR></BLOCKQUOTE></DIV>
<BLOCKQUOTE type="cite">
<DIV dir=ltr>
<DIV dir=ltr>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: 'Calibri'; COLOR: #000000">
<DIV>Hi Josef and Rob</DIV>
<DIV>Thanks for the reply.</DIV>
<DIV>I do agree cuda-nvml-devel was not there while installing slurm-llnl in
Ubuntu 22.04. </DIV>
<DIV>Later I installed it. </DIV>
<DIV>I did not build slurm but I installed it from apt install slurm command.
</DIV>
<DIV> </DIV>
<DIV>Is there any method to use it post slurm installation?</DIV>
<DIV> </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: 'Calibri'; COLOR: #000000">With Warm
Regards<BR>Ravi K.<BR>Ph: +91-9901072688
<BR></DIV></DIV></DIV></DIV></BLOCKQUOTE></DIV></DIV></DIV></BODY></HTML>