<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style id="bidiui-paragraph-margins" type="text/css">body p { margin-bottom: 0cm; margin-top: 0pt; } </style>
</head>
<body bidimailui-detected-decoding-type="UTF-8"
bidimailui-charset-is-forced="true" style="direction: ltr;">
<p>Hi Ravi,<br>
</p>
<div class="moz-cite-prefix">On 20/11/2023 6:36, Ravi Konila wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8ED1EDA8185C4F1CAA1D0AB2D216B4B8@RAVIKONILAPC">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div
style="FONT-SIZE: 12pt; FONT-FAMILY: 'Calibri'; COLOR: #000000">
<div>Hello Everyone</div>
<div> </div>
<div><snipped><br>
</div>
<div>My question is related to submission of jobs to those
GPUs. How do a student submit the job to a particular GPU
out of 4 GPUs? For example, studentA should submit the job
to GPU ID 1 instead of GPU ID 0. </div>
</div>
</div>
</blockquote>
<p>In classical HPC this is a counterproductive - you don't want to
assign specific resources to jobs, as this would lead to jobs
waiting needlessly while resources are available, so I think some
background for this request might help understand the need and
possible solutions.</p>
<p><br>
</p>
<p>That said, it might be possible by assigning different artificial
types to each gpu, e.g. in gres.conf Name=gpu type=gpu0
file=/dev/nvidia0 etc...</p>
<p>Then submission would be of the form</p>
<p>sbatch --gpus=gpu0</p>
<p><br>
</p>
<p>The issue would be with submitting in the general case, where you
want any gpu. For that you might have to fall back to using gres
as in</p>
<p>sbatch --gres=gpu:3</p>
<p><br>
</p>
<p>This is obviously cumbersome and less convenient, and I'm not
sure this is not an XY problem.<br>
</p>
<blockquote type="cite"
cite="mid:8ED1EDA8185C4F1CAA1D0AB2D216B4B8@RAVIKONILAPC">
<div dir="ltr">
<div
style="FONT-SIZE: 12pt; FONT-FAMILY: 'Calibri'; COLOR: #000000">
<div> </div>
<div>Also we are planning for MIG in the server and we would
like few students to submit the jobs to 20G partition and
non critical jobs to 5G partition. </div>
<div>How should be the slurm.conf and gres.conf in this case.
</div>
</div>
</div>
</blockquote>
Can you elaborate on the use case? It's unclear to me if the
students are expected to decide on their own when to submit to 20G
and when to 5G, if students with access to 20G should also use the
5G together with the rest of the students, or if all students should
have access to both partitions and some other criteria should be
used to determine placement.<br>
<blockquote type="cite"
cite="mid:8ED1EDA8185C4F1CAA1D0AB2D216B4B8@RAVIKONILAPC">
<div dir="ltr">
<div
style="FONT-SIZE: 12pt; FONT-FAMILY: 'Calibri'; COLOR: #000000">
<div> </div>
<div>Currently our configuration is as below:</div>
<div> </div>
<div>gres.conf</div>
<div>Name=gpu type=A100 file=/dev/nvidia[0-2,4]</div>
<div> </div>
<div>------------</div>
<div>slurm.conf</div>
<div>.</div>
<div>.</div>
<div>.</div>
<div>GresTypes=gpu</div>
<div>NodeName=rl-dgxs-r21-l2 Gres=gpu:A100:4 CPUs=128
RealMemory=500000 State=UNKNOWN</div>
<div>PartitionName=LocalGPUQ Nodes=ALL Default=YES
MaxTime=INFINITE State=UP</div>
<div> </div>
<div>-------------</div>
<div> </div>
<div>Any suggestions or help in this regard is highly
appreciated. </div>
<div> </div>
<div
style="FONT-SIZE: 12pt; FONT-FAMILY: 'Calibri'; COLOR: #000000">With
Warm Regards<br>
Ravi Konila<br>
</div>
</div>
</div>
</blockquote>
<p>Best regards,</p>
<p>--Dani_L.<br>
</p>
</body>
</html>