<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 8 4 3 5 4 4 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 2 6 3 5 4 5 2 3 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<div id="mail-editor-reference-message-container">
<div>
<div>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">In my HPC center, I found a SLURM job that was submitted with --gres=gpu:6 whereas the cluster has only four GPUs per node each. It is a parallel job. Here are some relevant field printout:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">AllocCPUS 30</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">AllocGRES gpu:6</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">AllocTRES billing=30,cpu=30,gres/gpu=6,node=3</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">CPUTime 1-01:23:00</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">CPUTimeRAW 91380</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">Elapsed 00:50:46</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">JobID 20073</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">JobIDRaw 20073</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">JobName simple_cuda</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">NCPUS 30</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">NGPUS 6.0</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">What happened in this case? This job was asking for 3 nodes, 10 core per node. When the user specified “--gres=gpu:6”, does this mean six GPUs for the entire job, or six GPUs per node? Per the description in
<a href="https://slurm.schedmd.com/gres.html#Running_Jobs">https://slurm.schedmd.com/gres.html#Running_Jobs</a>, it says: gres is “Generic resources required per node”. So it is illogical to request six GPUs per node. So what happened? Did SLURM quietly ignore
the request and grant just one, or grant the max number (4)? Because apparently the job ran without error.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">Wirawan Purwanto<br>
Computational Scientist, HPC Group<br>
Information Technology Services<br>
Old Dominion University<br>
Norfolk, VA 23529</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>