<div dir="ltr">Dear Marcus,<div><br></div><div> Thanks a lot for your reply. I will write it into our User Manual, and let users know how to apply for multiple GPU cards.</div><div><br></div><div>Best regards,</div><div>Ran</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 16, 2019 at 5:40 PM Marcus Wagner <<a href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
Dear Ran,<br>
<br>
you can only ask for GPUS PER NODE, as gres are ressources per node.<br>
<br>
So, you can ask for 5 gpus and then get 5 gpus on each of the two
nodes.<br>
At the moment it is not possible to ask for 8 gpus on one node and 2
on another. <br>
That MIGHT change with slurm 19.05, since SchedMD is overhauling
besides pother things the gpu handling within slurm.<br>
<br>
<br>
Best<br>
Marcus<br>
<br>
<div class="gmail-m_-2833122721317663176moz-cite-prefix">On 4/16/19 9:15 AM, Ran Du wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Dear Antony,
<div><br>
</div>
<div> It's worked!</div>
<div><br>
</div>
<div> I checked the allocation, and here is the record: </div>
<div><br>
</div>
<div> Nodes=gpu012 CPU_IDs=0-2 Mem=3072
GRES_IDX=gpu:v100(IDX:0-7) Nodes=gpu013 CPU_IDs=0 Mem=1024
GRES_IDX=gpu:v100(IDX:0-7)</div>
<div><br>
</div>
<div> The job has got what it applied for. </div>
<div><br>
</div>
<div> And another question is : how to apply for multiple
cards could not be divided exactly by 8？ For example, to
apply for 10 GPU cards, 8 cards on one node and 2 cards on
another node?</div>
<div><br>
</div>
<div> Thanks a lot again for your kind help.</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Ran</div>
<div> </div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Apr 15, 2019 at 8:25
PM Ran Du <<a href="mailto:bella.ran.du@gmail.com" target="_blank">bella.ran.du@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Dear Antony,
<div><br>
</div>
<div> Thanks a lot for your reply, I tried to submit a
job with your advice, and no more sbatch errors. </div>
<div><br>
</div>
<div> But because our cluster is under maintenance, I
have to wait till tomorrow to see if GPU cards are
allocated correctly. I will let you know as soon as the
job is submitted successfully.</div>
<div><br>
</div>
<div> Thanks a lot for your kind help.</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Ran</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Apr 15, 2019 at
4:40 PM Antony Cleave <<a href="mailto:antony.cleave@gmail.com" target="_blank">antony.cleave@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">Ask for 8 gpus on 2 nodes instead.
<div dir="auto"><br>
</div>
<div dir="auto">In your script just change the 16 to 8
and it should do what you want.</div>
<div dir="auto"><br>
</div>
<div dir="auto">You are currently asking for 2 nodes
with 16 gpu each as <span style="font-family:sans-serif">Gres resources are
per node.</span></div>
<div dir="auto"><br>
</div>
<div dir="auto">Antony </div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, 15 Apr 2019,
09:08 Ran Du, <<a href="mailto:bella.ran.du@gmail.com" target="_blank">bella.ran.du@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Dear all,
<div><br>
</div>
<div> Does anyone know how to set #SBATCH
options to get multiple GPU cards from
different worker nodes?</div>
<div><br>
</div>
<div> One of our users would like to apply
for 16 NVIDIA V100 cards for his job, and
there are 8 GPU cards on each worker node, I
have tried the following #SBATCH options:</div>
<div><br>
</div>
<div> #SBATCH --partition=gpu </div>
<div> #SBATCH --qos=normal</div>
<div> #SBATCH --account=u07</div>
<div> #SBATCH --job-name=cross </div>
<div> #SBATCH --nodes=2 </div>
<div> #SBATCH --mem-per-cpu=1024 </div>
<div> #SBATCH --output=test.32^4.16gpu.log </div>
<div> #SBATCH --gres=gpu:v100:16</div>
<div><br>
</div>
<div> but got the sbatch error message : </div>
<div> sbatch: error: Batch job submission
failed: Requested node configuration is not
available</div>
<div><br>
</div>
<div> And I found a similar question on
stack overflow: </div>
<div> <a href="https://stackoverflow.com/questions/45200926/how-to-access-to-gpus-on-different-nodes-in-a-cluster-with-slurm" rel="noreferrer" target="_blank">https://stackoverflow.com/questions/45200926/how-to-access-to-gpus-on-different-nodes-in-a-cluster-with-slurm</a></div>
<div><br>
</div>
<div> And it is said that multiple GPU
cards allocation on different worker nodes are
not available, the post is in 2017, is it
still true at present?</div>
<div><br>
</div>
<div> Thanks a lot for your help.</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Ran </div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_-2833122721317663176moz-signature" cols="72">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="gmail-m_-2833122721317663176moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de" target="_blank">wagner@itc.rwth-aachen.de</a>
<a class="gmail-m_-2833122721317663176moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de" target="_blank">www.itc.rwth-aachen.de</a>
</pre>
</div>
</blockquote></div>