<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Dear Ran,<br>
    <br>
    you can only ask for GPUS PER NODE, as gres are ressources per node.<br>
    <br>
    So, you can ask for 5 gpus and then get 5 gpus on each of the two
    nodes.<br>
    At the moment it is not possible to ask for 8 gpus on one node and 2
    on another. <br>
    That MIGHT change with slurm 19.05, since SchedMD is overhauling
    besides pother things the gpu handling within slurm.<br>
    <br>
    <br>
    Best<br>
    Marcus<br>
    <br>
    <div class="moz-cite-prefix">On 4/16/19 9:15 AM, Ran Du wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAFfzCHi2PrcHGBHXWVvWSZwCZxQgAc+an=Km1hpUf66RZpGw0g@mail.gmail.com">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div dir="ltr">Dear Antony,
          <div><br>
          </div>
          <div>      It's worked!</div>
          <div><br>
          </div>
          <div>      I checked the allocation, and here is the record: </div>
          <div><br>
          </div>
          <div>      Nodes=gpu012 CPU_IDs=0-2 Mem=3072
            GRES_IDX=gpu:v100(IDX:0-7) Nodes=gpu013 CPU_IDs=0 Mem=1024
            GRES_IDX=gpu:v100(IDX:0-7)</div>
          <div><br>
          </div>
          <div>      The job has got what it applied for.  </div>
          <div><br>
          </div>
          <div>      And another question is : how to apply for multiple
            cards could not be divided exactly by 8? For example, to
            apply for 10 GPU cards, 8 cards on one node and 2 cards on
            another node?</div>
          <div><br>
          </div>
          <div>     Thanks a lot again for your kind help.</div>
          <div><br>
          </div>
          <div>Best regards,</div>
          <div>Ran</div>
          <div>      </div>
        </div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Mon, Apr 15, 2019 at 8:25
          PM Ran Du <<a href="mailto:bella.ran.du@gmail.com"
            moz-do-not-send="true">bella.ran.du@gmail.com</a>> wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">Dear Antony,
            <div><br>
            </div>
            <div>       Thanks a lot for your reply, I tried to submit a
              job with your advice, and no more sbatch errors. </div>
            <div><br>
            </div>
            <div>       But because our cluster is under maintenance, I
              have to wait till tomorrow to see if GPU cards are
              allocated correctly.  I will let you know as soon as the
              job is submitted successfully.</div>
            <div><br>
            </div>
            <div>       Thanks a lot for your kind help.</div>
            <div><br>
            </div>
            <div>Best regards,</div>
            <div>Ran</div>
          </div>
          <br>
          <div class="gmail_quote">
            <div dir="ltr" class="gmail_attr">On Mon, Apr 15, 2019 at
              4:40 PM Antony Cleave <<a
                href="mailto:antony.cleave@gmail.com" target="_blank"
                moz-do-not-send="true">antony.cleave@gmail.com</a>>
              wrote:<br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div dir="auto">Ask for 8 gpus on 2 nodes instead.
                <div dir="auto"><br>
                </div>
                <div dir="auto">In your script just change the 16 to 8
                  and it should do what you want.</div>
                <div dir="auto"><br>
                </div>
                <div dir="auto">You are currently asking for 2 nodes
                  with 16 gpu each as <span
                    style="font-family:sans-serif">Gres resources are
                    per node.</span></div>
                <div dir="auto"><br>
                </div>
                <div dir="auto">Antony </div>
              </div>
              <br>
              <div class="gmail_quote">
                <div dir="ltr" class="gmail_attr">On Mon, 15 Apr 2019,
                  09:08 Ran Du, <<a
                    href="mailto:bella.ran.du@gmail.com" target="_blank"
                    moz-do-not-send="true">bella.ran.du@gmail.com</a>>
                  wrote:<br>
                </div>
                <blockquote class="gmail_quote" style="margin:0px 0px
                  0px 0.8ex;border-left:1px solid
                  rgb(204,204,204);padding-left:1ex">
                  <div dir="ltr">
                    <div dir="ltr">
                      <div dir="ltr">Dear all,
                        <div><br>
                        </div>
                        <div>     Does anyone know how to set #SBATCH
                          options to get multiple GPU cards from
                          different worker nodes?</div>
                        <div><br>
                        </div>
                        <div>     One of our users would like to apply
                          for 16 NVIDIA V100 cards for his job, and 
                          there are 8 GPU cards on each worker node, I
                          have tried the following #SBATCH options:</div>
                        <div><br>
                        </div>
                        <div>      #SBATCH --partition=gpu </div>
                        <div>      #SBATCH --qos=normal</div>
                        <div>      #SBATCH --account=u07</div>
                        <div>      #SBATCH --job-name=cross </div>
                        <div>      #SBATCH --nodes=2 </div>
                        <div>      #SBATCH --mem-per-cpu=1024 </div>
                        <div>      #SBATCH --output=test.32^4.16gpu.log </div>
                        <div>      #SBATCH --gres=gpu:v100:16</div>
                        <div><br>
                        </div>
                        <div>      but got the sbatch error message : </div>
                        <div>      sbatch: error: Batch job submission
                          failed: Requested node configuration is not
                          available</div>
                        <div><br>
                        </div>
                        <div>      And I found a similar question on
                          stack overflow: </div>
                        <div> <a
href="https://stackoverflow.com/questions/45200926/how-to-access-to-gpus-on-different-nodes-in-a-cluster-with-slurm"
                            rel="noreferrer" target="_blank"
                            moz-do-not-send="true">https://stackoverflow.com/questions/45200926/how-to-access-to-gpus-on-different-nodes-in-a-cluster-with-slurm</a></div>
                        <div><br>
                        </div>
                        <div>      And it is said that multiple GPU
                          cards allocation on different worker nodes are
                          not available, the post is in 2017, is it
                          still true at present?</div>
                        <div><br>
                        </div>
                        <div>      Thanks a lot for your help.</div>
                        <div><br>
                        </div>
                        <div>Best regards,</div>
                        <div>Ran     </div>
                      </div>
                    </div>
                  </div>
                </blockquote>
              </div>
            </blockquote>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="72">-- 
Marcus Wagner, Dipl.-Inf.

IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de">www.itc.rwth-aachen.de</a>
</pre>
  </body>
</html>