<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <tt><br>
    </tt><tt>Do one more pass through making sure  </tt><tt><br>
    </tt><tt>s/</tt><tt>1080GTX/1080gtx and s/K20/k20</tt><tt><br>
    </tt><tt><br>
    </tt><tt>shutdown all slurmd, slurmctld, start slurmctl, start
      slurmd</tt><tt><br>
    </tt><tt><br>
    </tt><tt><br>
    </tt><tt>I find it less confusing to have a global gres.conf file. 
      I haven't used a list (nvidia[0-1), mainly because I want to
      specify the</tt><tt> the cores to use for each gpu.</tt><tt><br>
    </tt><tt><br>
    </tt><tt>gres.conf would look </tt><tt>something like...</tt><tt><br>
    </tt><tt><br>
    </tt><tt>NodeName=tiger[02-04,06-09,11-14,16-19,21-22] Name=gpu
      Type=k80 File=/dev/nvidia0 Cores=0</tt><tt><br>
    </tt><tt>NodeName=tiger[02-04,06-09,11-14,16-19,21-22] Name=gpu
      Type=k80 File=/dev/nvidia1 Cores=1</tt><tt><br>
    </tt><tt>NodeName=tiger[01,05,10,15,20] Name=gpu Type=1080gtx
      File=/dev/nvidia0 Cores=0</tt><tt><br>
    </tt><tt>
      NodeName=tiger[01,05,10,15,20] Name=gpu Type=1080gtx
      File=/dev/nvidia1 Cores=1</tt><tt><br>
    </tt><tt>
    </tt><tt><br>
    </tt><tt>which can be distributed to all nodes. </tt><tt><br>
    </tt><tt><br>
    </tt><tt>-b</tt><br>
    <br>
    <br>
    <div class="moz-cite-prefix">On 12/04/2018 09:55 AM, Lou Nicotra
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAOPM4OX0OFpY1J5m+BfPR8vjSf7Nc828NR1sgR8qL0=aakOQmQ@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=utf-8">
      <div dir="ltr">
        <div dir="ltr">Brian, the specific node does not show any
          gres...
          <div>
            <div>root@panther02 slurm# scontrol show partition=tiger_1</div>
            <div>PartitionName=tiger_1</div>
            <div>   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL</div>
            <div>   AllocNodes=ALL Default=YES QoS=N/A</div>
            <div>   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO
              GraceTime=0 Hidden=NO</div>
            <div>   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0
              LLN=NO MaxCPUsPerNode=UNLIMITED</div>
            <div>   Nodes=tiger[01-22]</div>
            <div>   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO
              ReqResv=NO OverSubscribe=NO</div>
            <div>   OverTimeLimit=NONE PreemptMode=OFF</div>
            <div>   State=UP TotalCPUs=1056 TotalNodes=22
              SelectTypeParameters=NONE</div>
            <div>   JobDefaults=(null)</div>
            <div>   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED</div>
            <div><br>
            </div>
            <div>root@panther02 slurm#  scontrol show node=tiger11</div>
            <div>NodeName=tiger11 Arch=x86_64 CoresPerSocket=12</div>
            <div>   CPUAlloc=0 CPUTot=48 CPULoad=11.50</div>
            <div>   AvailableFeatures=HyperThread</div>
            <div>   ActiveFeatures=HyperThread</div>
            <div>   Gres=(null)</div>
            <div>   NodeAddr=X.X.X.X NodeHostName=tiger11 Version=18.08</div>
            <div>   OS=Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19
              22:10:57 UTC 2015</div>
            <div>   RealMemory=1 AllocMem=0 FreeMem=269695 Sockets=2
              Boards=1</div>
            <div>   State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1
              Owner=N/A MCS_label=N/A</div>
            <div>   Partitions=tiger_1,compute_1</div>
            <div>   BootTime=2018-04-02T13:30:12
              SlurmdStartTime=2018-12-03T16:13:22</div>
            <div>   CfgTRES=cpu=48,mem=1M,billing=48</div>
            <div>   AllocTRES=</div>
            <div>   CapWatts=n/a</div>
            <div>   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0</div>
            <div>   ExtSensorsJoules=n/s ExtSensorsWatts=0
              ExtSensorsTemp=n/s</div>
          </div>
          <div><br>
          </div>
          <div>So, something is not setup correctly... Could it be a
            18.X bug?</div>
          <div><br>
          </div>
          <div>Thanks.</div>
          <div><br>
          </div>
        </div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr">On Tue, Dec 4, 2018 at 9:31 AM Lou Nicotra <<a
            href="mailto:lnicotra@interactions.com"
            moz-do-not-send="true">lnicotra@interactions.com</a>>
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0 0 0
          .8ex;border-left:1px #ccc solid;padding-left:1ex">
          <div dir="ltr">Thanks Michael. I will try 17.x as I also could
            not see anything wrong with my settings... Will report back
            afterwards...
            <div><br>
            </div>
            <div>Lou</div>
          </div>
          <br>
          <div class="gmail_quote">
            <div dir="ltr">On Tue, Dec 4, 2018 at 9:11 AM Michael Di
              Domenico <<a href="mailto:mdidomenico4@gmail.com"
                target="_blank" moz-do-not-send="true">mdidomenico4@gmail.com</a>>
              wrote:<br>
            </div>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">unfortunately,
              someone smarter then me will have to help further.  I'm<br>
              not sure i see anything specifically wrong.  The one thing
              i might try<br>
              is backing the software down to a 17.x release series.  I
              recently<br>
              tried 18.x and had some issues.  I can't say whether it'll
              be any<br>
              different, but you might be exposing an undiagnosed bug in
              the 18.x<br>
              branch<br>
              On Mon, Dec 3, 2018 at 4:17 PM Lou Nicotra <<a
                href="mailto:lnicotra@interactions.com" target="_blank"
                moz-do-not-send="true">lnicotra@interactions.com</a>>
              wrote:<br>
              ><br>
              > Made the change in the gres.conf on local server file
              and restarted slurmd and slurmctld on master....
              Unfortunately same error...<br>
              ><br>
              > Distributed corrected gres.conf to all k20 servers,
              restarted slurmd and slurmdctl...   Still has same
              error...<br>
              ><br>
              > On Mon, Dec 3, 2018 at 4:04 PM Brian W. Johanson <<a
                href="mailto:bjohanso@psc.edu" target="_blank"
                moz-do-not-send="true">bjohanso@psc.edu</a>> wrote:<br>
              >><br>
              >> Is that a lowercase k in k20 specified in the
              batch script and nodename and a uppercase K specified in
              gres.conf?<br>
              >><br>
              >> On 12/03/2018 09:13 AM, Lou Nicotra wrote:<br>
              >><br>
              >> Hi All, I have recently set up a slurm cluster
              with my servers and I'm running into an issue while
              submitting GPU jobs. It has something to to with gres
              configurations, but I just can't seem to figure out what
              is wrong. Non GPU jobs run fine.<br>
              >><br>
              >> The error is as follows:<br>
              >> sbatch: error: Batch job submission failed:
              Invalid Trackable RESource (TRES) specification  after
              submitting a batch job.<br>
              >><br>
              >> My batch job is as follows:<br>
              >> #!/bin/bash<br>
              >> #SBATCH --partition=tiger_1   # partition name<br>
              >> #SBATCH --gres=gpu:k20:1<br>
              >> #SBATCH --gres-flags=enforce-binding<br>
              >> #SBATCH --time=0:20:00  # wall clock limit<br>
              >> #SBATCH --output=gpu-%J.txt<br>
              >> #SBATCH --account=lnicotra<br>
              >> module load cuda<br>
              >> python gpu1<br>
              >><br>
              >> Where gpu1 is a GPU test script that runs
              correctly while invoked via python. Tiger_1 partition has
              servers with GPUs, with a mix of 1080GTX and K20 as
              specified in slurm.conf<br>
              >><br>
              >> I have defined GRES resources in the slurm.conf
              file:<br>
              >> # GPU GRES<br>
              >> GresTypes=gpu<br>
              >> NodeName=tiger[01,05,10,15,20] Gres=gpu:1080gtx:2<br>
              >> NodeName=tiger[02-04,06-09,11-14,16-19,21-22]
              Gres=gpu:k20:2<br>
              >><br>
              >> And have a local gres.conf on the servers
              containing GPUs...<br>
              >> lnicotra@tiger11 ~# cat /etc/slurm/gres.conf<br>
              >> # GPU Definitions<br>
              >> # NodeName=tiger[02-04,06-09,11-14,16-19,21-22]
              Name=gpu Type=K20 File=/dev/nvidia[0-1]<br>
              >> Name=gpu Type=K20 File=/dev/nvidia[0-1] Cores=0,1<br>
              >><br>
              >> and a similar one for the 1080GTX<br>
              >> # GPU Definitions<br>
              >> # NodeName=tiger[01,05,10,15,20] Name=gpu
              Type=1080GTX File=/dev/nvidia[0-1]<br>
              >> Name=gpu Type=1080GTX File=/dev/nvidia[0-1]
              Cores=0,1<br>
              >><br>
              >> The account manager seems to know about the
              GPUs...<br>
              >> lnicotra@tiger11 ~# sacctmgr show tres<br>
              >>     Type            Name     ID<br>
              >> -------- --------------- ------<br>
              >>      cpu                      1<br>
              >>      mem                      2<br>
              >>   energy                      3<br>
              >>     node                      4<br>
              >>  billing                      5<br>
              >>       fs            disk      6<br>
              >>     vmem                      7<br>
              >>    pages                      8<br>
              >>     gres             gpu   1001<br>
              >>     gres         gpu:k20   1002<br>
              >>     gres     gpu:1080gtx   1003<br>
              >><br>
              >> Can anyone point out what am I missing?<br>
              >><br>
              >> Thanks!<br>
              >> Lou<br>
              >><br>
              >><br>
              >> --<br>
              >><br>
              >> Lou Nicotra<br>
              >><br>
              >> IT Systems Engineer - SLT<br>
              >><br>
              >> Interactions LLC<br>
              >><br>
              >> o:  908-673-1833<br>
              >><br>
              >> m: 908-451-6983<br>
              >><br>
              >> <a href="mailto:lnicotra@interactions.com"
                target="_blank" moz-do-not-send="true">lnicotra@interactions.com</a><br>
              >><br>
              >> <a href="http://www.interactions.com"
                rel="noreferrer" target="_blank" moz-do-not-send="true">www.interactions.com</a><br>
              >><br>
              >>
*******************************************************************************<br>
              >><br>
              >> This e-mail and any of its attachments may
              contain Interactions LLC proprietary information, which is
              privileged, confidential, or subject to copyright
              belonging to the Interactions LLC. This e-mail is intended
              solely for the use of the individual or entity to which it
              is addressed. If you are not the intended recipient of
              this e-mail, you are hereby notified that any
              dissemination, distribution, copying, or action taken in
              relation to the contents of and attachments to this e-mail
              is strictly prohibited and may be unlawful. If you have
              received this e-mail in error, please notify the sender
              immediately and permanently delete the original and any
              copy of this e-mail and any printout. Thank You.<br>
              >><br>
              >>
*******************************************************************************<br>
              >><br>
              >><br>
              ><br>
              ><br>
              > --<br>
              ><br>
              > Lou Nicotra<br>
              ><br>
              > IT Systems Engineer - SLT<br>
              ><br>
              > Interactions LLC<br>
              ><br>
              > o:  908-673-1833<br>
              ><br>
              > m: 908-451-6983<br>
              ><br>
              > <a href="mailto:lnicotra@interactions.com"
                target="_blank" moz-do-not-send="true">lnicotra@interactions.com</a><br>
              ><br>
              > <a href="http://www.interactions.com"
                rel="noreferrer" target="_blank" moz-do-not-send="true">www.interactions.com</a><br>
              ><br>
              >
*******************************************************************************<br>
              ><br>
              > This e-mail and any of its attachments may contain
              Interactions LLC proprietary information, which is
              privileged, confidential, or subject to copyright
              belonging to the Interactions LLC. This e-mail is intended
              solely for the use of the individual or entity to which it
              is addressed. If you are not the intended recipient of
              this e-mail, you are hereby notified that any
              dissemination, distribution, copying, or action taken in
              relation to the contents of and attachments to this e-mail
              is strictly prohibited and may be unlawful. If you have
              received this e-mail in error, please notify the sender
              immediately and permanently delete the original and any
              copy of this e-mail and any printout. Thank You.<br>
              ><br>
              >
*******************************************************************************<br>
              <br>
            </blockquote>
          </div>
          <br clear="all">
          <div><br>
          </div>
          -- <br>
          <div dir="ltr" class="m_-5309998978109404036gmail_signature"
            data-smartmail="gmail_signature">
            <div dir="ltr">
              <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><b><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#6fa8dc">Lou
                    Nicotra</span></b><span
                  style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
              <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">IT
                  Systems Engineer -
                  SLT</span><span
                  style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
              <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">Interactions
                  LLC</span></p>
              <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span
                  style="font-size:9.5pt;font-family:Arial,sans-serif">o:
                   </span><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666"><a
                    href="tel:781-405-5114" target="_blank"
                    moz-do-not-send="true"><span style="color:#1155cc">908-673-1833</span></a></span><span
                  style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
              <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">m: <a
                    href="tel:781-405-5114" target="_blank"
                    moz-do-not-send="true"><span style="color:#1155cc">908-451-6983</span></a></span><span
                  style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
              <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><u><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#1155cc"><a
                      href="mailto:lnicotra@interactions.com"
                      target="_blank" moz-do-not-send="true">lnicotra@interactions.com</a></span></u><span
                  style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
              <span
style="font-size:9.5pt;line-height:107%;font-family:"Arial",sans-serif;color:#666666"><a
                  href="http://www.interactions.com/" target="_blank"
                  moz-do-not-send="true"><span style="color:#1155cc">www.interactions.com</span></a></span><br>
            </div>
          </div>
        </blockquote>
      </div>
      <br clear="all">
      <div><br>
      </div>
      -- <br>
      <div dir="ltr" class="gmail_signature"
        data-smartmail="gmail_signature">
        <div dir="ltr">
          <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><b><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#6fa8dc">Lou
                Nicotra</span></b><span
              style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
          <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">IT
              Systems Engineer -
              SLT</span><span
              style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
          <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">Interactions
              LLC</span></p>
          <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span
              style="font-size:9.5pt;font-family:Arial,sans-serif">o:  </span><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666"><a
                href="tel:781-405-5114" target="_blank"
                moz-do-not-send="true"><span style="color:#1155cc">908-673-1833</span></a></span><span
              style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
          <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#666666">m: <a
                href="tel:781-405-5114" target="_blank"
                moz-do-not-send="true"><span style="color:#1155cc">908-451-6983</span></a></span><span
              style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
          <p
style="margin-bottom:0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-repeat:initial"><u><span
style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#1155cc"><a
                  href="mailto:lnicotra@interactions.com"
                  target="_blank" moz-do-not-send="true">lnicotra@interactions.com</a></span></u><span
              style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p>
          <span
style="font-size:9.5pt;line-height:107%;font-family:"Arial",sans-serif;color:#666666"><a
              href="http://www.interactions.com/" target="_blank"
              moz-do-not-send="true"><span style="color:#1155cc">www.interactions.com</span></a></span><br>
        </div>
      </div>
      <br>
      <font size="3" face="Times New Roman">
      </font>
      <p style="margin:0in 0in 8pt"><font size="3" face="Calibri">******************************<wbr>******************************<wbr>*******************</font></p>
      <font size="3" face="Times New Roman">
      </font>
      <p style="margin:0in 0in 8pt"><font size="3" face="Calibri">This
          e-mail and any of its attachments may contain
          Interactions LLC proprietary information, which is privileged,
          confidential, or subject to copyright belonging to the
          Interactions
          LLC. This e-mail is intended solely for the use of the
          individual or
          entity to which it is addressed. If you are not the intended
          recipient of this
          e-mail, you are hereby notified that any dissemination,
          distribution, copying,
          or action taken in relation to the contents of and attachments
          to this e-mail
          is strictly prohibited and may be unlawful. If you have
          received this e-mail in
          error, please notify the sender immediately and permanently
          delete the original
          and any copy of this e-mail and any printout. Thank You.  </font></p>
      <font size="3" face="Times New Roman">
      </font>
      <p style="margin:0in 0in 8pt"><font face="Calibri"><font size="3">******************************<wbr>******************************<wbr>*******************<span> 
            </span></font></font></p>
      <font size="3" face="Times New Roman">
      </font>
    </blockquote>
    <br>
  </body>
</html>