<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">Unfortunately that didn't work, <br>
    </div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">However i modified my slurm.conf to lie
      and say i had 16 cpu on 1 thread and now everything is working
      fine. <br>
    </div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">One issue with CLOUD state machines is
      that is when i run scontrol show nodes they don't show up, is
      there a way i can get their info when they are not 'running'<br>
    </div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">Thanks<br>
    </div>
    <div class="moz-cite-prefix">Nathan <br>
    </div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">On 20/5/19 12:04 am, Riebs, Andy wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:TU4PR8401MB0896C88C1D7E18857247BDAE9B050@TU4PR8401MB0896.NAMPRD84.PROD.OUTLOOK.COM">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <meta name="Generator" content="Microsoft Word 15 (filtered
        medium)">
      <style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman",serif;
        color:black;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:#0563C1;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:#954F72;
        text-decoration:underline;}
p
        {mso-style-priority:99;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:12.0pt;
        font-family:"Times New Roman",serif;
        color:black;}
span.EmailStyle18
        {mso-style-type:personal-reply;
        font-family:"Calibri",sans-serif;
        color:#1F497D;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
      <div class="WordSection1">
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Just
            looking at this quickly, have you tried specifying
            “hint=multithread” as an sbatch parameter?
            <o:p></o:p></span></p>
        <p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
        <div>
          <div style="border:none;border-top:solid #E1E1E1
            1.0pt;padding:3.0pt 0in 0in 0in">
            <p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext"
                  lang="EN-US">From:</span></b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext"
                lang="EN-US"> slurm-users
                [<a class="moz-txt-link-freetext" href="mailto:slurm-users-bounces@lists.schedmd.com">mailto:slurm-users-bounces@lists.schedmd.com</a>]
                <b>On Behalf Of </b>nathan norton<br>
                <b>Sent:</b> Saturday, May 18, 2019 6:03 PM<br>
                <b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a><br>
                <b>Subject:</b> [slurm-users] final stages of cloud
                infrastructure set up<o:p></o:p></span></p>
          </div>
        </div>
        <p class="MsoNormal"><o:p> </o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Hi,
          </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">I am in the process of setting up Slurm using
            Amazon cloud infrastructure. All is going well, I can
            elastically start and stop nodes when jobs run.  I am
            running into a few small teething issues, that are probably
            due to me not understanding some of the terminology here. At
            a high level all the nodes given to end users in the cloud
            are hyper threaded, so I want to use my nodes as hyper
            threaded nodes.  All nodes are running centos7 latest. I
            would also like the jobs to be run in a cgroup and not
            migrate around after it starts. As I said before I think
            most of it is working except for the few issues below here.
          </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">My use case is, I have an in house built binary
            application that is single threaded and does no  message
            passing or anything like that. The application is not memory
            bound it is only compute bound.</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">So on a node I would like to be able to run 16
            instances in parallel. As can be seen below if I launch the
            single app via srun it runs on each thread on a CPU.
             However if I run the via sbatch command as can be seen it
            only runs on CPU 0-7 instead of CPU 0-15.
          </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Another question would be how would be the best
            way to retry failed jobs, I can rerun the batch again, but I
            only want to rerun a single step in the batch?</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Please see below for the output of various
            commands as well as my slurm.conf file as well.</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Many thanks</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Nathan.
          </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">______________________________________________________________________</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">btuser@bt_slurm_login001[domain ]% slurmd -V</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">slurm 18.08.6-2</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">______________________________________________________________________</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">btuser@bt_slurm_login001[domain ]% cat
            nathan.batch.sh</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">#!/bin/bash</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">#SBATCH --job-name=nathan_test</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">#SBATCH --ntasks=1</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">#SBATCH --array=1-32</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">#SBATCH --ntasks-per-core=2</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">hostname</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">srun --hint=multithread -n1  --exclusive  
            --cpu_bind=threads cat /proc/self/status | grep -i
            cpus_allowed_list</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">btuser@bt_slurm_login001[domain ]%</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">btuser@bt_slurm_login001[domain ]% sbatch
            Nathan.batch.sh</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Submitted batch job 106491</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">btuser@bt_slurm_login001[domain ]% cat
            slurm-106491_*</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">btuser@bt_slurm_login001[domain ]% cat
            slurm-106491_*</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-89.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      1</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-89.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      2</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-89.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      3</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-89.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      4</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-89.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      5</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-89.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      6</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-89.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      7</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-90.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      0</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-90.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      1</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-90.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      2</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-88.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      0</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-90.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      3</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-90.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      4</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-90.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      5</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-90.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      6</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-90.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      7</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-91.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      0</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-91.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      1</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-91.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      2</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-91.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      3</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-91.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      4</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-88.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      1</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-91.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      5</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-91.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      6</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-91.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      7</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-88.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      2</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-88.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      3</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-88.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      4</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-88.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      5</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-88.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      6</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-88.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      7</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ip-10-0-8-89.ec2.internal</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      0</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">______________________________________________________________________</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">btuser@bt_slurm_login001[domain ]% srun -n32 
            --exclusive   --cpu_bind=threads cat /proc/self/status |
            grep -i
cpus_allowed_list                                                                                                         </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      12</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      13</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      15</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      0</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      8</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      1</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      9</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      2</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      10</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      11</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      4</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      5</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      6</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      14</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      7</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      3</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      0</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      8</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      1</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      9</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      2</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      10</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      3</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      11</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      4</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      12</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      5</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      6</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      14</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      7</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      15</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cpus_allowed_list:      13</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">______________________________________________________________________</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">[sysadmin@ip-10-0-8-88 ~]$ slurmd -C</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">NodeName=ip-10-0-8-88 CPUs=16 Boards=1
            SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2
            RealMemory=30986</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">UpTime=0-00:06:10</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">______________________________________________________________________</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">Cloud server stats:</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">[sysadmin@ip-10-0-8-88 ~]$ lscpu  -e</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">0   0    0      0    0:0:0:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">1   0    0      1    1:1:1:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">2   0    0      2    2:2:2:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">3   0    0      3    3:3:3:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">4   0    0      4    4:4:4:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">5   0    0      5    5:5:5:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">6   0    0      6    6:6:6:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">7   0    0      7    7:7:7:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">8   0    0      0    0:0:0:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">9   0    0      1    1:1:1:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">10  0    0      2    2:2:2:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">11  0    0      3    3:3:3:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">12  0    0      4    4:4:4:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">13  0    0      5    5:5:5:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">14  0    0      6    6:6:6:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">15  0    0      7    7:7:7:0       yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">______________________________________________________________________</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"># slurm.conf file generated by configurator
            easy.html.</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SlurmctldHost=bt_slurm_master</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">MailProg=/dev/null</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">MpiDefault=none</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ProctrackType=proctrack/cgroup</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ReturnToService=1</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SlurmctldPidFile=/var/run/slurmd/slurmctld.pid</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SlurmdPidFile=/var/run/slurmd/slurmd.pid</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SlurmdSpoolDir=/var/spool/slurmctld/slurmd</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SlurmUser=slurm</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">StateSaveLocation=/var/spool/slurmctld</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SwitchType=switch/none</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">TaskPlugin=task/cgroup</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">TaskPluginParam=Threads</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SlurmdTimeout=500</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">FastSchedule=1</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SchedulerType=sched/backfill</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SelectType=select/cons_res</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SelectTypeParameters=CR_CPU</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">AccountingStorageType=accounting_storage/none</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ClusterName=simplecluster</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">JobAcctGatherType=jobacct_gather/none</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">PropagatePrioProcess=2</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">MaxTasksPerNode=16</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ResumeProgram=/bt/admin/slurm/etc/slurm_ec2_startup.sh</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ResumeTimeout=900 
          </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ResumeRate=0            
          </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SuspendProgram=/bt/admin/slurm/etc/slurm_ec2_shutdown.sh</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SuspendTime=600      
          </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SuspendTimeout=120
          </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">TreeWidth=1024 
          </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">SuspendRate=0    
          </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">NodeName=ip-10-0-8-[2-100] CPUs=16
            RealMemory=27648 Sockets=1 CoresPerSocket=8
            ThreadsPerCore=2  State=CLOUD</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">NodeName=bt_slurm_login00[1-10] RealMemory=512
            State=DOWN</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">PartitionName=backtest Nodes=ip-10-0-8-[2-100]
            Default=YES MaxTime=300 Oversubscribe=NO State=UP</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">                                      
          </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">______________________________________________________________________</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">[sysadmin@bt_slurm_master ~]$ cat
            /etc/slurm/cgroup.conf</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">###</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">#</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"># Slurm cgroup support configuration file</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">#</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"># See man slurm.conf and man cgroup.conf for
            further</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"># information on cgroup configuration
            parameters</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">#--</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">CgroupAutomount=yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">CgroupMountpoint="/sys/fs/cgroup"</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">TaskAffinity=yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ConstrainCores=yes</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">ConstrainRAMSpace=no</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">______________________________________________________________________</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">JobId=107424 ArrayJobId=107423 ArrayTaskId=1
            JobName=aijing_test</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   UserId=btuser(1001) GroupId=users(100)
            MCS_label=N/A</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   Priority=4294901612 Nice=0 Account=(null)
            QOS=(null)</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   JobState=COMPLETED Reason=None
            Dependency=(null)</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0
            ExitCode=0:0</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   DerivedExitCode=0:0</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   RunTime=00:01:06 TimeLimit=05:00:00
            TimeMin=N/A</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   SubmitTime=2019-05-17T01:55:56
            EligibleTime=2019-05-17T01:55:56</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   AccrueTime=Unknown</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   StartTime=2019-05-17T01:55:56
            EndTime=2019-05-17T01:57:02 Deadline=N/A</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   PreemptTime=None SuspendTime=None
            SecsPreSuspend=0</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   LastSchedEval=2019-05-17T01:55:56</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   Partition=backtest
            AllocNode:Sid=bt_slurm_login001:14002</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">  ReqNodeList=(null) ExcNodeList=(null)</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   NodeList=ip-10-0-8-88</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   BatchHost=ip-10-0-8-88</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1
            ReqB:S:C:T=0:0:*:*</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   TRES=cpu=1,node=1,billing=1</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   Socks/Node=* NtasksPerN:B:S:C=0:0:*:*
            CoreSpec=*</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">     Nodes=ip-10-0-8-88 CPU_IDs=0-1 Mem=0
            GRES_IDX=</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   MinCPUsNode=1 MinMemoryNode=0
            MinTmpDiskNode=0</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   Features=(null) DelayBoot=00:00:00</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   OverSubscribe=OK Contiguous=0
            Licenses=(null) Network=(null)</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">  
            Command=/bt/data/backtester/destination_tables_94633/batch_run.sh</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">  
            WorkDir=/bt/data/backtester/destination_tables_94633</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">  
            StdErr=/bt/data/backtester/destination_tables_94633/slurm-107423_1.out</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   StdIn=/dev/null</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">  
            StdOut=/bt/data/backtester/destination_tables_94633/slurm-107423_1.out</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU">   Power=</span><o:p></o:p></p>
        <p class="MsoNormal"
          style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
            lang="EN-AU"> </span><o:p></o:p></p>
      </div>
    </blockquote>
    <p><br>
    </p>
  </body>
</html>