<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html;
      charset=windows-1252">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>I use rpm's for our installs here.  I usually pause all the jobs
      prior to the upgrade, then I follow the guide here:</p>
    <p><br>
    </p>
    <p><a class="moz-txt-link-freetext" href="https://slurm.schedmd.com/quickstart_admin.html">https://slurm.schedmd.com/quickstart_admin.html</a></p>
    <p><br>
    </p>
    <p>I haven't done the upgrade to 18.08 though yet, and so I haven't
      had to contend with the automatic restart that seems to be the
      case with the new rpm spec script (we went to 17.11 prior to the
      rpm spec reorg).  Frankly I wish that they didn't do the automatic
      restart as I like to manage that myself.</p>
    <p><br>
    </p>
    <p>As Chris said though you definitely want to do the slurmdbd
      upgrade from the commandline.  I've had it where when just
      restarting the service it times out and the database only gets
      partially update.  In which case I had to restore from the
      mysqldump I had made and tried again.  Also highly recommend doing
      mysqldumps prior to major version updates.</p>
    <p><br>
    </p>
    <p>-Paul Edmon-<br>
    </p>
    <br>
    <div class="moz-cite-prefix">On 09/25/2018 09:54 AM, Baker D.J.
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:AM6PR04MB464665A871ECD5C55F642C52FE160@AM6PR04MB4646.eurprd04.prod.outlook.com">
      <meta http-equiv="Content-Type" content="text/html;
        charset=windows-1252">
      <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;"
        dir="ltr">
        <div id="divtagdefaultwrapper" style="font-size: 12pt; color:
          rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif,
          EmojiFont, "Apple Color Emoji", "Segoe UI
          Emoji", NotoColorEmoji, "Segoe UI Symbol",
          "Android Emoji", EmojiSymbols;" dir="ltr">
          <p style="margin-top:0;margin-bottom:0">Thank you for your
            comments. <span style="font-size: 12pt;">I could potentially
              force the upgrade of the slurm and slurm-slumdbd rpms
              using something like:</span></p>
          <div style="font-family: Calibri, Helvetica, sans-serif,
            EmojiFont, "Apple Color Emoji", "Segoe UI
            Emoji", NotoColorEmoji, "Segoe UI Symbol",
            "Android Emoji", EmojiSymbols; font-size: 16px;
            margin-top: 0px; margin-bottom: 0px;">
            <br>
          </div>
          <div style="font-family: Calibri, Helvetica, sans-serif,
            EmojiFont, "Apple Color Emoji", "Segoe UI
            Emoji", NotoColorEmoji, "Segoe UI Symbol",
            "Android Emoji", EmojiSymbols; font-size: 16px;
            margin-top: 0px; margin-bottom: 0px;">
            rpm -Uvh --noscripts --nodeps --force
            slurm-18.08.0-1.el7.x86_64.rpm
            slurm-slurmdbd-18.08.0-1.el7.x86_64.rpm<br>
          </div>
          <div style="font-family: Calibri, Helvetica, sans-serif,
            EmojiFont, "Apple Color Emoji", "Segoe UI
            Emoji", NotoColorEmoji, "Segoe UI Symbol",
            "Android Emoji", EmojiSymbols; font-size: 16px;
            margin-top: 0px; margin-bottom: 0px;">
            <br>
          </div>
          <div style="font-family: Calibri, Helvetica, sans-serif,
            EmojiFont, "Apple Color Emoji", "Segoe UI
            Emoji", NotoColorEmoji, "Segoe UI Symbol",
            "Android Emoji", EmojiSymbols; font-size: 16px;
            margin-top: 0px; margin-bottom: 0px;">
            That will certainly work, however the slurmctld (or in the
            case of my test node, the slurmd) will be killed. The logic
            is that at v17.02 the slurm rpm provides slurmctld and
            slurmd. So upgrading that rpm will destroy/kill the existing
            slurmctld or slurmd processes. That is...</div>
          <div style="font-family: Calibri, Helvetica, sans-serif,
            EmojiFont, "Apple Color Emoji", "Segoe UI
            Emoji", NotoColorEmoji, "Segoe UI Symbol",
            "Android Emoji", EmojiSymbols; font-size: 16px;
            margin-top: 0px; margin-bottom: 0px;">
            <br>
          </div>
          <div style="font-family: Calibri, Helvetica, sans-serif,
            EmojiFont, "Apple Color Emoji", "Segoe UI
            Emoji", NotoColorEmoji, "Segoe UI Symbol",
            "Android Emoji", EmojiSymbols; font-size: 16px;">
            # rpm -q --whatprovides /usr/sbin/slurmctld</div>
          <div style="font-family: Calibri, Helvetica, sans-serif,
            EmojiFont, "Apple Color Emoji", "Segoe UI
            Emoji", NotoColorEmoji, "Segoe UI Symbol",
            "Android Emoji", EmojiSymbols; font-size: 16px;">
            slurm-17.02.8-1.el7.x86_64</div>
          <div style="font-family: Calibri, Helvetica, sans-serif,
            EmojiFont, "Apple Color Emoji", "Segoe UI
            Emoji", NotoColorEmoji, "Segoe UI Symbol",
            "Android Emoji", EmojiSymbols; font-size: 16px;
            margin-top: 0px; margin-bottom: 0px;">
            <br>
          </div>
          <div style="font-family: Calibri, Helvetica, sans-serif,
            EmojiFont, "Apple Color Emoji", "Segoe UI
            Emoji", NotoColorEmoji, "Segoe UI Symbol",
            "Android Emoji", EmojiSymbols; font-size: 16px;
            margin-top: 0px; margin-bottom: 0px;">
            So if I force the upgrade of that rpm then I delete and
            kill <font size="2"
              face="Calibri,Helvetica,sans-serif,EmojiFont,Apple Color
              Emoji,Segoe UI Emoji,NotoColorEmoji,Segoe UI
              Symbol,Android Emoji,EmojiSymbols"><span style="font-size:
                16px;">/usr/sbin/slurmctld. In the new rpm structure
                slurmctld is now provided by its own rpm.</span></font></div>
          <br>
          I would have thought that someone would have crossed this
          bridge, but maybe most admins don't use rpms...</div>
        <div id="divtagdefaultwrapper" style="font-size: 12pt; color:
          rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif,
          EmojiFont, "Apple Color Emoji", "Segoe UI
          Emoji", NotoColorEmoji, "Segoe UI Symbol",
          "Android Emoji", EmojiSymbols;" dir="ltr">
          <br>
        </div>
        <div id="divtagdefaultwrapper" style="font-size: 12pt; color:
          rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif,
          EmojiFont, "Apple Color Emoji", "Segoe UI
          Emoji", NotoColorEmoji, "Segoe UI Symbol",
          "Android Emoji", EmojiSymbols;" dir="ltr">
          Best regards,</div>
        <div id="divtagdefaultwrapper" style="font-size: 12pt; color:
          rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif,
          EmojiFont, "Apple Color Emoji", "Segoe UI
          Emoji", NotoColorEmoji, "Segoe UI Symbol",
          "Android Emoji", EmojiSymbols;" dir="ltr">
          David<br>
          <br>
          <div style="color: rgb(0, 0, 0);">
            <hr style="display:inline-block;width:98%" tabindex="-1">
            <div id="divRplyFwdMsg" dir="ltr"><font
                style="font-size:11pt" face="Calibri, sans-serif"
                color="#000000"><b>From:</b> slurm-users
                <a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on behalf
                of Chris Samuel <a class="moz-txt-link-rfc2396E" href="mailto:chris@csamuel.org"><chris@csamuel.org></a><br>
                <b>Sent:</b> 25 September 2018 13:00<br>
                <b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a><br>
                <b>Subject:</b> Re: [slurm-users] Upgrading a slurm on a
                cluster, 17.02 --> 18.08</font>
              <div> </div>
            </div>
            <div class="BodyFragment"><font size="2"><span
                  style="font-size:11pt;">
                  <div class="PlainText">On Tuesday, 25 September 2018
                    9:41:10 PM AEST Baker D. J.  wrote:<br>
                    <br>
                    > I guess that the only solution is to upgrade
                    all the slurm at once. That<br>
                    > means that the slurmctld will be killed (unless
                    it has been stopped first).<br>
                    <br>
                    We don't use RPMs from Slurm [1], but the rpm
                    command does have a --noscripts <br>
                    option to (allegedly, I've never used it) suppress
                    the execution of pre/post <br>
                    install scripts.<br>
                    <br>
                    A big warning would be do not use systemctl to start
                    the new slurmdbd for the <br>
                    first time when upgrading!<br>
                    <br>
                    Stop the older one first (and then take a database
                    dump) and then run the new <br>
                    slurmdbd process with the "-Dvvv" options (inside
                    screen, just in case) so <br>
                    that you can watch its progress and systemd won't
                    decide it's been taking too <br>
                    long to start and try and kill it part way through
                    the database upgrades).<br>
                    <br>
                    Once that's completed successfully then you can ^C
                    it and start it up via the <br>
                    systemctl once more.<br>
                    <br>
                    Hope that's useful!<br>
                    <br>
                    All the best,<br>
                    Chris<br>
                    <br>
                    [1] - I've always installed into
                    ${shared_local_area}/slurm/${version} and had <br>
                    a symlink called "latest" that points at the
                    currently blessed version of <br>
                    Slurm.  Then I stop slurmdbd, upgrade that as above,
                    then I can do slurmctld <br>
                    (with partitions marked down, just in case).  Once
                    those are done I can <br>
                    restart slurmd's around the cluster.<br>
                    <br>
                    -- <br>
                     Chris Samuel  :  <a
href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&amp;data=01%7C01%7Cd.j.baker%40soton.ac.uk%7C35d2a0583f124e84bf0d08d622deab4e%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=uTVuUTGI3fpPZqffe1p5RifQ1%2BG%2FbsrW0ixkCeu%2FxKw%3D&amp;reserved=0"
                      id="LPlnk546531" class="OWAAutoLink"
                      previewremoved="true" moz-do-not-send="true">
https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&amp;data=01%7C01%7Cd.j.baker%40soton.ac.uk%7C35d2a0583f124e84bf0d08d622deab4e%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&amp;sdata=uTVuUTGI3fpPZqffe1p5RifQ1%2BG%2FbsrW0ixkCeu%2FxKw%3D&amp;reserved=0</a> 
                    :  Melbourne, VIC<br>
                    <br>
                    <br>
                    <br>
                    <br>
                  </div>
                </span></font></div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>