<div dir="ltr">Brian / Christopher, that looks like a good process, thanks guys, I will do some testing and let you know.<div><br></div><div>if I mark a partition down and it has running jobs, what happens to those jobs, do they keep running?<br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div><br></div><div>Sid Young</div><div>W: <a href="https://off-grid-engineering.com" target="_blank">https://off-grid-engineering.com</a><br></div><div>W: (personal) <a href="https://sidyoung.com/" target="_blank">https://sidyoung.com/</a></div><div>W: (personal) <a href="https://z900collector.wordpress.com/" target="_blank">https://z900collector.wordpress.com/</a></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 1, 2022 at 3:27 PM Brian Andrus <<a href="mailto:toomuchit@gmail.com">toomuchit@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div>
    <p>One possibility:<br>
    </p>
    <p>Sounds like your concern is folks with interactive jobs from the
      login node that are running under screen/tmux.</p>
    <p>That being the case, you need running jobs to end and not allow
      new users to start tmux sessions. <br>
    </p>
    <p>Definitely doing 'scontrol update state=down partition=xxxx' for
      each partition. Also:<br>
    </p>
    <p>touch /etc/nologin</p>
    <p>That will prevent new logins.</p>
    <p>Send a message to all active folks</p>
    <p>wall "system going down at XX:XX, please end your sessions"</p>
    <p>Then wait for folks to drain off your login node and do your
      stuff.</p>
    <p>When done, remove the /etc/nologin file and folks will be able to
      login again.</p>
    <p>Brian Andrus<br>
    </p>
    <div>On 1/31/2022 9:18 PM, Sid Young wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div dir="ltr"><br clear="all">
          <div>
            <div dir="ltr">
              <div dir="ltr">
                <div dir="ltr">
                  <div dir="ltr">
                    <div dir="ltr">
                      <div dir="ltr">
                        <div dir="ltr">
                          <div dir="ltr">
                            <div><br>
                            </div>
                            <div><br>
                            </div>
                            <div>Sid Young</div>
                            <div>W: <a href="https://off-grid-engineering.com" target="_blank">https://off-grid-engineering.com</a><br>
                            </div>
                            <div>W: (personal) <a href="https://sidyoung.com/" target="_blank">https://sidyoung.com/</a></div>
                            <div>W: (personal) <a href="https://z900collector.wordpress.com/" target="_blank">https://z900collector.wordpress.com/</a></div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
          <br>
        </div>
        <br>
        <div class="gmail_quote">
          <div dir="ltr" class="gmail_attr">On Tue, Feb 1, 2022 at 3:02
            PM Christopher Samuel <<a href="mailto:chris@csamuel.org" target="_blank">chris@csamuel.org</a>>
            wrote:<br>
          </div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 1/31/22 4:41 pm, Sid
            Young wrote:<br>
            <br>
            > I need to replace a faulty DIMM chim in our login node
            so I need to stop <br>
            > new jobs being kicked off while letting the old ones
            end.<br>
            > <br>
            > I thought I would just set all nodes to drain to stop
            new jobs from <br>
            > being kicked off...<br>
            <br>
            That would basically be the way, but is there any reason why
            compute <br>
            jobs shouldn't start whilst the login node is down?<br>
          </blockquote>
          <div><br>
          </div>
          <div>My concern was to keep the running jobs going and stop
            new jobs, so when the last running job ends,<br>
          </div>
          <div> I could reboot the login node knowing that any terminal
            windows "screen"/"tmux" sessions would effectively</div>
          <div>have ended as the job(s) had now ended <br>
          </div>
          <div><br>
          </div>
          <div>
            <div>I'm not sure if there was an accepted procedure or best
              practice way to tackle shutting down the Login node for
              this use case.</div>
            <div><br>
            </div>
            <div>On the bright side I am down to two jobs left so any
              day now :)</div>
            <div><br>
            </div>
            <div>Sid</div>
            <div><br>
            </div>
            <div><br>
            </div>
            <br>
          </div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
            <br>
            All the best,<br>
            Chris<br>
            -- <br>
               Chris Samuel  :  <a href="http://www.csamuel.org/" rel="noreferrer" target="_blank">http://www.csamuel.org/</a> 
            :  Berkeley, CA, USA<br>
            <br>
          </blockquote>
        </div>
      </div>
    </blockquote>
  </div>

</blockquote></div>