<div dir="ltr"><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent"> "complete network wide network outage tomorrow night from 10pm across the whole institute".</span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent"> ^^^^^^</span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent"><br></span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent">Lachlan, I advise running the following script on all login nodes:</span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent"><br></span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent">#!/bin/bash</span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent">#</span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent">cat << EOF > /etc/motd</span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent">HPC Managers are in the pub.</span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent">At this hour of the day you should also be.</span></div><div><br></div><div>In case of HPC actually on fire, Lachlan can be contacted at:</div><div>In Front of the Bar</div><div>The Dog and Duck<span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent"></span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent">EOF</span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent"><br></span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent"><br></span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent"><br></span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent"><br></span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent"><br></span></div><div><span style="text-align:left;color:rgb(34,34,34);text-transform:none;text-indent:0px;letter-spacing:normal;font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant:normal;font-weight:400;text-decoration:none;word-spacing:0px;display:inline;white-space:normal;direction:ltr;float:none;background-color:transparent"><br></span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 9 November 2017 at 04:57, Jonathon A Anderson <span dir="ltr"><<a href="mailto:jonathon.anderson@colorado.edu" target="_blank">jonathon.anderson@colorado.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">In your situation, where you're blocking user access to the login node, it probably doesn't matter. We use DOWN in most events, as INACTIVE would prevent new jobs from being queued against the partition at all. DOWN allows the jobs to be queued, and just doesn't permit them to run. (In either case, HOLDing PENDING jobs is redundant.)<br>
<br>
~jonathon<br>
<br>
______________________________<wbr>__________<br>
From: slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com">slurm-users-bounces@lists.<wbr>schedmd.com</a>> on behalf of Lachlan Musicman <<a href="mailto:datakid@gmail.com">datakid@gmail.com</a>><br>
Sent: Wednesday, November 8, 2017 5:00:12 PM<br>
To: Slurm User Community List<br>
Subject: [slurm-users] Quick hold on all partitions, all jobs<br>
<div class="HOEnZb"><div class="h5"><br>
The IT team sent an email saying "complete network wide network outage tomorrow night from 10pm across the whole institute".<br>
<br>
Our plan is to put all queued jobs on hold, suspend all running jobs, and turning off the login node.<br>
<br>
I've just discovered that the partitions have a state, and it can be set to UP, DOWN, DRAIN or INACTIVE.<br>
<br>
In this situation - most likely a 4 hour outage with nothing else affected - would you mark your partitions DOWN or INACTIVE?<br>
<br>
Ostensibly all users should be off the systems (because no network), but there's always one that sets an at or cron job or finds that corner case.<br>
<br>
Cheers<br>
L.<br>
<br>
<br>
------<br>
"The antidote to apocalypticism is apocalyptic civics. Apocalyptic civics is the insistence that we cannot ignore the truth, nor should we panic about it. It is a shared consciousness that our institutions have failed and our ecosystem is collapsing, yet we are still here — and we are creative agents who can shape our destinies. Apocalyptic civics is the conviction that the only way out is through, and the only way through is together. "<br>
<br>
Greg Bloom @greggish <a href="https://twitter.com/greggish/status/873177525903609857" target="_blank" rel="noreferrer">https://twitter.com/greggish/<wbr>status/873177525903609857</a><br>
<br>
</div></div></blockquote></div><br></div>