<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">I think cgroups is prob more elegant ………. but here is another script
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://github.com/FredHutch/IT/blob/master/py/loadwatcher.py#L59">https://github.com/FredHutch/IT/blob/master/py/loadwatcher.py#L59</a>
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The email text is hard coded so please change before using. We put this in place in Oct 2017 when things where getting out of control because folks were using much more multithreaded software than before. Since then we had 95 users removed
from one of the login nodes and several 100 warnings sent. The <span style="font-size:9.0pt;font-family:Consolas;color:#032F62;background:white">
killall -9 -v -g –u username </span><o:p></o:p></p>
<p class="MsoNormal">has been very effective. We have 3 login nodes with 28 cores and almost 400G RAM.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Dirk<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoPlainText">-----Original Message-----<br>
From: hpcxxxxxx@lists.fhcrc.org [mailto:hpcxxxxxxxxxxx@lists.fhcrc.org] On Behalf Of loadwatchxxxxxxxxxxx@fhcrc.org<br>
Sent: Tuesday, November 14, 2017 11:45 AM<br>
To: Doe, John <xxxxxxxxx @fredhutch.org><br>
Subject: [hpcpol] RHINO3: Your jobs have been removed!<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">This is a notification message from loadwatcher.py, running on host RHINO3. Please review the following message:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">jdoe, your CPU utilization on rhino3 is currently 4499 %!<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">For short term jobs you can use no more than 400 % or 4.0 CPU cores on the Rhino machines.<o:p></o:p></p>
<p class="MsoPlainText">We have removed all your processes from this computer.<o:p></o:p></p>
<p class="MsoPlainText">Please try again and submit batch jobs<o:p></o:p></p>
<p class="MsoPlainText">or use the 'grabnode' command for interactive jobs.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">see <a href="http://scicomp.fhcrc.org/Gizmo%20Cluster%20Quickstart.aspx">
<span style="color:windowtext;text-decoration:none">http://scicomp.fhcrc.org/Gizmo%20Cluster%20Quickstart.aspx</span></a><o:p></o:p></p>
<p class="MsoPlainText">or <a href="http://scicomp.fhcrc.org/Grab%20Commands.aspx">
<span style="color:windowtext;text-decoration:none">http://scicomp.fhcrc.org/Grab%20Commands.aspx</span></a><o:p></o:p></p>
<p class="MsoPlainText">or <a href="http://scicomp.fhcrc.org/SciComp%20Office%20Hours.aspx">
<span style="color:windowtext;text-decoration:none">http://scicomp.fhcrc.org/SciComp%20Office%20Hours.aspx</span></a><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">If output is being captured, you may find additional information in your logs.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#123054"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#123054">Dirk Petersen</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#123054"><br>
<span style="background:white">Scientific Computing Director<br>
Fred Hutch </span><br>
<span style="background:white">1100 Fairview Ave. N.<o:p></o:p></span></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#123054;background:white">Mail Stop M4-A882</span><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#123054"><br>
<span style="background:white">Seattle, WA 98109</span><br>
<b><span style="background:white">Phone:</span></b><span style="background:white"> 206.667.5926<o:p></o:p></span></span></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#123054;background:white">Skype:</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#123054;background:white"> internetchen<br>
</span><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#123054"><br>
</span><span style="font-size:10.5pt;color:black"><img border="0" width="109" height="50" style="width:1.1333in;height:.525in" id="Picture_x0020_1" src="cid:image001.png@01D3A677.B651A780" alt="cid:8C6A9079-96CB-447C-94D9-DD59438042C1"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>