<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">No cluster mgr/framework in use... Custom-compiled and packaged the Slurm 16.05.4 release into .rpm/.deb files, and used them to install the different nodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Although the homedirs are no longer shared, the nodes do have access to shared storage, one mounted as a subdir of the home directory (which you can symlink
stuff from that to the homedir level to “auto-magically” via a conf file that works with a system we designed.) So shared dotfiles, subdirs/files in homedir, etc are all possible.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Have not investigated containerized Slurm setup – will have to put that on the exploration list. If the workloads were Dockerized, I’d probably run them via
Kubernetes rather than Slurm...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-Will<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> slurm-users [mailto:slurm-users-bounces@lists.schedmd.com]
<b>On Behalf Of </b>John Hearns<br>
<b>Sent:</b> Friday, May 25, 2018 5:44 AM<br>
<b>To:</b> Slurm User Community List<br>
<b>Subject:</b> Re: [slurm-users] Controller / backup controller q's<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Will, I know I will regret chiming in here. Are you able to say what cluster manager or framework you are using?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I don't see a problem in running two different distributions. But as Per says look at your development environment.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">For my part, I would ask have you thought about containerisation? ie CentOS comoute nodes and run Singularity?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">ALso the 'unique home directory per node' gives me the heebie-jeebies. I guess technically is it OK.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">However many commercial packages crate dot files or dit directories in user home directories.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I am thinking of things like Ansys and Matlab etc. etc. etc. here<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">What will you do if these dotfiles are not consistent across the cluster?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Before anyoen says it, I was arguing somewhere else recently that 'home directories' are an outdated concept when you are running HPC.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I still think that, and this is a classic case in point.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Forgive me if I have misunderstood your setup.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 25 May 2018 at 11:30, Pär Lindfors <<a href="mailto:paran@nsc.liu.se" target="_blank">paran@nsc.liu.se</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi Will,<br>
<br>
On 05/24/2018 05:43 PM, Will Dennis wrote:<br>
> (we were using CentOS 7.x<br>
> originally, now the compute nodes are on Ubuntu 16.04.) Currently, we<br>
> have a single controller (slurmctld) node, an accounting db node> (slurmdbd), and 10 compute/worker nodes (slurmd.)<br>
<br>
Time to start upgrading to Ubuntu 18.04 now then? :-)<br>
<br>
For a 10 node cluster it might make more sense to run slurmctld and<br>
slurmdbd on the same hardware as neither have very high hardware<br>
requirements.<br>
<br>
On our current clusters we run both services on the same machine. The<br>
main disadvantage of this is that it makes upgrades inconvenient as it<br>
prevents upgrading slurmdbd and slurmctld independently. For future<br>
installations we will probably try running slurmdbd in a VM.<br>
<br>
> The problem is that the controller is still running CentOS 7 with our<br>
> older NFS-mounted /home scheme, but the compute nodes are now all Ubuntu<br>
> 16.04 with local /home fs’s.<br>
<br>
Does each user have a different local home directory on each compute<br>
node? That is not something I would recommend, unless you are very good<br>
at training your users to avoid submitting jobs in their home<br>
directories. I assume you have some other shared file system across the<br>
cluster?<br>
<br>
> 1) Can we leave the current controller machine on C7 OS, and just<br>
> have the users log into other machines (that have the same config as the<br>
> compute nodes) to submit jobs?<br>
> Or should the controller node really be<br>
> on the same OS as the compute nodes for some reason?<br>
<br>
I recommend separating them, for systems administration and user<br>
convenience reasons.<br>
<br>
With users logged into the the same machine that is running your<br>
controller or other cluster services, the users can impact the operation<br>
of the entire cluster when they make mistakes. Typical user mistakes<br>
involves using all CPU resources, using all memory, filling up or<br>
overloading filesystems... Much better to have this happen on dedicated<br>
login machines.<br>
<br>
If the login machine uses a different OS than the worker nodes, users<br>
will also run into problems if they compile software there, as system<br>
library versions won't match what is available on the compute nodes.<br>
<br>
Technically as long as you use the same Slurm version it should work.<br>
You should however check that your Slurm binaries on different OS are<br>
build with the exact same features enabled. Many are enabled at compile<br>
time, so check and compare the output from ./configure.<br>
<br>
> 2) Can I add a backup controller node that runs a different...<br>
> 3) What are the steps to replace a primary controller, given that a<br>
...<br>
We are not currently using a backup controller, so I can't answer that part.<br>
<br>
slurmctld keeps it state files in the directory configured as<br>
StateSaveLocation, so for slurmctld you typically only need to save the<br>
configuration files, and that directory. (note this does not include<br>
munge or the slurmdbd)<br>
<br>
Regards,<br>
Pär Lindfors, NSC<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>