<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi John,
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">Regarding NFS shares and Python, and plenty of other packages too,</div>
<div class="">pay attention to where the NFS server is located on your network.</div>
<div class="">The NFS server should be part of your cluster, or at least have a network interface on your cluster fabric.</div>
<div class=""><br class="">
</div>
<div class="">If you perhaps have a home directory server which is a campus NFS server and you are NATting via your head node,</div>
<div class="">then every time a parallel multimode job starts up you will pull in libraries multiple times and this will be a real</div>
<div class="">performance bottleneck.</div>
</div>
</blockquote>
</div>
<div class="">
<div dir="ltr" class="">
<div class="">The NFS server is part of the cluster (same IP subnet/vlan, I know that in the networking world it is a wrong assumption but the NFS server is physically in the same rack server as the cluster.</div>
<div class="">NFS server / headnode: <span style="font-family: Courier; background-color: rgb(255, 255, 255);" class="">inet 10.112.0.25</span><span style="font-family: Courier; background-color: rgb(255, 255, 255);" class="">
</span><span style="font-family: Courier; background-color: rgb(255, 255, 255);" class="">netmask 255.255.255.192</span><span style="font-family: Courier; background-color: rgb(255, 255, 255);" class="">
</span><span style="font-family: Courier; background-color: rgb(255, 255, 255);" class="">broadcast 10.112.0.63</span></div>
<div class="">Execute nodes (1 example):<span style="font-family: Menlo; font-size: 11px; background-color: rgb(255, 255, 255);" class="">inet 10.112.0.5</span><span style="font-family: Menlo; font-size: 11px; background-color: rgb(255, 255, 255);" class="">
</span><span style="font-family: Menlo; font-size: 11px; background-color: rgb(255, 255, 255);" class="">netmask 255.255.255.192</span><span style="font-family: Menlo; font-size: 11px; background-color: rgb(255, 255, 255);" class="">
</span><span style="font-family: Menlo; font-size: 11px; background-color: rgb(255, 255, 255);" class="">broadcast 10.112.0.63</span></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">You do have to have a home directory mounted on the nodes - either the users real home directory or something</div>
<div class="">which loosk like a home directory. Ooodles of software packages depend on dor files int eh home directory,</div>
<div class="">and you won't get far without one.</div>
</div>
</blockquote>
</div>
<div class="">Right now each node has a user home directory.</div>
<div class="">Do you suggest that i should move / create users' home directory to the NFS share?</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">Eric, my advice would be to definitely learn the Modules system and implement modules for your users.</div>
</div>
</blockquote>
I definitely have to learn more about Modules system and their implementation. My work takes more into that direction.</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">Also if you could give us some idea of your storage layout this would be good.</div>
</div>
</blockquote>
</div>
<div class="">I hope this what you meant</div>
<div class=""><br class="">
</div>
<div class="">Headnode:</div>
<div class="">
<div style="margin: 0px; line-height: normal; font-family: Courier; color: rgb(0, 164, 0); background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><b class="">eric@radoncmaster</b></span><span style="font-variant-ligatures: no-common-ligatures; color: #4d2f2d" class="">:</span><span style="font-variant-ligatures: no-common-ligatures; color: #4324d4" class=""><b class="">/</b></span><span style="font-variant-ligatures: no-common-ligatures; color: #4d2f2d" class="">$
df -h</span></div>
<div style="margin: 0px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Filesystem Size Used Avail Use% Mounted on</span></div>
<div style="margin: 0px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">udev 7.8G 0 7.8G 0% /dev</span></div>
<div style="margin: 0px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 1.6G 740K 1.6G 1% /run</span></div>
<div style="margin: 0px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">/dev/sda1 902G 3.3G 853G 1% /</span></div>
<div style="margin: 0px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 7.9G 0 7.9G 0% /dev/shm</span></div>
<div style="margin: 0px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 5.0M 0 5.0M 0% /run/lock</span></div>
<div style="margin: 0px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup</span></div>
<div style="margin: 0px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">/dev/sdb1 3.6T 572M 3.4T 1% /media/cluster</span></div>
<div style="margin: 0px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 1.6G 0 1.6G 0% /run/user/1000</span></div>
</div>
<div class=""><br class="">
</div>
<div class="">Execute node (1 example)</div>
<div class="">
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><b class="">eric@radonc01</b>:<b class="">~</b>$ df -h</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Filesystem Size Used Avail Use% Mounted on</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">udev 32G 0 32G 0% /dev</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 6.3G 984K 6.3G 1% /run</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">/dev/mapper/radonc01--vg-root 2.7T 2.5G 2.5T 1% /</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 32G 0 32G 0% /dev/shm</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 5.0M 0 5.0M 0% /run/lock</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 32G 0 32G 0% /sys/fs/cgroup</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">/dev/sda2 473M 128M 321M 29% /boot</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">10.112.0.25:/media/cluster 3.6T 571M 3.4T 1% /nfs/cluster</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">tmpfs 6.3G 0 6.3G 0% /run/user/1000</span></div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="text-align: -webkit-auto; orphans: 2; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="orphans: auto; widows: auto;" class=""><span style="text-align: -webkit-auto; background-color: rgba(255, 255, 255, 0);" class="">_____________________________________________________________________________________________________</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><br class="">
</span></div>
<span style="background-color: rgba(255, 255, 255, 0);" class=""><b class="">
<div style="orphans: auto; widows: auto;" class=""><b style="text-align: -webkit-auto;" class="">Eric F. Alemany</b></div>
</b>
<div style="orphans: auto; widows: auto;" class=""><i style="text-align: -webkit-auto;" class="">System Administrator for Research</i></div>
</span>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><br class="">
</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="text-align: -webkit-auto; background-color: rgba(255, 255, 255, 0);" class="">Division of Radiation & Cancer Biology</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="text-align: -webkit-auto; background-color: rgba(255, 255, 255, 0);" class="">Department of Radiation Oncology</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><br class="">
</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="text-align: -webkit-auto; background-color: rgba(255, 255, 255, 0);" class="">Stanford University School of Medicine</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="text-align: -webkit-auto; background-color: rgba(255, 255, 255, 0);" class="">Stanford, California 94305</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><br class="">
</span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><font style="text-align: -webkit-auto;" class="">Tel:</font><a href="tel:1-650-498-7969" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="1" style="text-align: -webkit-auto;" class="">1-650-498-7969</a><font style="text-align: -webkit-auto;" class="">
No Texting</font></span></div>
<div style="orphans: auto; widows: auto;" class=""><span style="background-color: rgba(255, 255, 255, 0);" class=""><font style="text-align: -webkit-auto;" class="">Fax:</font><a href="tel:1-650-723-7382" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="2" style="text-align: -webkit-auto;" class="">1-650-723-7382</a></span></div>
<div style="orphans: auto; widows: auto;" class=""><br class="">
</div>
</div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<font style="background-color: rgba(255, 255, 255, 0);" class=""></font></div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On May 11, 2018, at 12:11 AM, John Hearns <<a href="mailto:hearnsj@googlemail.com" class="">hearnsj@googlemail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="">Regarding NFS shares and Python, and plenty of other packages too,</div>
<div class="">pay attention to where the NFS server is located on your network.</div>
<div class="">The NFS server should be part of your cluster, or at least have a network interface on your cluster fabric.</div>
<div class=""><br class="">
</div>
<div class="">If you perhaps have a home directory server which is a campus NFS server and you are NATting via your head node,</div>
<div class="">then every time a parallel multimode job starts up you will pull in libraries multiple times and this will be a real</div>
<div class="">performance bottleneck.</div>
<div class=""><br class="">
</div>
<div class="">You do have to have a home directory mounted on the nodes - either the users real home directory or something</div>
<div class="">which loosk like a home directory. Ooodles of software packages depend on dor files int eh home directory,</div>
<div class="">and you won't get far without one.</div>
<div class=""><br class="">
</div>
<div class="">Eric, my advice would be to definitely learn the Modules system and implement modules for your users.</div>
<div class="">Also if you could give us some idea of your storage layout this would be good.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On 11 May 2018 at 08:55, Miguel Gutiérrez Páez <span dir="ltr" class="">
<<a href="mailto:mgutierrez@gmail.com" target="_blank" class="">mgutierrez@gmail.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">
<div class="">Hi,<br class="">
<br class="">
</div>
I install all my apps in a shared storage, and change environment variables (path, vars, etc.) with lmod. It's very useful.<br class="">
<br class="">
</div>
Regards.<br class="">
</div>
<div class="HOEnZb">
<div class="h5"><br class="">
<div class="gmail_quote">
<div dir="ltr" class="">El vie., 11 may. 2018 a las 6:19, Eric F. Alemany (<<a href="mailto:ealemany@stanford.edu" target="_blank" class="">ealemany@stanford.edu</a>>) escribió:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Hi Lachlan,
<div class=""><br class="">
</div>
<div class="">Thank you for sharing your environment. Everyone has their own set of rules and i appreciate everyone’s input.</div>
<div class="">It seems as if the NFS share is a great place to start.</div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">Eric</div>
<div class=""></div>
</div>
<div style="word-wrap:break-word" class="">
<div class="">
<div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="text-align:-webkit-auto;word-wrap:break-word" class="">
<div class=""><span style="text-align:-webkit-auto;background-color:rgba(255,255,255,0)" class="">______________________________<wbr class="">______________________________<wbr class="">______________________________<wbr class="">___________</span></div>
<div class=""><span style="background-color:rgba(255,255,255,0)" class=""><br class="">
</span></div>
<span style="background-color:rgba(255,255,255,0)" class=""><b class="">
<div class=""><b style="text-align:-webkit-auto" class="">Eric F. Alemany</b></div>
</b>
<div class=""><i style="text-align:-webkit-auto" class="">System Administrator for Research</i></div>
</span>
<div class=""><span style="background-color:rgba(255,255,255,0)" class=""><br class="">
</span></div>
<div class=""><span style="text-align:-webkit-auto;background-color:rgba(255,255,255,0)" class="">Division of Radiation & Cancer Biology</span></div>
<div class=""><span style="text-align:-webkit-auto;background-color:rgba(255,255,255,0)" class="">Department of Radiation Oncology</span></div>
<div class=""><span style="background-color:rgba(255,255,255,0)" class=""><br class="">
</span></div>
<div class=""><span style="text-align:-webkit-auto;background-color:rgba(255,255,255,0)" class="">Stanford University School of Medicine</span></div>
<div class=""><span style="text-align:-webkit-auto;background-color:rgba(255,255,255,0)" class="">Stanford, California 94305</span></div>
<div class=""><span style="background-color:rgba(255,255,255,0)" class=""><br class="">
</span></div>
<div class=""><span style="background-color:rgba(255,255,255,0)" class=""><font style="text-align:-webkit-auto" class="">Tel:</font><a style="text-align:-webkit-auto" href="tel:1-650-498-7969" target="_blank" class="">1-650-498-7969</a><font style="text-align:-webkit-auto" class="">
No Texting</font></span></div>
<div class=""><span style="background-color:rgba(255,255,255,0)" class=""><font style="text-align:-webkit-auto" class="">Fax:</font><a style="text-align:-webkit-auto" href="tel:1-650-723-7382" target="_blank" class="">1-650-723-7382</a></span></div>
<div class=""><br class="">
</div>
</div>
<div style="word-wrap:break-word" class=""><font style="background-color:rgba(255,255,255,0)" class=""></font></div>
</div>
</div>
<br class="m_7121086110896900777m_7148566682384290610Apple-interchange-newline">
</div>
<br class="">
</div>
</div>
<div style="word-wrap:break-word" class="">
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On May 10, 2018, at 4:23 PM, Lachlan Musicman <<a href="mailto:datakid@gmail.com" target="_blank" class="">datakid@gmail.com</a>> wrote:</div>
<br class="m_7121086110896900777m_7148566682384290610Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="gmail_extra">
<div class="gmail_quote">On 11 May 2018 at 01:35, Eric F. Alemany <span dir="ltr" class="">
<<a href="mailto:ealemany@stanford.edu" target="_blank" class="">ealemany@stanford.edu</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto" class="">
<div class="">Hi All,</div>
<div id="m_7121086110896900777m_7148566682384290610m_6404797890502024991AppleMailSignature" class="">
<br class="">
</div>
<div id="m_7121086110896900777m_7148566682384290610m_6404797890502024991AppleMailSignature" class="">
I know this might sounds as a very basic question: where in the cluster should I install Python and R?</div>
<div id="m_7121086110896900777m_7148566682384290610m_6404797890502024991AppleMailSignature" class="">
Headnode?</div>
<div id="m_7121086110896900777m_7148566682384290610m_6404797890502024991AppleMailSignature" class="">
Execute nodes ?</div>
<div id="m_7121086110896900777m_7148566682384290610m_6404797890502024991AppleMailSignature" class="">
<br class="">
</div>
<div id="m_7121086110896900777m_7148566682384290610m_6404797890502024991AppleMailSignature" class="">
And is there a particular directory (path) I need to install Python and R.<br class="">
<br class="">
Background:</div>
<div id="m_7121086110896900777m_7148566682384290610m_6404797890502024991AppleMailSignature" class="">
SLURM on Ubuntu 18.04</div>
<div id="m_7121086110896900777m_7148566682384290610m_6404797890502024991AppleMailSignature" class="">
1 headnode </div>
<div id="m_7121086110896900777m_7148566682384290610m_6404797890502024991AppleMailSignature" class="">
4 execute nodes</div>
<div id="m_7121086110896900777m_7148566682384290610m_6404797890502024991AppleMailSignature" class="">
NFS shared drive among all nodes.</div>
</div>
</blockquote>
</div>
<br class="">
<br class="">
</div>
<div class="gmail_extra">Eric,<br class="">
<br class="">
</div>
<div class="gmail_extra">To echo the others: we have a /binaries nfs share that utilises the standard Environment Modules software so that researchers can manipulate their $PATH on the fly with module load/module unload. That share is mounted on all the nodes.<br class="">
<br class="">
</div>
<div class="gmail_extra">For Python, I use virtualenv's but instead of activating, the path is changed by the Module file. Personally, I find conda doesn't work very well in a shared environment. It's fine on a personal level/<br class="">
<br class="">
</div>
<div class="gmail_extra">For R, we have resorted to only installing the main point release because we have >700 libraries installed within R and I don't want to reinstall them every time. We do also have packrat installed so researchers can install their own
libraries locally as well. <br class="">
</div>
<div class="gmail_extra"><br class="">
</div>
<div class="gmail_extra"></div>
<div class="gmail_extra"><br class="">
</div>
<div class="gmail_extra">Cheers<br class="">
</div>
<div class="gmail_extra">L.<br class="">
</div>
<div class="gmail_extra"><br class="">
</div>
<div class="gmail_extra"><br class="">
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>