<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.m-4729733019118038510gmail-m-6044234129629131185font
{mso-style-name:m_-4729733019118038510gmail-m_-6044234129629131185font;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Actually, I’ve gotten it all working; I just had overlooked some things in slurm.conf.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I had previously trying to get Son of Grid Engine working, but after ripping out half my hair, I went to Slurm, since it’s what our research computing center uses.
<span style="font-family:"Segoe UI Emoji",sans-serif">😊</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Matt Hohmeister<o:p></o:p></p>
<p class="MsoNormal">Systems and Network Administrator<o:p></o:p></p>
<p class="MsoNormal">Department of Psychology<o:p></o:p></p>
<p class="MsoNormal">Florida State University<o:p></o:p></p>
<p class="MsoNormal">PO Box 3064301<o:p></o:p></p>
<p class="MsoNormal">Tallahassee, FL 32306-4301<o:p></o:p></p>
<p class="MsoNormal">Phone: +1 850 645 1902<o:p></o:p></p>
<p class="MsoNormal">Fax: +1 850 644 7739<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Sean Caron<br>
<b>Sent:</b> Wednesday, April 11, 2018 3:36 PM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com>; Sean Caron <scaron@umich.edu><br>
<b>Subject:</b> Re: [slurm-users] FSU & Slurm<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi Matt,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">As a protest to asking questions on this list and getting solicitations for pay-for support, let me give you some advice for free :)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If you look at your slurm.conf you'll see there are two directories that your slurm user and group need to have write access to.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">One is whatever you configure as SlurmdSpoolDir. This needs to be available on all worker nodes that are running slurmd. Set ownership to slurm user and slurm group and mode 755.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The other is StateSaveLocation. This needs to be present just on your controller (where slurmctld runs). Again, this should have ownership of slurm user and slurm group and mode 755.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">You probably want to use something more specific than just /var/spool for your StateSaveLocation<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Sean<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Apr 11, 2018 at 1:48 PM, Jess Arrington <<a href="mailto:jess@schedmd.com" target="_blank">jess@schedmd.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="color:#0B5394">Hi Matt,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#0B5394"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222;background:white">I hope your day is treating you well.<br>
<br>
</span><span style="font-family:"Arial",sans-serif;background:white"><br>
</span><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222;background:white">Thank you for your posts on the Slurm user list. </span><span style="font-family:"Arial",sans-serif;background:white"><br>
<br>
<br>
</span><span style="font-size:9.5pt;font-family:"Arial",sans-serif;background:white">By chance, do you work with Paul Van Der Mark?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;background:white"><br>
Would there be interest on your side to see a Slurm support contract for your systems at </span><span style="font-size:9.5pt;font-family:"Arial",sans-serif">FSU<span style="color:#222222">?</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222;background:white"><br>
Sites running Slurm with support give us feedback that support is invaluable and a great return back to the organization with much better system utilization with optimized configs by our experts <span class="m-4729733019118038510gmail-m-6044234129629131185font">(which
pays for the support contract in and of itself) </span>and their sites not having to rely on in-house best effort support hacks that get very expensive and turn into complicated chaos and potential down systems.<br>
<br>
<br>
Additionally, support keeps the Slurm project alive and going strong <br>
<br>
<br>
Please let me know your thoughts or if you would like me to reach out to another contact at FSU to chat about this further.</span><span style="color:#0B5394"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#222222"><br>
<br>
<br>
Take care,<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#0B5394"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#0B5394"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:9.5pt">Jess Arrington</span></b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Director of Sales | 801-616-7823
<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D204-2BN-2B1200-2BE-2B-2523203-2BLehi-2C-2BUT-2B84043-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=HPMtquzZjKY31rtkyGRFnQ&r=Y7_jKRiyJUHl8NulOtnzB4UPVSMWmGk9Sds6aXi7m3U&m=8yB0SkVystFGD-TJ40nZcctsJi2KZKItVMPqZCkrGl0&s=rftyBrt83wE0kthyAQSLm2BY7GFZE6IA7LKFL2AOfbg&e=">204
N 1200 E #203 Lehi, UT 84043</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Apr 11, 2018 at 6:26 AM, Matt Hohmeister <<a href="mailto:hohmeister@psy.fsu.edu" target="_blank">hohmeister@psy.fsu.edu</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’m brand-new to Slurm, and setting it up on a single RHEL 7.4 VM as a proof of concept before I deploy it. After following the instructions on
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.slothparadise.com_how-2Dto-2Dinstall-2Dslurm-2Don-2Dcentos-2D7-2Dcluster_&d=DwMFaQ&c=HPMtquzZjKY31rtkyGRFnQ&r=Y7_jKRiyJUHl8NulOtnzB4UPVSMWmGk9Sds6aXi7m3U&m=8yB0SkVystFGD-TJ40nZcctsJi2KZKItVMPqZCkrGl0&s=_q5sO-LaJk4lnhH0SfJWMgyuoX8UBSrQ8xm09qfEKTE&e=" target="_blank">
https://www.slothparadise.com/how-to-install-slurm-on-centos-7-cluster/</a> (sorry, site not working now), I can get slurmd to start perfectly, but slurmctld fails to start with the following
<span style="font-family:Consolas">journalctl -xe</span>; I was wondering if anyone has run into this or could shed some light on this…thanks in advance!<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">Apr 11 08:18:30 psy-slurm polkitd[680]: Registered Authentication Agent for unix-process:1779:31362 (system bus name :1.26 [/usr/bin/pkttyagent --notify-fd 5 --fallbac</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">Apr 11 08:18:30 psy-slurm systemd[1]: Starting Slurm controller daemon...</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">-- Subject: Unit slurmctld.service has begun start-up</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">-- Defined-By: systemd</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">-- Support: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.freedesktop.org_mailman_listinfo_systemd-2Ddevel&d=DwMFaQ&c=HPMtquzZjKY31rtkyGRFnQ&r=Y7_jKRiyJUHl8NulOtnzB4UPVSMWmGk9Sds6aXi7m3U&m=8yB0SkVystFGD-TJ40nZcctsJi2KZKItVMPqZCkrGl0&s=IblrcsfHqVpgFyEMwN0EEP79-4O-Hu67St1xNF1e734&e=" target="_blank">
http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">--</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">-- Unit slurmctld.service has begun starting up.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">Apr 11 08:18:30 psy-slurm systemd[1]: PID file /var/run/slurmctld.pid not readable (yet?) after start.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">Apr 11 08:18:30 psy-slurm systemd[1]: Started Slurm controller daemon.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">-- Subject: Unit slurmctld.service has finished start-up</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">-- Defined-By: systemd</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">-- Support: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.freedesktop.org_mailman_listinfo_systemd-2Ddevel&d=DwMFaQ&c=HPMtquzZjKY31rtkyGRFnQ&r=Y7_jKRiyJUHl8NulOtnzB4UPVSMWmGk9Sds6aXi7m3U&m=8yB0SkVystFGD-TJ40nZcctsJi2KZKItVMPqZCkrGl0&s=IblrcsfHqVpgFyEMwN0EEP79-4O-Hu67St1xNF1e734&e=" target="_blank">
http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">--</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">-- Unit slurmctld.service has finished starting up.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">--</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">-- The start-up result is done.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">Apr 11 08:18:30 psy-slurm polkitd[680]: Unregistered Authentication Agent for unix-process:1779:31362 (system bus name :1.26, object path /org/freedesktop/PolicyKit1/A</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">Apr 11 08:18:30 psy-slurm slurmctld[1787]: fatal: Incorrect permissions on state save loc: /var/spool</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">Apr 11 08:18:30 psy-slurm systemd[1]: slurmctld.service: main process exited, code=exited, status=1/FAILURE</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">Apr 11 08:18:30 psy-slurm systemd[1]: Unit slurmctld.service entered failed state.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:Consolas">Apr 11 08:18:30 psy-slurm systemd[1]: slurmctld.service failed.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Matt Hohmeister<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Systems and Network Administrator<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Department of Psychology<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Florida State University<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">PO Box 3064301<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tallahassee, FL 32306-4301<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Phone: +1 850 645 1902<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Fax: +1 850 644 7739<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>