<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Courier New \;color\:\#1F497D";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Courier New \;color\:purple";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.gmail-m2476182093536264129gmail-m7923536546226815014msolistparagraph, li.gmail-m2476182093536264129gmail-m7923536546226815014msolistparagraph, div.gmail-m2476182093536264129gmail-m7923536546226815014msolistparagraph
{mso-style-name:gmail-m_2476182093536264129gmail-m7923536546226815014msolistparagraph;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-postmall21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1609309107;
mso-list-type:hybrid;
mso-list-template-ids:-582049804 343605222 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Many thanks Brian!<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Your comments pushed me in the right direction. Slurm.conf is in a shared directory, so that was not a problem. However, having the DNS properly set up on the master and running in slurm in config-less mode enabled slurmctld on the master and slurmd on the client to communicate. Getting so far, the rest is rather downhill. I had some later hick-ups getting everything to work, in particular setting cgroup up, but a high debug level gave sufficient information to sort the problems out one by one. Now, slurm runs properly. Again, many thanks!<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Best regards,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Lars<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Från:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> slurm-users [mailto:slurm-users-bounce</span><span lang=SV style='font-size:11.0pt;font-family:"Calibri",sans-serif'>s@lists.schedmd.com] <b>För </b>Brian Andrus<br><b>Skickat:</b> den 24 augusti 2020 16:24<br><b>Till:</b> slurm-users@lists.schedmd.com<br><b>Ämne:</b> Re: [slurm-users] [EXT] Slurmd problem on client<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p>IIRC, that is because it is trying to do the 'configless' feature of slurm 20 where it uses DNS entries to find the config.<o:p></o:p></p><p>This will happen if /etc/slurm.conf does not exist on the node.<o:p></o:p></p><p>Check that you have that and that it is the same as the one on the master.<o:p></o:p></p><p>Brian Andrus<o:p></o:p></p><div><p class=MsoNormal>On 8/24/2020 7:03 AM, Lars Kloo wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Dear Sean,</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>’/usr/local/sbin/slurmd -D –vvvv’ gave the following error (same as when running from systemctl):</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>slurmd: error: _fetch_child: failed to fetch remote configs</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>I have debug level 5 for both slurmctld and slurmd in slurm.conf, so there may be little more to extract in form of messages.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>I am starting to think that the error is in the set-up files of named, alternatively in the network interface scripts. They should work, but slurmd seems to require more.</span><o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='font-family:"Calibri",sans-serif'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>/etc/resolv.conf looks correct with both internal and external nameservers and domains on the master, and only the internal on the client</span><o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='font-family:"Calibri",sans-serif'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>However, tracking the master named log file while starting slurmd on the client, it looks like slurmd is not offered the internal domain. When those attempts are exhausted, slurmd is directed to the external nameserver/domain (which will not give the information necessary). The difference is that the external domain is explicitly given in the named log file, whereas the internal domain is not.</span><o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='font-family:"Calibri",sans-serif'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Disabling IPv6 in /etc/named.conf removes the error messages in the named log file, but the above slurmd error persists. </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Possibly, my approach to solving the DNS/SRV problem may be too primitive.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Best regards,</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Lars</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'> </span><o:p></o:p></p><p class=MsoNormal><b><span lang=SV style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Från:</span></b><span lang=SV style='font-size:11.0pt;font-family:"Calibri",sans-serif'> slurm-users [<a href="mailto:slurm-users-bounces@lists.schedmd.com">mailto:slurm-users-bounces@lists.schedmd.com</a>] <b>För </b>Sean Crosby<br><b>Skickat:</b> den 24 augusti 2020 13:44<br><b>Till:</b> Slurm User Community List <a href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br><b>Ämne:</b> Re: [slurm-users] [EXT] Slurmd problem on client</span><o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><div><div><p class=MsoNormal>Make sure slurmd on the client is stopped, and then run it in verbose mode in the foreground<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal>e.g.<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal>/usr/local/slurm/latest/sbin/slurmd -D -vvvvv<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal>Then post the output<o:p></o:p></p></div><div><div><div><p class=MsoNormal style='margin-bottom:12.0pt'>--<br>Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead<br>Research Computing Services | Business Services<br>The University of Melbourne, Victoria 3010 Australia<o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><div><div><p class=MsoNormal>On Mon, 24 Aug 2020 at 21:11, Lars Kloo <<a href="mailto:larsa@kth.se">larsa@kth.se</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><table class=MsoNormalTable border=1 cellspacing=0 cellpadding=0 width="100%" style='width:100.0%;background:lemonchiffon;float:left'><tr><td style='padding:3.75pt 3.75pt 3.75pt 3.75pt'><p class=MsoNormal><b>UoM notice: </b>External email. Be cautious of links, attachments, or impersonation attempts<o:p></o:p></p></td></tr></table><p class=MsoNormal> <o:p></o:p></p><div class=MsoNormal align=center style='text-align:center'><hr size=2 width="100%" align=center></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Thanks Sean,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Yes, the regular slurm commands work from the client.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>The firewalld daemon have been stopped/disabled, and iptables are set to let everything through, on both the master and the client. I should have mentioned that in the list of prerequisites in my initial e-mail.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Best regards,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Lars</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span lang=SV style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Från:</span></b><span lang=SV style='font-size:11.0pt;font-family:"Calibri",sans-serif'> slurm-users [mailto:<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>] <b>För </b>Sean Crosby<br><b>Skickat:</b> den 24 augusti 2020 12:45<br><b>Till:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br><b>Ämne:</b> Re: [slurm-users] [EXT] Slurmd problem on client</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hi Lars,<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Do the regular slurm commands work from the client?<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>e.g. <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>squeue<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>scontrol show part<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>If they don't, it would be a sign of communication problems.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Is there a software firewall running on the master/client? <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Sean<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'>--<br>Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead<br>Research Computing Services | Business Services<br>The University of Melbourne, Victoria 3010 Australia<o:p></o:p></p></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Mon, 24 Aug 2020 at 20:02, Lars Kloo <<a href="mailto:larsa@kth.se" target="_blank">larsa@kth.se</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor
currentcolor currentcolor rgb(204,204,204)'><div><div><table class=MsoNormalTable border=1 cellspacing=0 cellpadding=0 width="100%" style='width:100.0%;background:lemonchiffon;background-position-x:0%;background-position-y:0%;background-attachment:scroll;float:left'><tr><td style='padding:3.75pt 3.75pt 3.75pt 3.75pt'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b>UoM notice: </b>External email. Be cautious of links, attachments, or impersonation attempts<o:p></o:p></p></td></tr></table><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div class=MsoNormal align=center style='text-align:center'><hr size=2 width="100%" align=center></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hello,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I have a client slurmd problem, that I cannot really figure out how to solve. I would be grateful for any suggestions on how to move forward.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The master computer on a small local calculational cluster is getting quite old, and therefore I am currently in the process of exchanging it. I also use one calculational node for the basic master-client set-up of all programs, including slurm. Some basic data: CentOS 7.7, slurm 20.02.4.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Setting up the systemctld on the master node is (seemingly) straightforward. Getting slurmd to work on the client appears more complicated. I get the following error message (journalctl –xe) when starting slurmd on the client:<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Aug 24 11:01:34 cpu3.calc.cluster slurmd[9002]: error: _fetch_child: failed to fetch remote configs<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>No useful error messages are obtained from ‘systemctl –l status slurmd.service’ on the client, slurmd.log on the client, nor slurmctld.log on the master.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>In this context, the following should be noted:<o:p></o:p></p><p class=gmail-m2476182093536264129gmail-m7923536546226815014msolistparagraph>-<span style='font-size:7.0pt'> </span>root and test user exist on the master and client; same uid and gid on both machines<o:p></o:p></p><p class=gmail-m2476182093536264129gmail-m7923536546226815014msolistparagraph>-<span style='font-size:7.0pt'> </span>ping works in both directions (master <-> client)<o:p></o:p></p><p class=gmail-m2476182093536264129gmail-m7923536546226815014msolistparagraph>-<span style='font-size:7.0pt'> </span>passphrase-free ssh login work in both directions for both root and for a test user<o:p></o:p></p><p class=gmail-m2476182093536264129gmail-m7923536546226815014msolistparagraph>-<span style='font-size:7.0pt'> </span>munged is running and with the same key on both machines<o:p></o:p></p><p class=gmail-m2476182093536264129gmail-m7923536546226815014msolistparagraph>-<span style='font-size:7.0pt'> </span>the same slurm.conf is read from the master and from the client<o:p></o:p></p><p class=gmail-m2476182093536264129gmail-m7923536546226815014msolistparagraph>-<span style='font-size:7.0pt'> </span>named (bind) has been set up on the master, and nslookup and dig work properly on the client<o:p></o:p></p><p class=gmail-m2476182093536264129gmail-m7923536546226815014msolistparagraph>-<span style='font-size:7.0pt'> </span>the ‘forward’ zone file of named on the master (DNS) contains the recommended SRV record directing slurmctld requests to port 6817 on the master (syntax seems ok, i.e. no error messages)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I have also tried to start slurmd in a config-less mode (slurm.conf edited on the master) with the suggested environment variable set (slurmd on the client). Then, slurmd starts without error messages, but slurmctld on the master cannot communicate with slurmd on the client.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Has anyone encountered a similar problem --- and how did you solve it? Or, do you have any suggestions where to start looking?<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Many thanks for input, and best regards,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Lars<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New ;color:#1F497D",serif'>//////////////////////////////~~~_/)~~~\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New ;color:#1F497D",serif'> Lars Kloo, Prof.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New ;color:#1F497D",serif'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New ;color:#1F497D",serif'> Tillämpad fysikalisk kemi Applied Physical Chemistry</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New ;color:#1F497D",serif'> Institutionen för kemi Dept. of Chemistry</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New ;color:#1F497D",serif'> Kungliga Tekniska högskolan Royal Inst. of Technology (KTH)</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Courier New ;color:#1F497D",serif'> </span><span lang=EN-US style='font-family:"Courier New ;color:#1F497D",serif'>100 44 STOCKHOLM SE-100 44 Stockholm</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-family:"Courier New ;color:#1F497D",serif'> SWEDEN</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-family:"Courier New ;color:#1F497D",serif'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-family:"Courier New ;color:#1F497D",serif'> Tel: 08-790 9343 Tel: +46-8-790 9343</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-family:"Courier New ;color:#1F497D",serif'> Fax: 08-790 9349 Fax: +46-8-790 9349</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-family:"Courier New ;color:#1F497D",serif'> E-post: </span><a href="mailto:lakloo@kth.se" target="_blank"><span lang=EN-US style='font-family:"Courier New ;color:purple",serif'>lakloo@kth.se</span></a><span lang=EN-US style='font-family:"Courier New ;color:#1F497D",serif'> E-mail: </span><a href="mailto:lakloo@kth.se" target="_blank"><span lang=EN-US style='font-family:"Courier New ;color:purple",serif'>lakloo@kth.se</span></a><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-family:"Courier New ;color:#1F497D",serif'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-family:"Courier New ;color:#1F497D",serif'> WWW: </span><a href="http://www.kth.se/che/divisions/tfk" target="_blank"><span lang=EN-US style='font-family:"Courier New ;color:purple",serif'>http://www.kth.se/che/divisions/tfk</span></a><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-family:"Courier New ;color:#1F497D",serif'>\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\~~~_/)~~~//////////////////////////////</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div></blockquote></div></div></div></blockquote></div></blockquote></div></body></html>