<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I’ve managed to track down the difference between the accounts which work and those which don’t – but I still don’t understand the
mechanism.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">The accounts which work all had their home directories used on an older system. The ones which fail were only ever used on the new
system. The relevant difference seems to be the way their ssh keys are set up. On the old system a standard ssh-keygen was run, creating ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub files and putting the pub file into authorized_keys.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">On the new warewulf based system ssh-keygen was again run, but the default key file names was changed. We now have ~/.ssh/cluster
and ~/.ssh/cluster.pub and there is a ~/.ssh/config file which contains:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"># Added by Warewulf 2019-12-10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Host pebble*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"> IdentityFile ~/.ssh/cluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"> StrictHostKeyChecking=no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">This all works fine, and I can ssh from the head node to the ‘pebble’ compute nodes just fine, however something in the code for the
slurm x11 forwarder is specifically looking for id_rsa files (or is ignoring the config file), since the forwarding fails if I don’t have these, and works as soon as I do.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Any ideas where this might be happening so I can either file a bug for change whatever setting this needs?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Simon.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>William Brown<br>
<b>Sent:</b> 24 January 2020 17:21<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Srun not setting DISPLAY with --x11 for one account<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">There are differences for X11 between Slurm versions so it may help to know which version you have.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I tried some of your commands on our slurm 19.05.3-2 cluster, and interestingly on the session on the compute node I don't see the cookie for the login node: This was with MobaXterm:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">[user@prdubrvm005 ~]$ xauth list<br>
<a href="http://prdubrvm005.research.rcsi.com/unix:10">prdubrvm005.research.rcsi.com/unix:10</a> MIT-MAGIC-COOKIE-1 2efc5dd851736e3848193f65d038eca8<br>
[user@prdubrvm005 ~]$ srun --pty --x11 --preserve-env /bin/bash<br>
[user@prdubrhpc1-02 ~]$ xauth list<br>
<a href="http://prdubrhpc1-02.research.rcsi.com/unix:95">prdubrhpc1-02.research.rcsi.com/unix:95</a> MIT-MAGIC-COOKIE-1 2efc5dd851736e3848193f65d038eca8<br>
[user@prdubrhpc1-02 ~]$ echo $DISPLAY<br>
localhost:95.0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Any per-user problem would make me suspect the user having a different shell, or something in their login script. Can you make their .bashrc and .bash_profile just exit? Or look for hidden configuration files for <something> in their
home directory?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">William<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, 24 Jan 2020 at 16:05, Simon Andrews <<a href="mailto:simon.andrews@babraham.ac.uk">simon.andrews@babraham.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I have a weird problem which I can’t get to the bottom of.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We have a cluster which allows users to start interactive sessions which forward any X11 sessions they generated on the head node. This generally works fine, but on the account
of one user it doesn’t work. The X11 connection to the head node is fine, but it won’t transfer to the compute node.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The symptoms are shown below:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">A good user gets this:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[good@headnode ~]$ xauth list<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="http://headnode.babraham.ac.uk/unix:12" target="_blank">headnode.babraham.ac.uk/unix:12</a> MIT-MAGIC-COOKIE-1 f04a2bf9a921a3357e44373655add14a<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[good@headnode ~]$ echo $DISPLAY<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">localhost:12.0<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[good@headnode ~]$ srun --pty -p interactive --x11 --preserve-env /bin/bash<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[good@compute ~]$ xauth list<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="http://headnode.babraham.ac.uk/unix:12" target="_blank">headnode.babraham.ac.uk/unix:12</a> MIT-MAGIC-COOKIE-1 f04a2bf9a921a3357e44373655add14a<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">compute/unix:25 MIT-MAGIC-COOKIE-1 f04a2bf9a921a3357e44373655add14a<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[good@compute ~]$ echo $DISPLAY<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">localhost:25.0<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">So the cookie is copied from the head node and forwarded and the DISPLAY variable is updated.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The bad user gets this:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[bad@headnode ~]$ xauth list<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="http://headnode.babraham.ac.uk/unix:10" target="_blank">headnode.babraham.ac.uk/unix:10</a> MIT-MAGIC-COOKIE-1 c39a493a37132d308b37469d363d8692<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[bad@headnode ~]$ echo $DISPLAY<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">localhost:10.0<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[bad@headnode ~]$ srun --pty -p interactive --x11 --preserve-env /bin/bash<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[bad@compute ~]$ xauth list<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="http://headnode.babraham.ac.uk/unix:10" target="_blank">headnode.babraham.ac.uk/unix:10</a> MIT-MAGIC-COOKIE-1 c39a493a37132d308b37469d363d8692<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[bad@compute ~]$ echo $DISPLAY<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">localhost:10.0<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">So the cookie isn’t copied and the DISPLAY isn’t updated. I can’t see any errors in the logs and I can’t see anything different about this account.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">If I do a straight forward ssh -Y from the head node to a compute node from the bad account then that works fine – it’s only whatever is specific about the way that srun forwards
X which fails.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Any ideas or suggestions for debugging would be appreciated as I’m running out of things to try!<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Simon.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#1F497D">The Babraham Institute, Babraham Research Campus, Cambridge CB22 3AT
</span><i><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#365F91">Registered Charity No. 1053902.</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#1F497D">The information transmitted in this email is directed only to the addressee. If you received this in error,
please contact the sender and delete this email from your system. The contents of this e-mail are the views of the sender and do not necessarily represent the views of the Babraham Institute.
</span><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#365F91">Full conditions at:
</span><a href="http://www.babraham.ac.uk/terms" target="_blank"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#365F91">www.babraham.ac.uk</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
<p class="MsoNormal" style="text-autospace:none"><o:p></o:p><o:p></o:p><span style="font-size:8.0pt;
font-family:"Arial","sans-serif";color:#1F497D">The Babraham Institute, Babraham Research Campus, Cambridge CB22 3AT
</span><i><span style="font-size:
8.0pt;font-family:"Arial","sans-serif";color:#365F91;mso-themecolor:accent1;
mso-themeshade:191">Registered Charity No. 1053902.</span></i><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:8.0pt;
font-family:"Arial","sans-serif";color:#1F497D">The information transmitted in this email is directed only to the addressee. If you received this in error, please contact the sender
and delete this email from your system. The contents of this e-mail are the views of the sender and do not necessarily represent the views of the Babraham Institute.
</span><span style="font-size:8.0pt;
font-family:"Arial","sans-serif";color:#365F91;mso-themecolor:accent1;
mso-themeshade:191">Full conditions at:
</span><a href="http://www.babraham.ac.uk/terms"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";
color:#365F91;mso-themecolor:accent1;mso-themeshade:191">www.babraham.ac.uk</span></a><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#365F91;
mso-themecolor:accent1;mso-themeshade:191"><o:p></o:p></span></p>
</body>
</html>