<!DOCTYPE html>
<html>
<head>
<title></title>
<style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style>
</head>
<body><div style="font-size:16px;">Sorry for the late response. <br></div>
<div style="font-size:16px;"><br></div>
<div style="font-size:16px;">The erratic behaviour of group affiliations seemed inconsistent. Some of the nodes experienced such oddity while other nodes were able to display the group memberships properly. <br></div>
<div style="font-size:16px;"><br></div>
<div style="font-size:16px;">I tried setting LaunchParameters=send_gids as per Douglas Jacobsen's suggestion by doing<br></div>
<div style="font-size:16px;">1. Changing slurm.conf to add LaunchParameters=send_gids<br></div>
<div style="font-size:16px;">2. Synchronising slurm.conf across the cluster.<br></div>
<div style="font-size:16px;">3. Restarting slurmctld<br></div>
<div style="font-size:16px;">4. scontrol reconfigure<br></div>
<div style="font-size:16px;"><br></div>
<div style="font-size:16px;">and so far, it appears to have made our problem go away. Users's group affiliations reflect correctly as expected. <br></div>
<div style="font-size:16px;"><br></div>
<div style="font-size:16px;">Thanks,</div>
<div id="sig56753105"><div class="signature">--<br></div>
<div class="signature"> Aravindh Sampathkumar<br></div>
<div class="signature"> aravindh@fastmail.com<br></div>
<div class="signature"><br></div>
</div>
<div><br></div>
<div><br></div>
<div>On Tue, Nov 13, 2018, at 10:21 AM, Antony Cleave wrote:<br></div>
<blockquote type="cite"><div><div style="font-size:16px;">Are you sure this isn't working as designed? <br></div>
<div><br></div>
<div>I remember there is something annoying about groups in the manual. Here it is. This is why I prefer accounts.<br></div>
<div><br></div>
<div><b style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-top-style:initial;border-right-style:initial;border-bottom-style:initial;border-left-style:initial;border-top-color:initial;border-right-color:initial;border-bottom-color:initial;border-left-color:initial;border-image-source:initial;border-image-slice:initial;border-image-width:initial;border-image-outset:initial;border-image-repeat:initial;font-size:18px;line-height:inherit;font-family:"source sans pro", helvetica, arial, sans-serif;vertical-align:baseline;color:rgb(70, 84, 92);background-color:rgb(255, 255, 255);">NOTE:</b><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(70, 84, 92)"><span class="font" style="font-family:"source sans pro", helvetica, arial, sans-serif"><span class="size" style="font-size:18px"> For performance reasons, Slurm maintains a list of user IDs allowed to use each partition and this is checked at job submission time. This list of user IDs is updated when the </span></span></span></span><b style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-top-style:initial;border-right-style:initial;border-bottom-style:initial;border-left-style:initial;border-top-color:initial;border-right-color:initial;border-bottom-color:initial;border-left-color:initial;border-image-source:initial;border-image-slice:initial;border-image-width:initial;border-image-outset:initial;border-image-repeat:initial;font-size:18px;line-height:inherit;font-family:"source sans pro", helvetica, arial, sans-serif;vertical-align:baseline;color:rgb(70, 84, 92);background-color:rgb(255, 255, 255);">slurmctld</b><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(70, 84, 92)"><span class="font" style="font-family:"source sans pro", helvetica, arial, sans-serif"><span class="size" style="font-size:18px">daemon is restarted, reconfigured (e.g. "scontrol reconfig") or the partition's </span></span></span></span><b style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-top-style:initial;border-right-style:initial;border-bottom-style:initial;border-left-style:initial;border-top-color:initial;border-right-color:initial;border-bottom-color:initial;border-left-color:initial;border-image-source:initial;border-image-slice:initial;border-image-width:initial;border-image-outset:initial;border-image-repeat:initial;font-size:18px;line-height:inherit;font-family:"source sans pro", helvetica, arial, sans-serif;vertical-align:baseline;color:rgb(70, 84, 92);background-color:rgb(255, 255, 255);">AllowGroups</b><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(70, 84, 92)"><span class="font" style="font-family:"source sans pro", helvetica, arial, sans-serif"><span class="size" style="font-size:18px"> value is reset, even if is value is unchanged (e.g. "scontrol update PartitionName=name AllowGroups=group"). For a user's access to a partition to change, both his group membership must change and Slurm's internal user ID list must change using one of the methods described above.</span></span></span></span><br></div>
<div><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(70, 84, 92)"><span class="font" style="font-family:"source sans pro", helvetica, arial, sans-serif"><span class="size" style="font-size:18px"></span></span></span></span><br></div>
<div><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(70, 84, 92)"><span class="font" style="font-family:"source sans pro", helvetica, arial, sans-serif"><span class="size" style="font-size:18px">Are you adding groups after submission too? Does changing allow groups on the partition fix it too? </span></span></span></span><br></div>
<div><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(70, 84, 92)"><span class="font" style="font-family:"source sans pro", helvetica, arial, sans-serif"><span class="size" style="font-size:18px"></span></span></span></span><br></div>
<div><span class="highlight" style="background-color:rgb(255, 255, 255)"><span class="colour" style="color:rgb(70, 84, 92)"><span class="font" style="font-family:"source sans pro", helvetica, arial, sans-serif"><span class="size" style="font-size:18px">Antony</span></span></span></span><br></div>
</div>
<div style="font-size:16px;"><br></div>
<div defang_data-gmailquote="yes"><div dir="ltr">On Tue, 13 Nov 2018, 09:13 Joerg Sassmannshausen <<a href="mailto:joerg.sassmannshausen@crick.ac.uk">joerg.sassmannshausen@crick.ac.uk</a> wrote:<br></div>
<blockquote defang_data-gmailquote="yes" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;"><div style="font-size:16px;">Dear all,<br></div>
<div style="font-size:16px;"> <br></div>
<div style="font-size:16px;"> I am wondering if that is the same issue we are having here as well.<br></div>
<div style="font-size:16px;"> When I am adding users in the secondary group some time *after* the<br></div>
<div style="font-size:16px;"> initial user installation, the user cannot access the slurm partition it<br></div>
<div style="font-size:16px;"> suppose to. We found two remedies here, more or less by chance:<br></div>
<div style="font-size:16px;"> - rebooting both the slurm server and slurm DB server<br></div>
<div style="font-size:16px;"> - be patient and wait for long enough<br></div>
<div style="font-size:16px;"> <br></div>
<div style="font-size:16px;"> Obviously, both remedies are not suitable if you are running a large<br></div>
<div style="font-size:16px;"> research environment. The reboot was happening as we physically had to<br></div>
<div style="font-size:16px;"> move the servers and the waiting for long enough was simply as we did<br></div>
<div style="font-size:16px;"> not have an answer to the question.<br></div>
<div style="font-size:16px;"> As already mentioned in a different posting, we have deleted the user in<br></div>
<div style="font-size:16px;"> slurm and re-installed it, updated the sssd on the slurm server, all in<br></div>
<div style="font-size:16px;"> vain.<br></div>
<div style="font-size:16px;"> <br></div>
<div style="font-size:16px;"> However, reading the threat, the latter case points to a caching<br></div>
<div style="font-size:16px;"> problem, similar to the one described here. We are also using FreeIPA<br></div>
<div style="font-size:16px;"> and hence sssd for the ID lookup.<br></div>
<div style="font-size:16px;"> <br></div>
<div style="font-size:16px;"> Poking the list a bit further on this subject: does anybody have similar<br></div>
<div style="font-size:16px;"> experiences when the lookup is done directly on AD? We are planning to<br></div>
<div style="font-size:16px;"> move to AD and if that is also an issue at least are warned here.<br></div>
<div style="font-size:16px;"> <br></div>
<div style="font-size:16px;"> All the best<br></div>
<div style="font-size:16px;"> <br></div>
<div style="font-size:16px;"> Jörg<br></div>
<div style="font-size:16px;"> <br></div>
<div style="font-size:16px;"> On 10/11/18 11:17, Douglas Jacobsen wrote:<br></div>
<div style="font-size:16px;"> > We've had issues getting sssd to work reliably on compute nodes (at<br></div>
<div style="font-size:16px;"> > least at scale), the reason is not fully understood, but basically if<br></div>
<div style="font-size:16px;"> > the connection times out with sssd it'll black list the server for 60s,<br></div>
<div style="font-size:16px;"> > which then causes those kinds of issues.<br></div>
<div style="font-size:16px;"> ><br></div>
<div style="font-size:16px;"> > Setting LaunchParameters=send_gids will sidestep this issue by doing the<br></div>
<div style="font-size:16px;"> > lookups exclusively on the controller node, where more frequent<br></div>
<div style="font-size:16px;"> > connections can prevent time decay disconnections and reduce the<br></div>
<div style="font-size:16px;"> > likelihood of cache misses.<br></div>
<div style="font-size:16px;"> ><br></div>
<div style="font-size:16px;"> > On Fri, Nov 9, 2018 at 11:16 PM Chris Samuel <<a href="mailto:chris@csamuel.org">chris@csamuel.org</a><br></div>
<div style="font-size:16px;"> > <mailto:<a href="mailto:chris@csamuel.org">chris@csamuel.org</a>>> wrote:<br></div>
<div style="font-size:16px;"> ><br></div>
<div style="font-size:16px;"> > On Friday, 9 November 2018 2:47:51 AM AEDT Aravindh Sampathkumar wrote:<br></div>
<div style="font-size:16px;"> ><br></div>
<div style="font-size:16px;"> > > navtp@console2:~> ssh c07b07 id<br></div>
<div style="font-size:16px;"> > > uid=29865(navtp) gid=510(finland)<br></div>
<div style="font-size:16px;"> > groups=510(finland),508(nav),5001(ghpc)<br></div>
<div style="font-size:16px;"> > > context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023<br></div>
<div style="font-size:16px;"> ><br></div>
<div style="font-size:16px;"> > Do you have SElinux configured by some chance?<br></div>
<div style="font-size:16px;"> ><br></div>
<div style="font-size:16px;"> > If so you might want to check if it works with it disabled first..<br></div>
<div style="font-size:16px;"> ><br></div>
<div style="font-size:16px;"> > All the best,<br></div>
<div style="font-size:16px;"> > Chris<br></div>
<div style="font-size:16px;"> > --<br></div>
<div style="font-size:16px;"> > Chris Samuel : <a href="http://www.csamuel.org/">http://www.csamuel.org/</a><br></div>
<div style="font-size:16px;"> > <<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&data=02%7C01%7C%7Cbf873add236a4bc74b0a08d646ff523c%7C4eed7807ebad415aa7a99170947f4eae%7C0%7C0%7C636774459751813515&sdata=L5%2Fg8HVibwr3xnv4%2FzlnwMBj8HgMlytUYposfbGi%2Bq8%3D&reserved=0">https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&data=02%7C01%7C%7Cbf873add236a4bc74b0a08d646ff523c%7C4eed7807ebad415aa7a99170947f4eae%7C0%7C0%7C636774459751813515&sdata=L5%2Fg8HVibwr3xnv4%2FzlnwMBj8HgMlytUYposfbGi%2Bq8%3D&reserved=0</a>><br></div>
<div style="font-size:16px;"> > : Melbourne, VIC<br></div>
<div style="font-size:16px;"> ><br></div>
<div style="font-size:16px;"> ><br></div>
<div style="font-size:16px;"> ><br></div>
<div style="font-size:16px;"> ><br></div>
<div style="font-size:16px;"> > --<br></div>
<div style="font-size:16px;"> > Sent from Gmail Mobile<br></div>
<div style="font-size:16px;"> <br></div>
<div style="font-size:16px;"> --<br></div>
<div style="font-size:16px;"> Dr. Jörg Saßmannshausen, MRSC<br></div>
<div style="font-size:16px;"> HPC & Research Data System Engineer<br></div>
<div style="font-size:16px;"> Scientific Computing<br></div>
<div style="font-size:16px;"> The Francis Crick Institute<br></div>
<div style="font-size:16px;"> 1 Midland Way<br></div>
<div style="font-size:16px;"> London, NW1 1AT<br></div>
<div style="font-size:16px;"> email: <a href="mailto:joerg.sassmannshausen@crick.ac.uk">joerg.sassmannshausen@crick.ac.uk</a><br></div>
<div style="font-size:16px;"> phone: 020 379 65139<br></div>
<div style="font-size:16px;"> The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 1 Midland Road London NW1 1AT<br></div>
</blockquote></div>
</blockquote><div style="font-size:16px;"><br></div>
</body>
</html>