<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:PMingLiU;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@PMingLiU";
panose-1:2 1 6 1 0 1 1 1 1 1;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Recently encountered an odd issue where some users were getting sporadic permission denied on certain directories with their stderr/stdout. We realized that this was caused by a change in their nested group
permissions on AD several days ago.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">At first we thought it was the compute nodes themselves so we cleared sssd, restarted slurmd and even restarted the node completely. This did not resolve the issue. User was able to ssh directly onto the nodes
and access the directories, this issue only manifest itself when the jobs were going through slurm.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We later read on slurm.conf:<o:p></o:p></span></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt">disable_send_gids<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt">By default the slurmctld will lookup and send the user_name and extended gids for a job, rather than individual on each node as part of each task launch. Which avoids issues around name service scalability
when launching jobs involving many nodes. Using this option will reverse this functionality.<o:p></o:p></span></i></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We checked sssd and getent on the slurmctld for the users and they were resolving correctly. The fix was to clear sssd and restart slurmctld.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m wondering if the slurmctld does some kind of caching with the extended gids and if there were a better way of handling this?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#009051;background:white">Luis Huang | </span></b><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#009051;background:white">Systems Administrator
II, Research Computing</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#444444;background:white"><br>
</span><b><span style="font-size:8.5pt;font-family:"Arial",sans-serif;color:#7F7F7F;background:white">New York Genome Center</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#444444;background:white"><br>
</span><span style="font-size:8.5pt;font-family:"Arial",sans-serif;color:#535353;background:white">101 Avenue of the Americas</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#444444;background:white"><br>
</span><span style="font-size:8.5pt;font-family:"Arial",sans-serif;color:#535353;background:white">New York, NY 10013</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#444444;background:white"> <br>
</span><span style="font-size:8.5pt;font-family:"Arial",sans-serif;color:#535353;background:white">O: (646) 977-7291</span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#444444;background:white"><br>
</span><span style="font-size:8.5pt;font-family:"Arial",sans-serif;color:#535353;background:white"><a href="mailto:lhuang@nygenome.org" target="_blank"><span style="color:#0072C6">lhuang@nygenome.org</span></a></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<hr>
<div style="font-size:7.5pt; font-family: arial; font-style:normal; font-weight:normal; ">
This message is for the recipient’s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the
sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email.</div>
</body>
</html>