<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi Sean,<div><br></div><div>Thank you! It was a permissions issue and it’s not complaining anymore about cred/munge.</div><div><br></div><div>I appreciate your help.</div><div><br></div><div>Thanks,</div><div><br></div><div>Jesse</div><div><div><br><blockquote type="cite"><div>On Jan 23, 2024, at 3:34 PM, Sean Crosby <scrosby@unimelb.edu.au> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><span style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"></span><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span><div style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif;">slurmctld runs as the user slurm, whereas slurmd runs as root.</span></div><div style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif;"><br></span></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Make sure the permissions on /app/slurm-24.0.8/lib/slurm allow the user slurm to read the files</span></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></span></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">e.g. you could do (as root)</span></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></span></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">sudo -u slurm ls /app/slurm-24.0.8/lib/slurm</span></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></span></div><div class="elementToProof" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">and see if the slurm user can read the directory (as well as the libraries within it)<br></span></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></span></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Sean</span></div><span style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"></span><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"></div><hr tabindex="-1" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 1065.25px;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Jesse Aiton <<a href="mailto:jesse@clarkeconsulting.com">jesse@clarkeconsulting.com</a>><br><b>Sent:</b><span class="Apple-converted-space"> </span>Wednesday, 24 January 2024 10:14<br><b>To:</b><span class="Apple-converted-space"> </span><a href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a><span class="Apple-converted-space"> </span><<a href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a>><br><b>Subject:</b><span class="Apple-converted-space"> </span>[EXT] [slurm-users] error: Couldn't find the specified plugin name for cred/munge looking at all files</font><div> </div></div><div class="BodyFragment" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><font size="2"><span style="font-size: 11pt;"><div class="PlainText">External email: Please exercise caution<br><br>Hello Slurm Folks,<br><br>I have a weird issue where on the same server, which acts as both a controller and a node, slurmctld can’t find cred_munge.so<br><br>slurmctld: debug3: Trying to load plugin /app/slurm-24.0.8/lib/slurm/cred_munge.so<br>slurmctld: debug4: /app/slurm-24.0.8/lib/slurm/cred_munge.so: Does not exist or not a regular file.<br>slurmctld: error: Couldn't find the specified plugin name for cred/munge looking at all files<br>slurmctld: error: cannot open plugin directory /app/slurm-24.0.8/lib/slurm<br>slurmctld: error: cannot find cred plugin for cred/munge<br>slurmctld: error: cannot create cred context for cred/munge<br>slurmctld: fatal: failed to initialize cred plugin<br><br>But slurmd can:<br><br>slurmd: debug3: Trying to load plugin /app/slurm-24.0.8/lib/slurm/cred_munge.so<br>slurmd: debug3: plugin_load_from_file->_verify_syms: found Slurm plugin name:Munge credential signature plugin type:cred/munge version:0x180800<br>slurmd: cred/munge: init: Munge credential signature plugin loaded<br>slurmd: debug3: Success.<br><br>This is on Ubuntu 20.04 and happens both with Slurm 20.11.09 and 24.0.8<span class="Apple-converted-space"> </span><br><br>Thank you,<br><br>Jesse<br><br><br># slurm.conf file generated by configurator easy.html.<br># Put this file on all nodes of your cluster.<br># See the slurm.conf man page for more information.<br>#<br>ClusterName=prod-cluster<br>SlurmctldHost=controller<br>#<br>#MailProg=/bin/mail<br>#MpiDefault=<br>#MpiParams=ports=#-#<br>ProctrackType=proctrack/cgroup<br>ReturnToService=1<br>SlurmctldPidFile=/var/run/slurmctld.pid<br>#SlurmctldPort=6817<br>SlurmdPidFile=/var/run/slurmd.pid<br>#SlurmdPort=6818<br>SlurmdSpoolDir=/var/spool/slurmd<br>SlurmUser=slurm<br>#SlurmdUser=root<br>StateSaveLocation=/var/spool/slurmctld<br>#SwitchType=<br>TaskPlugin=task/affinity,task/cgroup<br>#<br>#<br># TIMERS<br>#KillWait=30<br>#MinJobAge=300<br>#SlurmctldTimeout=120<br>#SlurmdTimeout=300<br>#<br>#<br># SCHEDULING<br>SchedulerType=sched/backfill<br>SelectType=select/cons_tres<br>#<br>#<br># LOGGING AND ACCOUNTING<br>#AccountingStorageType=<br>#JobAcctGatherFrequency=30<br>#JobAcctGatherType=<br>#SlurmctldDebug=info<br>SlurmctldLogFile=/var/log/slurmctld.log<br>#SlurmdDebug=info<br>SlurmdLogFile=/var/log/slurmd.log<br>#<br>#<br># COMPUTE NODES<br>NodeName=controller CPUs=1 State=UNKNOWN<br>NodeName=node CPUs=1 State=UNKNOWN<br>PartitionName=prod-part Nodes=ALL Default=YES MaxTime=INFINITE State=UP</div></span></font></div></div></blockquote></div><br></div></body></html>