<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-AU" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi slurm-users,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hoping you can point me in the right direction….I am trying to setup the Slurm REST API under parallelcluster v3.1.1. I am following the instructions at https://slurm.schedmd.com/jwt.html. I have added the AuthAltTypes=rest_auth/jwt and
the AuthAultParameters=jwt_key=… to /opt/slum/etc/slurm.conf but restarting slurmctld fails with:
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">slurmctld: debug: slurmctld log levels: stderr=debug3 logfile=debug3 syslog=quiet<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug: Log file re-opened<o:p></o:p></p>
<p class="MsoNormal">slurmctld: pidfile not locked, assuming no running daemon<o:p></o:p></p>
<p class="MsoNormal">slurmctld: Stack size set to 10485760<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug: slurmscriptd: Got ack from slurmctld, initialization successful<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug: slurmctld: slurmscriptd fork()'d and initialized.<o:p></o:p></p>
<p class="MsoNormal">slurmctld: slurmctld version 21.08.5 started on cluster parallelcluster<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Trying to load plugin /opt/slurm/lib/slurm/cred_munge.so<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug: _slurmscriptd_mainloop: started<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Called _msg_readable<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug: _slurmctld_listener_thread: started listening to slurmscriptd<o:p></o:p></p>
<p class="MsoNormal">slurmctld: cred/munge: init: Munge credential signature plugin loaded<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Success.<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Called _msg_readable<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Trying to load plugin /opt/slurm/lib/slurm/auth_munge.so<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug: auth/munge: init: Munge authentication plugin loaded<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Success.<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Trying to load plugin /opt/slurm/lib/slurm/rest_auth_jwt.so<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Couldn't find sym 'auth_p_create' in the plugin<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Couldn't find sym 'auth_p_destroy' in the plugin<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Couldn't find sym 'auth_p_verify' in the plugin<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Couldn't find sym 'auth_p_get_uid' in the plugin<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Couldn't find sym 'auth_p_get_gid' in the plugin<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Couldn't find sym 'auth_p_get_host' in the plugin<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Couldn't find sym 'auth_p_pack' in the plugin<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Couldn't find sym 'auth_p_unpack' in the plugin<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Couldn't find sym 'auth_p_thread_config' in the plugin<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Couldn't find sym 'auth_p_thread_clear' in the plugin<o:p></o:p></p>
<p class="MsoNormal">slurmctld: debug3: Couldn't find sym 'auth_p_token_generate' in the plugin<o:p></o:p></p>
<p class="MsoNormal">slurmctld: fatal: plugin_load_and_link: Plugin loading failed due to missing symbols. Plugin is corrupted.<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Looks like I am missing something when trying to enable JWT authentication with the slurm services. I am using the slurm install that comes out of the box with parallelcluster 3.1.1. I was wondering if there are any suggestions about how
to fix/resolve this error?. Thanks. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Slurmrestd auth types:<o:p></o:p></p>
<p class="MsoNormal">$ slurmrestd -a list<o:p></o:p></p>
<p class="MsoNormal">slurmrestd: Possible REST authentication plugins:<o:p></o:p></p>
<p class="MsoNormal">slurmrestd: rest_auth/local<o:p></o:p></p>
<p class="MsoNormal">slurmrestd: rest_auth/jwt<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Also, step 2 in the JWT instructions at <a href="https://slurm.schedmd.com/jwt.html">
https://slurm.schedmd.com/jwt.html</a> is a little confusing – the key is saved to the StateSaveLocation but the rest of the instructions reference /etc/slurm? Am I missing something here (likely!) ?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">dd if=/dev/random of=/var/spool/slurm/statesave/jwt_hs256.key bs=32 count=1<o:p></o:p></p>
<p class="MsoNormal">chown slurm:slurm /etc/slurm/jwt_hs256.key<o:p></o:p></p>
<p class="MsoNormal">chmod 0600 /etc/slurm/jwt_hs256.key<o:p></o:p></p>
<p class="MsoNormal">chown root:root /etc/slurm<o:p></o:p></p>
<p class="MsoNormal">chmod 0755 /etc/slurm<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">j<o:p></o:p></p>
</div>
</body>
</html>