<div dir="ltr">Guillaume, thanks for your reply and your workaround.<br><div><br></div><div>Do you try adding `SLUTM_JWT=daemon` fo slurmrestd process, and then any value is is authenticated successfully for request header `X-SLURM-USER-TOKEN`, `X-SLURM-USER-NAME` .</div><div><br></div><div>You can see the information in my last email:<br></div><div><br></div><div>```</div><div># start process with SLURM_JWT<br>[root@slurmctl 1235]# cd /proc/2108/<br>[root@slurmctl 2108]# cat environ |tr "\0" "\n"<br>TERM=xterm<br>TINI_VERSION=v0.18.0<br>SHLVL=1<br>HOSTNAME=slurmctl<br>SLURM_JWT=randomtoken                   <===== SLURM_JWT environ<br>SUPERVISOR_ENABLED=1<br>SUPERVISOR_PROCESS_NAME=slurmrestd<br>PWD=/<br>SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock<br>SUPERVISOR_GROUP_NAME=slurmrestd<br>PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin<br>HOME=/root<br>SLURMRESTD_SECURITY=disable_user_check<br>_=/usr/bin/supervisord<br><br># request OK<br>[root@slurmctl 2108]# curl <a href="http://172.17.0.4:19090/slurm/v0.0.35/jobs" target="_blank">172.17.0.4:19090/slurm/v0.0.35/jobs</a> -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm"<br>[<br> ]<br>[root@slurmctl 2108]# curl <a href="http://172.17.0.4:19090/slurm/v0.0.35/jobs" target="_blank">172.17.0.4:19090/slurm/v0.0.35/jobs</a> -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: errorvalue"<br>[<br> ]<br></div><div>```</div><div><br></div><div>So, I'm confused for slurmrestd JWT authentication.</div><div><br></div><div>--------------------</div><div>Thanks, Chenyang Yan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 21, 2022 at 4:10 PM Guillaume COCHARD <<a href="mailto:guillaume.cochard@cc.in2p3.fr">guillaume.cochard@cc.in2p3.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div>Hello,<br></div><div><br></div><div>We had the same error and we fixed it by adding `Environment="SLURM_JWT=daemon"` to the [Service] section of the unit file (in our case /usr/lib/systemd/system/slurmrestd.service ).<br></div><div><br></div><div> We have a bug (feature?) that makes us unable to use root or slurm user as user for slurmrestd service so maybe you'll encounter that as well. According to the release notes ( <a href="https://github.com/SchedMD/slurm/blob/master/NEWS" target="_blank">https://github.com/SchedMD/slurm/blob/master/NEWS</a> ) this should be starting on version 22.05.0pre1 but it is already the case in our version 21.08.6. We have an administrator user for this can of usage.<br></div><div><br></div><div>Our service file is like this:<br></div><div><br></div><div>$ cat /usr/lib/systemd/system/slurmrestd.service <br>[Unit]<br>Description=Slurm REST daemon<br>After=network.target munge.service<br>ConditionPathExists=/etc/slurm/slurm.conf<br><br>[Service]<br>User=slurmadm<br>Type=simple<br>EnvironmentFile=-/etc/sysconfig/slurmrestd<br>Environment="SLURM_JWT=daemon"<br>ExecStart=/usr/sbin/slurmrestd -v -a rest_auth/jwt localhost:6820<br>ExecReload=/bin/kill -HUP $MAINPID<br><br>[Install]<br>WantedBy=multi-user.target<br></div><div><br></div><div><br></div><div>Guillaume<br></div><div><br></div><div><br></div><div><br></div><hr id="gmail-m_2433880600494962054zwchr"><div><b>De: </b>"Chenyang Yan" <<a href="mailto:memory.yancy@gmail.com" target="_blank">memory.yancy@gmail.com</a>><br><b>À: </b><a href="mailto:slurm-users@schedmd.com" target="_blank">slurm-users@schedmd.com</a><br><b>Envoyé: </b>Samedi 19 Mars 2022 14:09:06<br><b>Objet: </b>[slurm-users] Slurmrestd authentication failed: Unspecified error<br></div><br><div><div dir="ltr">Hello,<br><div>I have met a similar issue with slurmrestd authentication failed error, similar question: <a href="https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html" rel="nofollow noopener noreferrer" target="_blank">https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html</a><div><br>I have installed `slurm 21.08.6` on CentOS 7.9.2009 container, basic service is running fine<br>```<br>[root@slurmctl supervisor]# ls -l /.dockerenv<br>-rwxr-xr-x. 1 root root 0 Mar 17 23:17 /.dockerenv<br>[root@slurmctl supervisor]# srun --partition normal hostname<br>slurmctl<br>```<br><br>Slurmrestd is compiled with `--enable-slurmrestd` successfully, slurmrestd JWT configuration is as follows:<br>```<br>[root@slurmctl slurmctld]# dd if=/dev/random of=/var/spool/slurm/jwt_hs256.key bs=32 count=1<br>[root@slurmctl slurmctld]# scontrol show config |grep -i auth<br>AuthAltTypes            = auth/jwt<br>AuthAltParameters       = jwt_key=/var/spool/slurm/jwt_hs256.key<br>AuthInfo                = (null)<br>AuthType                = auth/munge<br>[root@slurmctl slurmctld]# ll -l /var/spool/slurm/jwt_hs256.key<br>-rw-r--r--. 1 root root 32 Mar 17 23:21 /var/spool/slurm/jwt_hs256.key<br><br># start slurmrestd process<br>[root@slurmctl slurmctld]# SLURMRESTD_SECURITY=disable_user_check /usr/sbin/slurmrestd -vvvv <a href="http://0.0.0.0:19090" rel="nofollow noopener noreferrer" target="_blank">0.0.0.0:19090</a><br><br># check process and environ<br>[root@slurmctl slurmctld]# ps -ef |grep slurmrestd<br>root      1235   236  0 23:26 ?        00:00:00 /usr/sbin/slurmrestd -vvvvv <a href="http://0.0.0.0:19090" rel="nofollow noopener noreferrer" target="_blank">0.0.0.0:19090</a><br></div><div><br>[root@slurmctl slurmctld]# cd /proc/1235/<br>[root@slurmctl 1235]# cat environ | tr '\0' "\n"<br>TERM=xterm<br>TINI_VERSION=v0.18.0<br>SHLVL=1<br>HOSTNAME=slurmctl<br>SUPERVISOR_ENABLED=1<br>SUPERVISOR_PROCESS_NAME=slurmrestd<br>PWD=/<br>SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock<br>SUPERVISOR_GROUP_NAME=slurmrestd<br>PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin<br>HOME=/root<br>SLURMRESTD_SECURITY=disable_user_check<br>_=/usr/bin/supervisord<br>```<br><br>I have generated correct token to request, but slurmrestd log message reported authentication failed: Unspecified error<br>```<br>[root@slurmctl 1235]# scontrol token username=slurm<br>SLURM_JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw<br>[root@slurmctl 1235]# token="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw"<br>[root@slurmctl 1235]# curl <a href="http://172.17.0.4:19090/openapi" rel="nofollow noopener noreferrer" target="_blank">172.17.0.4:19090/openapi</a> -H "X-SLURM-USER-TOKEN: $token" -H "X-SLURM-USER-NAME: slurm"<br>Authentication failure<br><br><br>slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[172.17.0.1]:38090] attempting user_name slurm token authentication pass through<br>slurmrestd: error: operations_router: [[172.17.0.1]:38090] authentication failed: Unspecified error<br>slurmrestd: debug2: _on_message_complete_request: [[172.17.0.1]:38090] on_http_request rejected: Unspecified error<br>```<br><br>But! I have found that I'm setting `SLURM_JWT` environment variable for process, whatever token value is authenticated normally<br>```<br># start process with SLURM_JWT<br>[root@slurmctl 1235]# cd /proc/2108/<br>[root@slurmctl 2108]# cat environ |tr "\0" "\n"<br>TERM=xterm<br>TINI_VERSION=v0.18.0<br>SHLVL=1<br>HOSTNAME=slurmctl<br>SLURM_JWT=randomtoken<br>SUPERVISOR_ENABLED=1<br>SUPERVISOR_PROCESS_NAME=slurmrestd<br>PWD=/<br>SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock<br>SUPERVISOR_GROUP_NAME=slurmrestd<br>PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin<br>HOME=/root<br>SLURMRESTD_SECURITY=disable_user_check<br>_=/usr/bin/supervisord<br><br># request OK<br>[root@slurmctl 2108]# curl <a href="http://172.17.0.4:19090/slurm/v0.0.35/jobs" rel="nofollow noopener noreferrer" target="_blank">172.17.0.4:19090/slurm/v0.0.35/jobs</a> -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm"<br>[<br> ]<br>[root@slurmctl 2108]# curl <a href="http://172.17.0.4:19090/slurm/v0.0.35/jobs" rel="nofollow noopener noreferrer" target="_blank">172.17.0.4:19090/slurm/v0.0.35/jobs</a> -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: errorvalue"<br>[<br> ]<br>```<br><br>So, I'm confused about JWT authentication.<br>Q1: What is used for the `SLURM_JWT` environment variable, is it required for JWT? Related search from github source repo: <a href="https://github.com/SchedMD/slurm/search?q=SLURM_JWT" rel="nofollow noopener noreferrer" target="_blank">https://github.com/SchedMD/slurm/search?q=SLURM_JWT</a><br></div><div>Q2: How to use slurmrestd JWT authentication?<br><br>--------<br>Thanks, Chenyang Yan</div></div></div><br></div></div></div></blockquote></div>