<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Hello,<br></div><div><br data-mce-bogus="1"></div><div>We had the same error and we fixed it by adding `Environment="SLURM_JWT=daemon"` to the [Service] section of the unit file (in our case /usr/lib/systemd/system/slurmrestd.service ).<br></div><div><br data-mce-bogus="1"></div><div> We have a bug (feature?) that makes us unable to use root or slurm user as user for slurmrestd service so maybe you'll encounter that as well. According to the release notes ( https://github.com/SchedMD/slurm/blob/master/NEWS ) this should be starting on version 22.05.0pre1 but it is already the case in our version 21.08.6. We have an administrator user for this can of usage.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Our service file is like this:<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>$ cat /usr/lib/systemd/system/slurmrestd.service <br>[Unit]<br>Description=Slurm REST daemon<br>After=network.target munge.service<br>ConditionPathExists=/etc/slurm/slurm.conf<br><br>[Service]<br>User=slurmadm<br>Type=simple<br>EnvironmentFile=-/etc/sysconfig/slurmrestd<br>Environment="SLURM_JWT=daemon"<br>ExecStart=/usr/sbin/slurmrestd -v -a rest_auth/jwt localhost:6820<br>ExecReload=/bin/kill -HUP $MAINPID<br><br>[Install]<br>WantedBy=multi-user.target<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Guillaume<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>De: </b>"Chenyang Yan" <memory.yancy@gmail.com><br><b>À: </b>slurm-users@schedmd.com<br><b>Envoyé: </b>Samedi 19 Mars 2022 14:09:06<br><b>Objet: </b>[slurm-users] Slurmrestd authentication failed: Unspecified error<br></div><br><div data-marker="__QUOTED_TEXT__"><div dir="ltr">Hello,<br><div>I have met a similar issue with slurmrestd authentication failed error, similar question: <a href="https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html" target="_blank" rel="nofollow noopener noreferrer">https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html</a><div><br>I have installed `slurm 21.08.6` on CentOS 7.9.2009 container, basic service is running fine<br>```<br>[root@slurmctl supervisor]# ls -l /.dockerenv<br>-rwxr-xr-x. 1 root root 0 Mar 17 23:17 /.dockerenv<br>[root@slurmctl supervisor]# srun --partition normal hostname<br>slurmctl<br>```<br><br>Slurmrestd is compiled with `--enable-slurmrestd` successfully, slurmrestd JWT configuration is as follows:<br>```<br>[root@slurmctl slurmctld]# dd if=/dev/random of=/var/spool/slurm/jwt_hs256.key bs=32 count=1<br>[root@slurmctl slurmctld]# scontrol show config |grep -i auth<br>AuthAltTypes = auth/jwt<br>AuthAltParameters = jwt_key=/var/spool/slurm/jwt_hs256.key<br>AuthInfo = (null)<br>AuthType = auth/munge<br>[root@slurmctl slurmctld]# ll -l /var/spool/slurm/jwt_hs256.key<br>-rw-r--r--. 1 root root 32 Mar 17 23:21 /var/spool/slurm/jwt_hs256.key<br><br># start slurmrestd process<br>[root@slurmctl slurmctld]# SLURMRESTD_SECURITY=disable_user_check /usr/sbin/slurmrestd -vvvv <a href="http://0.0.0.0:19090" target="_blank" rel="nofollow noopener noreferrer">0.0.0.0:19090</a><br><br># check process and environ<br>[root@slurmctl slurmctld]# ps -ef |grep slurmrestd<br>root 1235 236 0 23:26 ? 00:00:00 /usr/sbin/slurmrestd -vvvvv <a href="http://0.0.0.0:19090" target="_blank" rel="nofollow noopener noreferrer">0.0.0.0:19090</a><br data-mce-bogus="1"></div><div><br>[root@slurmctl slurmctld]# cd /proc/1235/<br>[root@slurmctl 1235]# cat environ | tr '\0' "\n"<br>TERM=xterm<br>TINI_VERSION=v0.18.0<br>SHLVL=1<br>HOSTNAME=slurmctl<br>SUPERVISOR_ENABLED=1<br>SUPERVISOR_PROCESS_NAME=slurmrestd<br>PWD=/<br>SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock<br>SUPERVISOR_GROUP_NAME=slurmrestd<br>PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin<br>HOME=/root<br>SLURMRESTD_SECURITY=disable_user_check<br>_=/usr/bin/supervisord<br>```<br><br>I have generated correct token to request, but slurmrestd log message reported authentication failed: Unspecified error<br>```<br>[root@slurmctl 1235]# scontrol token username=slurm<br>SLURM_JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw<br>[root@slurmctl 1235]# token="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw"<br>[root@slurmctl 1235]# curl <a href="http://172.17.0.4:19090/openapi" target="_blank" rel="nofollow noopener noreferrer">172.17.0.4:19090/openapi</a> -H "X-SLURM-USER-TOKEN: $token" -H "X-SLURM-USER-NAME: slurm"<br>Authentication failure<br><br><br>slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[172.17.0.1]:38090] attempting user_name slurm token authentication pass through<br>slurmrestd: error: operations_router: [[172.17.0.1]:38090] authentication failed: Unspecified error<br>slurmrestd: debug2: _on_message_complete_request: [[172.17.0.1]:38090] on_http_request rejected: Unspecified error<br>```<br><br>But! I have found that I'm setting `SLURM_JWT` environment variable for process, whatever token value is authenticated normally<br>```<br># start process with SLURM_JWT<br>[root@slurmctl 1235]# cd /proc/2108/<br>[root@slurmctl 2108]# cat environ |tr "\0" "\n"<br>TERM=xterm<br>TINI_VERSION=v0.18.0<br>SHLVL=1<br>HOSTNAME=slurmctl<br>SLURM_JWT=randomtoken<br>SUPERVISOR_ENABLED=1<br>SUPERVISOR_PROCESS_NAME=slurmrestd<br>PWD=/<br>SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock<br>SUPERVISOR_GROUP_NAME=slurmrestd<br>PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin<br>HOME=/root<br>SLURMRESTD_SECURITY=disable_user_check<br>_=/usr/bin/supervisord<br><br># request OK<br>[root@slurmctl 2108]# curl <a href="http://172.17.0.4:19090/slurm/v0.0.35/jobs" target="_blank" rel="nofollow noopener noreferrer">172.17.0.4:19090/slurm/v0.0.35/jobs</a> -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm"<br>[<br> ]<br>[root@slurmctl 2108]# curl <a href="http://172.17.0.4:19090/slurm/v0.0.35/jobs" target="_blank" rel="nofollow noopener noreferrer">172.17.0.4:19090/slurm/v0.0.35/jobs</a> -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: errorvalue"<br>[<br> ]<br>```<br><br>So, I'm confused about JWT authentication.<br>Q1: What is used for the `SLURM_JWT` environment variable, is it required for JWT? Related search from github source repo: <a href="https://github.com/SchedMD/slurm/search?q=SLURM_JWT" target="_blank" rel="nofollow noopener noreferrer">https://github.com/SchedMD/slurm/search?q=SLURM_JWT</a><br data-mce-bogus="1"></div><div>Q2: How to use slurmrestd JWT authentication?<br><br>--------<br>Thanks, Chenyang Yan</div></div></div><br></div></div></body></html>