<div dir="ltr">Hello,<div><br></div><div>I have met a similar issue with slurmrestd authentication failed error, similar question: <a href="https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html">https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html</a><div><br>I have installed `slurm 21.08.6` on CentOS 7.9.2009 container, basic service is running fine<br>```<br>[root@slurmctl supervisor]# ls -l /.dockerenv<br>-rwxr-xr-x. 1 root root 0 Mar 17 23:17 /.dockerenv<br>[root@slurmctl supervisor]# srun --partition normal hostname<br>slurmctl<br>```<br><br>Slurmrestd is compiled with `--enable-slurmrestd` successfully, slurmrestd JWT configuration is as follows:<br>```<br>[root@slurmctl slurmctld]# dd if=/dev/random of=/var/spool/slurm/jwt_hs256.key bs=32 count=1<br>[root@slurmctl slurmctld]# scontrol show config |grep -i auth<br>AuthAltTypes            = auth/jwt<br>AuthAltParameters       = jwt_key=/var/spool/slurm/jwt_hs256.key<br>AuthInfo                = (null)<br>AuthType                = auth/munge<br>[root@slurmctl slurmctld]# ll -l /var/spool/slurm/jwt_hs256.key<br>-rw-r--r--. 1 root root 32 Mar 17 23:21 /var/spool/slurm/jwt_hs256.key<br><br># start slurmrestd process<br>[root@slurmctl slurmctld]# SLURMRESTD_SECURITY=disable_user_check /usr/sbin/slurmrestd -vvvv <a href="http://0.0.0.0:19090">0.0.0.0:19090</a><br><br># check process and environ<br>[root@slurmctl slurmctld]# ps -ef |grep slurmrestd<br>root      1235   236  0 23:26 ?        00:00:00 /usr/sbin/slurmrestd -vvvvv <a href="http://0.0.0.0:19090">0.0.0.0:19090</a></div><div><br>[root@slurmctl slurmctld]# cd /proc/1235/<br>[root@slurmctl 1235]# cat environ | tr '\0' "\n"<br>TERM=xterm<br>TINI_VERSION=v0.18.0<br>SHLVL=1<br>HOSTNAME=slurmctl<br>SUPERVISOR_ENABLED=1<br>SUPERVISOR_PROCESS_NAME=slurmrestd<br>PWD=/<br>SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock<br>SUPERVISOR_GROUP_NAME=slurmrestd<br>PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin<br>HOME=/root<br>SLURMRESTD_SECURITY=disable_user_check<br>_=/usr/bin/supervisord<br>```<br><br>I have generated correct token to request, but slurmrestd log message reported authentication failed: Unspecified error<br>```<br>[root@slurmctl 1235]# scontrol token username=slurm<br>SLURM_JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw<br>[root@slurmctl 1235]# token="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw"<br>[root@slurmctl 1235]# curl <a href="http://172.17.0.4:19090/openapi">172.17.0.4:19090/openapi</a> -H "X-SLURM-USER-TOKEN: $token" -H "X-SLURM-USER-NAME: slurm"<br>Authentication failure<br><br><br>slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[172.17.0.1]:38090] attempting user_name slurm token authentication pass through<br>slurmrestd: error: operations_router: [[172.17.0.1]:38090] authentication failed: Unspecified error<br>slurmrestd: debug2: _on_message_complete_request: [[172.17.0.1]:38090] on_http_request rejected: Unspecified error<br>```<br><br>But! I have found that I'm setting `SLURM_JWT` environment variable for process, whatever token value is authenticated normally<br>```<br># start process with SLURM_JWT<br>[root@slurmctl 1235]# cd /proc/2108/<br>[root@slurmctl 2108]# cat environ |tr "\0" "\n"<br>TERM=xterm<br>TINI_VERSION=v0.18.0<br>SHLVL=1<br>HOSTNAME=slurmctl<br>SLURM_JWT=randomtoken<br>SUPERVISOR_ENABLED=1<br>SUPERVISOR_PROCESS_NAME=slurmrestd<br>PWD=/<br>SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock<br>SUPERVISOR_GROUP_NAME=slurmrestd<br>PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin<br>HOME=/root<br>SLURMRESTD_SECURITY=disable_user_check<br>_=/usr/bin/supervisord<br><br># request OK<br>[root@slurmctl 2108]# curl <a href="http://172.17.0.4:19090/slurm/v0.0.35/jobs">172.17.0.4:19090/slurm/v0.0.35/jobs</a> -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm"<br>[<br> ]<br>[root@slurmctl 2108]# curl <a href="http://172.17.0.4:19090/slurm/v0.0.35/jobs">172.17.0.4:19090/slurm/v0.0.35/jobs</a> -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: errorvalue"<br>[<br> ]<br>```<br><br>So, I'm confused about JWT authentication.<br>Q1: What is used for the `SLURM_JWT` environment variable, is it required for JWT? Related search from github source repo: <a href="https://github.com/SchedMD/slurm/search?q=SLURM_JWT">https://github.com/SchedMD/slurm/search?q=SLURM_JWT</a></div><div>Q2: How to use slurmrestd JWT authentication?<br><br>--------<br>Thanks, Chenyang Yan</div></div></div>