[slurm-users] Slurmrestd authentication failed: Unspecified error

[Chenyang Yan]

Hello,

I have met a similar issue with slurmrestd authentication failed error,
similar question:
https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html

I have installed `slurm 21.08.6` on CentOS 7.9.2009 container, basic
service is running fine
```
[root at slurmctl supervisor]# ls -l /.dockerenv
-rwxr-xr-x. 1 root root 0 Mar 17 23:17 /.dockerenv
[root at slurmctl supervisor]# srun --partition normal hostname
slurmctl
```

Slurmrestd is compiled with `--enable-slurmrestd` successfully, slurmrestd
JWT configuration is as follows:
```
[root at slurmctl slurmctld]# dd if=/dev/random
of=/var/spool/slurm/jwt_hs256.key bs=32 count=1
[root at slurmctl slurmctld]# scontrol show config |grep -i auth
AuthAltTypes            = auth/jwt
AuthAltParameters       = jwt_key=/var/spool/slurm/jwt_hs256.key
AuthInfo                = (null)
AuthType                = auth/munge
[root at slurmctl slurmctld]# ll -l /var/spool/slurm/jwt_hs256.key
-rw-r--r--. 1 root root 32 Mar 17 23:21 /var/spool/slurm/jwt_hs256.key

# start slurmrestd process
[root at slurmctl slurmctld]# SLURMRESTD_SECURITY=disable_user_check
/usr/sbin/slurmrestd -vvvv 0.0.0.0:19090

# check process and environ
[root at slurmctl slurmctld]# ps -ef |grep slurmrestd
root      1235   236  0 23:26 ?        00:00:00 /usr/sbin/slurmrestd -vvvvv
0.0.0.0:19090

[root at slurmctl slurmctld]# cd /proc/1235/
[root at slurmctl 1235]# cat environ | tr '\0' "\n"
TERM=xterm
TINI_VERSION=v0.18.0
SHLVL=1
HOSTNAME=slurmctl
SUPERVISOR_ENABLED=1
SUPERVISOR_PROCESS_NAME=slurmrestd
PWD=/
SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock
SUPERVISOR_GROUP_NAME=slurmrestd
PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin
HOME=/root
SLURMRESTD_SECURITY=disable_user_check
_=/usr/bin/supervisord
```

I have generated correct token to request, but slurmrestd log message
reported authentication failed: Unspecified error
```
[root at slurmctl 1235]# scontrol token username=slurm
SLURM_JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw
[root at slurmctl 1235]#
token="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw"
[root at slurmctl 1235]# curl 172.17.0.4:19090/openapi -H "X-SLURM-USER-TOKEN:
$token" -H "X-SLURM-USER-NAME: slurm"
Authentication failure


slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate:
[[172.17.0.1]:38090] attempting user_name slurm token authentication pass
through
slurmrestd: error: operations_router: [[172.17.0.1]:38090] authentication
failed: Unspecified error
slurmrestd: debug2: _on_message_complete_request: [[172.17.0.1]:38090]
on_http_request rejected: Unspecified error
```

But! I have found that I'm setting `SLURM_JWT` environment variable for
process, whatever token value is authenticated normally
```
# start process with SLURM_JWT
[root at slurmctl 1235]# cd /proc/2108/
[root at slurmctl 2108]# cat environ |tr "\0" "\n"
TERM=xterm
TINI_VERSION=v0.18.0
SHLVL=1
HOSTNAME=slurmctl
SLURM_JWT=randomtoken
SUPERVISOR_ENABLED=1
SUPERVISOR_PROCESS_NAME=slurmrestd
PWD=/
SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock
SUPERVISOR_GROUP_NAME=slurmrestd
PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin
HOME=/root
SLURMRESTD_SECURITY=disable_user_check
_=/usr/bin/supervisord

# request OK
[root at slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H
"X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm"
[
 ]
[root at slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H
"X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME:
errorvalue"
[
 ]
```

So, I'm confused about JWT authentication.
Q1: What is used for the `SLURM_JWT` environment variable, is it required
for JWT? Related search from github source repo:
https://github.com/SchedMD/slurm/search?q=SLURM_JWT
Q2: How to use slurmrestd JWT authentication?

--------
Thanks, Chenyang Yan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220319/dd1a44dd/attachment.htm>