[slurm-users] Slurmrestd authentication failed: Unspecified error

Mon Mar 21 08:07:35 UTC 2022

Hello, 

We had the same error and we fixed it by adding `Environment="SLURM_JWT=daemon"` to the [Service] section of the unit file (in our case /usr/lib/systemd/system/slurmrestd.service ). 

We have a bug (feature?) that makes us unable to use root or slurm user as user for slurmrestd service so maybe you'll encounter that as well. According to the release notes ( https://github.com/SchedMD/slurm/blob/master/NEWS ) this should be starting on version 22.05.0pre1 but it is already the case in our version 21.08.6. We have an administrator user for this can of usage. 

Our service file is like this: 

$ cat /usr/lib/systemd/system/slurmrestd.service 
[Unit] 
Description=Slurm REST daemon 
After=network.target munge.service 
ConditionPathExists=/etc/slurm/slurm.conf 

[Service] 
User=slurmadm 
Type=simple 
EnvironmentFile=-/etc/sysconfig/slurmrestd 
Environment="SLURM_JWT=daemon" 
ExecStart=/usr/sbin/slurmrestd -v -a rest_auth/jwt localhost:6820 
ExecReload=/bin/kill -HUP $MAINPID 

[Install] 
WantedBy=multi-user.target 

Guillaume 

De: "Chenyang Yan" <memory.yancy at gmail.com> 
À: slurm-users at schedmd.com 
Envoyé: Samedi 19 Mars 2022 14:09:06 
Objet: [slurm-users] Slurmrestd authentication failed: Unspecified error 

Hello, 
I have met a similar issue with slurmrestd authentication failed error, similar question: [ https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html | https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html ] 

I have installed `slurm 21.08.6` on CentOS 7.9.2009 container, basic service is running fine 
``` 
[root at slurmctl supervisor]# ls -l /.dockerenv 
-rwxr-xr-x. 1 root root 0 Mar 17 23:17 /.dockerenv 
[root at slurmctl supervisor]# srun --partition normal hostname 
slurmctl 
``` 

Slurmrestd is compiled with `--enable-slurmrestd` successfully, slurmrestd JWT configuration is as follows: 
``` 
[root at slurmctl slurmctld]# dd if=/dev/random of=/var/spool/slurm/jwt_hs256.key bs=32 count=1 
[root at slurmctl slurmctld]# scontrol show config |grep -i auth 
AuthAltTypes = auth/jwt 
AuthAltParameters = jwt_key=/var/spool/slurm/jwt_hs256.key 
AuthInfo = (null) 
AuthType = auth/munge 
[root at slurmctl slurmctld]# ll -l /var/spool/slurm/jwt_hs256.key 
-rw-r--r--. 1 root root 32 Mar 17 23:21 /var/spool/slurm/jwt_hs256.key 

# start slurmrestd process 
[root at slurmctl slurmctld]# SLURMRESTD_SECURITY=disable_user_check /usr/sbin/slurmrestd -vvvv [ http://0.0.0.0:19090/ | 0.0.0.0:19090 ] 

# check process and environ 
[root at slurmctl slurmctld]# ps -ef |grep slurmrestd 
root 1235 236 0 23:26 ? 00:00:00 /usr/sbin/slurmrestd -vvvvv [ http://0.0.0.0:19090/ | 0.0.0.0:19090 ] 

[root at slurmctl slurmctld]# cd /proc/1235/ 
[root at slurmctl 1235]# cat environ | tr '\0' "\n" 
TERM=xterm 
TINI_VERSION=v0.18.0 
SHLVL=1 
HOSTNAME=slurmctl 
SUPERVISOR_ENABLED=1 
SUPERVISOR_PROCESS_NAME=slurmrestd 
PWD=/ 
SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock 
SUPERVISOR_GROUP_NAME=slurmrestd 
PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin 
HOME=/root 
SLURMRESTD_SECURITY=disable_user_check 
_=/usr/bin/supervisord 
``` 

I have generated correct token to request, but slurmrestd log message reported authentication failed: Unspecified error 
``` 
[root at slurmctl 1235]# scontrol token username=slurm 
SLURM_JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw 
[root at slurmctl 1235]# token="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw" 
[root at slurmctl 1235]# curl [ http://172.17.0.4:19090/openapi | 172.17.0.4:19090/openapi ] -H "X-SLURM-USER-TOKEN: $token" -H "X-SLURM-USER-NAME: slurm" 
Authentication failure 

slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[172.17.0.1]:38090] attempting user_name slurm token authentication pass through 
slurmrestd: error: operations_router: [[172.17.0.1]:38090] authentication failed: Unspecified error 
slurmrestd: debug2: _on_message_complete_request: [[172.17.0.1]:38090] on_http_request rejected: Unspecified error 
``` 

But! I have found that I'm setting `SLURM_JWT` environment variable for process, whatever token value is authenticated normally 
``` 
# start process with SLURM_JWT 
[root at slurmctl 1235]# cd /proc/2108/ 
[root at slurmctl 2108]# cat environ |tr "\0" "\n" 
TERM=xterm 
TINI_VERSION=v0.18.0 
SHLVL=1 
HOSTNAME=slurmctl 
SLURM_JWT=randomtoken 
SUPERVISOR_ENABLED=1 
SUPERVISOR_PROCESS_NAME=slurmrestd 
PWD=/ 
SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock 
SUPERVISOR_GROUP_NAME=slurmrestd 
PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin 
HOME=/root 
SLURMRESTD_SECURITY=disable_user_check 
_=/usr/bin/supervisord 

# request OK 
[root at slurmctl 2108]# curl [ http://172.17.0.4:19090/slurm/v0.0.35/jobs | 172.17.0.4:19090/slurm/v0.0.35/jobs ] -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm" 
[ 
] 
[root at slurmctl 2108]# curl [ http://172.17.0.4:19090/slurm/v0.0.35/jobs | 172.17.0.4:19090/slurm/v0.0.35/jobs ] -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: errorvalue" 
[ 
] 
``` 

So, I'm confused about JWT authentication. 
Q1: What is used for the `SLURM_JWT` environment variable, is it required for JWT? Related search from github source repo: [ https://github.com/SchedMD/slurm/search?q=SLURM_JWT | https://github.com/SchedMD/slurm/search?q=SLURM_JWT ] 
Q2: How to use slurmrestd JWT authentication? 

-------- 
Thanks, Chenyang Yan 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220321/6cf20921/attachment.htm>