[slurm-users] Slurmrestd authentication failed: Unspecified error

Chenyang Yan memory.yancy at gmail.com
Mon Mar 21 11:33:32 UTC 2022 

Guillaume, thanks for your reply and your workaround.

Do you try adding `SLUTM_JWT=daemon` fo slurmrestd process, and then any
value is is authenticated successfully for request header
`X-SLURM-USER-TOKEN`, `X-SLURM-USER-NAME` .

You can see the information in my last email:

```
# start process with SLURM_JWT
[root at slurmctl 1235]# cd /proc/2108/
[root at slurmctl 2108]# cat environ |tr "\0" "\n"
TERM=xterm
TINI_VERSION=v0.18.0
SHLVL=1
HOSTNAME=slurmctl
SLURM_JWT=randomtoken                   <===== SLURM_JWT environ
SUPERVISOR_ENABLED=1
SUPERVISOR_PROCESS_NAME=slurmrestd
PWD=/
SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock
SUPERVISOR_GROUP_NAME=slurmrestd
PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin
HOME=/root
SLURMRESTD_SECURITY=disable_user_check
_=/usr/bin/supervisord

# request OK
[root at slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H
"X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm"
[
 ]
[root at slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H
"X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME:
errorvalue"
[
 ]
```

So, I'm confused for slurmrestd JWT authentication.

--------------------
Thanks, Chenyang Yan

On Mon, Mar 21, 2022 at 4:10 PM Guillaume COCHARD <
guillaume.cochard at cc.in2p3.fr> wrote:

> Hello,
>
> We had the same error and we fixed it by adding
> `Environment="SLURM_JWT=daemon"` to the [Service] section of the unit file
> (in our case /usr/lib/systemd/system/slurmrestd.service ).
>
> We have a bug (feature?) that makes us unable to use root or slurm user as
> user for slurmrestd service so maybe you'll encounter that as well.
> According to the release notes (
> https://github.com/SchedMD/slurm/blob/master/NEWS ) this should be
> starting on version 22.05.0pre1 but it is already the case in our version
> 21.08.6. We have an administrator user for this can of usage.
>
> Our service file is like this:
>
> $ cat /usr/lib/systemd/system/slurmrestd.service
> [Unit]
> Description=Slurm REST daemon
> After=network.target munge.service
> ConditionPathExists=/etc/slurm/slurm.conf
>
> [Service]
> User=slurmadm
> Type=simple
> EnvironmentFile=-/etc/sysconfig/slurmrestd
> Environment="SLURM_JWT=daemon"
> ExecStart=/usr/sbin/slurmrestd -v -a rest_auth/jwt localhost:6820
> ExecReload=/bin/kill -HUP $MAINPID
>
> [Install]
> WantedBy=multi-user.target
>
>
> Guillaume
>
>
>
> ------------------------------
> *De: *"Chenyang Yan" <memory.yancy at gmail.com>
> *À: *slurm-users at schedmd.com
> *Envoyé: *Samedi 19 Mars 2022 14:09:06
> *Objet: *[slurm-users] Slurmrestd authentication failed: Unspecified error
>
> Hello,
> I have met a similar issue with slurmrestd authentication failed error,
> similar question:
> https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html
>
> I have installed `slurm 21.08.6` on CentOS 7.9.2009 container, basic
> service is running fine
> ```
> [root at slurmctl supervisor]# ls -l /.dockerenv
> -rwxr-xr-x. 1 root root 0 Mar 17 23:17 /.dockerenv
> [root at slurmctl supervisor]# srun --partition normal hostname
> slurmctl
> ```
>
> Slurmrestd is compiled with `--enable-slurmrestd` successfully, slurmrestd
> JWT configuration is as follows:
> ```
> [root at slurmctl slurmctld]# dd if=/dev/random
> of=/var/spool/slurm/jwt_hs256.key bs=32 count=1
> [root at slurmctl slurmctld]# scontrol show config |grep -i auth
> AuthAltTypes            = auth/jwt
> AuthAltParameters       = jwt_key=/var/spool/slurm/jwt_hs256.key
> AuthInfo                = (null)
> AuthType                = auth/munge
> [root at slurmctl slurmctld]# ll -l /var/spool/slurm/jwt_hs256.key
> -rw-r--r--. 1 root root 32 Mar 17 23:21 /var/spool/slurm/jwt_hs256.key
>
> # start slurmrestd process
> [root at slurmctl slurmctld]# SLURMRESTD_SECURITY=disable_user_check
> /usr/sbin/slurmrestd -vvvv 0.0.0.0:19090
>
> # check process and environ
> [root at slurmctl slurmctld]# ps -ef |grep slurmrestd
> root      1235   236  0 23:26 ?        00:00:00 /usr/sbin/slurmrestd
> -vvvvv 0.0.0.0:19090
>
> [root at slurmctl slurmctld]# cd /proc/1235/
> [root at slurmctl 1235]# cat environ | tr '\0' "\n"
> TERM=xterm
> TINI_VERSION=v0.18.0
> SHLVL=1
> HOSTNAME=slurmctl
> SUPERVISOR_ENABLED=1
> SUPERVISOR_PROCESS_NAME=slurmrestd
> PWD=/
> SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock
> SUPERVISOR_GROUP_NAME=slurmrestd
>
> PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin
> HOME=/root
> SLURMRESTD_SECURITY=disable_user_check
> _=/usr/bin/supervisord
> ```
>
> I have generated correct token to request, but slurmrestd log message
> reported authentication failed: Unspecified error
> ```
> [root at slurmctl 1235]# scontrol token username=slurm
>
> SLURM_JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw
> [root at slurmctl 1235]#
> token="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw"
> [root at slurmctl 1235]# curl 172.17.0.4:19090/openapi -H
> "X-SLURM-USER-TOKEN: $token" -H "X-SLURM-USER-NAME: slurm"
> Authentication failure
>
>
> slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate:
> [[172.17.0.1]:38090] attempting user_name slurm token authentication pass
> through
> slurmrestd: error: operations_router: [[172.17.0.1]:38090] authentication
> failed: Unspecified error
> slurmrestd: debug2: _on_message_complete_request: [[172.17.0.1]:38090]
> on_http_request rejected: Unspecified error
> ```
>
> But! I have found that I'm setting `SLURM_JWT` environment variable for
> process, whatever token value is authenticated normally
> ```
> # start process with SLURM_JWT
> [root at slurmctl 1235]# cd /proc/2108/
> [root at slurmctl 2108]# cat environ |tr "\0" "\n"
> TERM=xterm
> TINI_VERSION=v0.18.0
> SHLVL=1
> HOSTNAME=slurmctl
> SLURM_JWT=randomtoken
> SUPERVISOR_ENABLED=1
> SUPERVISOR_PROCESS_NAME=slurmrestd
> PWD=/
> SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock
> SUPERVISOR_GROUP_NAME=slurmrestd
>
> PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin
> HOME=/root
> SLURMRESTD_SECURITY=disable_user_check
> _=/usr/bin/supervisord
>
> # request OK
> [root at slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H
> "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm"
> [
>  ]
> [root at slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H
> "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME:
> errorvalue"
> [
>  ]
> ```
>
> So, I'm confused about JWT authentication.
> Q1: What is used for the `SLURM_JWT` environment variable, is it required
> for JWT? Related search from github source repo:
> https://github.com/SchedMD/slurm/search?q=SLURM_JWT
> Q2: How to use slurmrestd JWT authentication?
>
> --------
> Thanks, Chenyang Yan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220321/3e2b6000/attachment-0001.htm>

More information about the slurm-users mailing list