Hi,
I feel like I must have missed something obvious here, so hopefully someone can spot it! We recently updated to Slurm 24.11.6 on our cluster and I made sure to compile --with slurmrestd and --with jwt, as there are a few things I've been meaning to try out with the REST API.
From the rpmbuild config.log I get:
$ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc/slurm --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-systemdsystemunitdir=/usr/lib/systemd/system --enable-pkgconfig --with-jwt -- configure:21020: checking for jwt.h configure:21020: gcc -c -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer conftest.c >&5 configure:21020: $? = 0 configure:21020: result: yes configure:21028: checking for jwt_add_header in -ljwt configure:21051: gcc -o conftest -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -Wl,-z,relro -Wl,-z,lazy conftest.c -ljwt -ldl -lrt -lm -lresolv >&5 configure:21051: $? = 0 configure:21061: result: yes -- config.status:1746: creating src/plugins/auth/jwt/Makefile -- config.status:1746: creating src/slurmrestd/plugins/auth/jwt/Makefile -- ac_cv_header_jwt_h=yes -- ac_cv_lib_jwt_jwt_add_header=yes -- JWT_LDFLAGS='-ljwt'
And:
configure:26873: checking whether to compile slurmrestd configure:26904: result: yes -- configure:26915: checking for slurmrestd default port configure:26933: result: 6820
And indeed I get a slurmrestd package out of it, which I've successfully installed with no errors. I've created a JWT key and rolled it out to the controller node, but when I go to check auth types I get:
slurmrestd -a rest_auth/jwt slurmrestd: fatal: Unable to find plugin: rest_auth/jwt
Running an strace on this process shows that it does in fact stat the .so that does in fact exist:
stat("/usr/lib64/slurm/auth_jwt.so", {st_mode=S_IFREG|0755, st_size=283728, ...}) = 0
And I'm kind of stumped now. I've checked through all the resolved bug tickets to see if I can spot what is wrong but I'm getting nothing, and I can't find anything in the mailing lists either, other than things I've checked. Has anyone else had similar issues, or can someone see the blindingly obvious thing I'm missing?
Cheers Jon
Jon Marshall
High Performance Computing Specialist
IT and Scientific Computing Team
Cancer Research UK Cambridge Institute
Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE
Webhttp://www.cruk.cam.ac.uk/ | Facebookhttp://www.facebook.com/cancerresearchuk | Twitterhttp://twitter.com/CR_UK
[Description: CRI Logo]http://www.cruk.cam.ac.uk/
Hi Jon,
Did you check the software prerequisite minimum versions listed in this page: https://slurm.schedmd.com/rest_quickstart.html#prereq
IHTH, Ole
On 8/13/2025 4:28 PM, Jon Marshall via slurm-users wrote:
Hi,
I feel like I must have missed something obvious here, so hopefully someone can spot it! We recently updated to Slurm 24.11.6 on our cluster and I made sure to compile --with slurmrestd and --with jwt, as there are a few things I've been meaning to try out with the REST API.
From the rpmbuild config.log I get:
$ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat- linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/ etc/slurm --datadir=/usr/share --includedir=/usr/include --libdir=/usr/ lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/ var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with- systemdsystemunitdir=/usr/lib/systemd/system --enable-pkgconfig --with-jwt -- configure:21020: checking for jwt.h configure:21020: gcc -c -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,- D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc- switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection - fcf-protection -fno-omit-frame-pointer conftest.c >&5 configure:21020: $? = 0 configure:21020: result: yes configure:21028: checking for jwt_add_header in -ljwt configure:21051: gcc -o conftest -DNUMA_VERSION1_COMPATIBILITY -O2 -g - pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,- D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc- switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection - fcf-protection -fno-omit-frame-pointer -Wl,-z,relro -Wl,-z,lazy conftest.c -ljwt -ldl -lrt -lm -lresolv >&5 configure:21051: $? = 0 configure:21061: result: yes -- config.status:1746: creating src/plugins/auth/jwt/Makefile -- config.status:1746: creating src/slurmrestd/plugins/auth/jwt/Makefile -- ac_cv_header_jwt_h=yes -- ac_cv_lib_jwt_jwt_add_header=yes -- JWT_LDFLAGS='-ljwt'
And:
configure:26873: checking whether to compile slurmrestd configure:26904: result: yes -- configure:26915: checking for slurmrestd default port configure:26933: result: 6820
And indeed I get a slurmrestd package out of it, which I've successfully installed with no errors. I've created a JWT key and rolled it out to the controller node, but when I go to check auth types I get:
slurmrestd -a rest_auth/jwt slurmrestd: fatal: Unable to find plugin: rest_auth/jwt
Running an strace on this process shows that it does in fact stat the .so that does in fact exist:
stat("/usr/lib64/slurm/auth_jwt.so", {st_mode=S_IFREG|0755, st_size=283728, ...}) = 0
And I'm kind of stumped now. I've checked through all the resolved bug tickets to see if I can spot what is wrong but I'm getting nothing, and I can't find anything in the mailing lists either, other than things I've checked. Has anyone else had similar issues, or can someone see the blindingly obvious thing I'm missing?
Hi Ole,
Sorry, missed your reply to this - I did indeed and the machine I am building the packages on definitely has jwt-devel packages installed, as noted in the rpmbuild config.
Basically it looks like it is getting built, and when the program runs it looks up auth_jwt.so which definitely exists, but then it craps out saying it can't find it.
Cheers Jon ________________________________ From: Ole Holm Nielsen via slurm-users slurm-users@lists.schedmd.com Sent: 14 August 2025 06:57 To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [slurm-users] Re: slurmrestd unable to find plugin rest_auth/jwt
Hi Jon,
Did you check the software prerequisite minimum versions listed in this page: https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fslurm.schedmd.com%2frest...
IHTH, Ole
On 8/13/2025 4:28 PM, Jon Marshall via slurm-users wrote:
Hi,
I feel like I must have missed something obvious here, so hopefully someone can spot it! We recently updated to Slurm 24.11.6 on our cluster and I made sure to compile --with slurmrestd and --with jwt, as there are a few things I've been meaning to try out with the REST API.
From the rpmbuild config.log I get:
$ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat- linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/ etc/slurm --datadir=/usr/share --includedir=/usr/include --libdir=/usr/ lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/ var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with- systemdsystemunitdir=/usr/lib/systemd/system --enable-pkgconfig --with-jwt -- configure:21020: checking for jwt.h configure:21020: gcc -c -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,- D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc- switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection - fcf-protection -fno-omit-frame-pointer conftest.c >&5 configure:21020: $? = 0 configure:21020: result: yes configure:21028: checking for jwt_add_header in -ljwt configure:21051: gcc -o conftest -DNUMA_VERSION1_COMPATIBILITY -O2 -g - pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,- D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc- switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection - fcf-protection -fno-omit-frame-pointer -Wl,-z,relro -Wl,-z,lazy conftest.c -ljwt -ldl -lrt -lm -lresolv >&5 configure:21051: $? = 0 configure:21061: result: yes -- config.status:1746: creating src/plugins/auth/jwt/Makefile -- config.status:1746: creating src/slurmrestd/plugins/auth/jwt/Makefile -- ac_cv_header_jwt_h=yes -- ac_cv_lib_jwt_jwt_add_header=yes -- JWT_LDFLAGS='-ljwt'
And:
configure:26873: checking whether to compile slurmrestd configure:26904: result: yes -- configure:26915: checking for slurmrestd default port configure:26933: result: 6820
And indeed I get a slurmrestd package out of it, which I've successfully installed with no errors. I've created a JWT key and rolled it out to the controller node, but when I go to check auth types I get:
slurmrestd -a rest_auth/jwt slurmrestd: fatal: Unable to find plugin: rest_auth/jwt
Running an strace on this process shows that it does in fact stat the .so that does in fact exist:
stat("/usr/lib64/slurm/https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fauth_jwt.so&c=E,1,pz...", {st_mode=S_IFREG|0755, st_size=283728, ...}) = 0
And I'm kind of stumped now. I've checked through all the resolved bug tickets to see if I can spot what is wrong but I'm getting nothing, and I can't find anything in the mailing lists either, other than things I've checked. Has anyone else had similar issues, or can someone see the blindingly obvious thing I'm missing?
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Further to this, it appears that I'm getting confused here - there is an auth_jwt.so, but I actually want rest_auth_jwt.so which is not getting built via rpmbuild. If I manually compile and install with --enable-slurmrestd and --with_jwt=/usr, then the required so is generated and installed.
I will continue to investigate the build logs for rpmbuild to see if I can figure out where it is going wrong.
Cheers Jon ________________________________ From: Jon Marshall via slurm-users slurm-users@lists.schedmd.com Sent: 27 August 2025 12:32 To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com; Ole Holm Nielsen Ole.H.Nielsen@fysik.dtu.dk Subject: [slurm-users] Re: slurmrestd unable to find plugin rest_auth/jwt
Hi Ole,
Sorry, missed your reply to this - I did indeed and the machine I am building the packages on definitely has jwt-devel packages installed, as noted in the rpmbuild config.
Basically it looks like it is getting built, and when the program runs it looks up auth_jwt.sohttps://linkprotect.cudasvc.com/url?a=https%3a%2f%2fauth_jwt.so&c=E,1,vzYqcGXmXu56Qol_c6IpWIu4S3GwabXV9He3gz42cfYo8y3I-825045IkkTKZHEwXV65y5A9-iue1UzMkRKs8SXtW9QHUGHZCRwhrTaT5NS229E,&typo=1&ancr_add=1 which definitely exists, but then it craps out saying it can't find it.
Cheers Jon ________________________________ From: Ole Holm Nielsen via slurm-users slurm-users@lists.schedmd.com Sent: 14 August 2025 06:57 To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [slurm-users] Re: slurmrestd unable to find plugin rest_auth/jwt
Hi Jon,
Did you check the software prerequisite minimum versions listed in this page: https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fslurm.schedmd.com%2frest...
IHTH, Ole
On 8/13/2025 4:28 PM, Jon Marshall via slurm-users wrote:
Hi,
I feel like I must have missed something obvious here, so hopefully someone can spot it! We recently updated to Slurm 24.11.6 on our cluster and I made sure to compile --with slurmrestd and --with jwt, as there are a few things I've been meaning to try out with the REST API.
From the rpmbuild config.log I get:
$ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat- linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/ etc/slurm --datadir=/usr/share --includedir=/usr/include --libdir=/usr/ lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/ var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with- systemdsystemunitdir=/usr/lib/systemd/system --enable-pkgconfig --with-jwt -- configure:21020: checking for jwt.h configure:21020: gcc -c -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,- D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc- switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection - fcf-protection -fno-omit-frame-pointer conftest.c >&5 configure:21020: $? = 0 configure:21020: result: yes configure:21028: checking for jwt_add_header in -ljwt configure:21051: gcc -o conftest -DNUMA_VERSION1_COMPATIBILITY -O2 -g - pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,- D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc- switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection - fcf-protection -fno-omit-frame-pointer -Wl,-z,relro -Wl,-z,lazy conftest.c -ljwt -ldl -lrt -lm -lresolv >&5 configure:21051: $? = 0 configure:21061: result: yes -- config.status:1746: creating src/plugins/auth/jwt/Makefile -- config.status:1746: creating src/slurmrestd/plugins/auth/jwt/Makefile -- ac_cv_header_jwt_h=yes -- ac_cv_lib_jwt_jwt_add_header=yes -- JWT_LDFLAGS='-ljwt'
And:
configure:26873: checking whether to compile slurmrestd configure:26904: result: yes -- configure:26915: checking for slurmrestd default port configure:26933: result: 6820
And indeed I get a slurmrestd package out of it, which I've successfully installed with no errors. I've created a JWT key and rolled it out to the controller node, but when I go to check auth types I get:
slurmrestd -a rest_auth/jwt slurmrestd: fatal: Unable to find plugin: rest_auth/jwt
Running an strace on this process shows that it does in fact stat the .so that does in fact exist:
stat("/usr/lib64/slurm/https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fauth_jwt.so&c=E,1,pz...", {st_mode=S_IFREG|0755, st_size=283728, ...}) = 0
And I'm kind of stumped now. I've checked through all the resolved bug tickets to see if I can spot what is wrong but I'm getting nothing, and I can't find anything in the mailing lists either, other than things I've checked. Has anyone else had similar issues, or can someone see the blindingly obvious thing I'm missing?
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
For anyone interested, I managed to resolve this by running:
rpmbuild -ta slurm-24.11.6.tar.bz2 --with slurmrestd --define "_with_jwt --with-jwt=/usr"
This has now successfully built the rest_auth_jwt.so and I've got the daemon running.
Cheers Jon ________________________________ From: Jon Marshall Jon.Marshall@cruk.cam.ac.uk Sent: 27 August 2025 14:16 To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com; Ole Holm Nielsen Ole.H.Nielsen@fysik.dtu.dk; Jon Marshall Jon.Marshall@cruk.cam.ac.uk Subject: Re: [slurm-users] Re: slurmrestd unable to find plugin rest_auth/jwt
Further to this, it appears that I'm getting confused here - there is an auth_jwt.so, but I actually want rest_auth_jwt.so which is not getting built via rpmbuild. If I manually compile and install with --enable-slurmrestd and --with_jwt=/usr, then the required so is generated and installed.
I will continue to investigate the build logs for rpmbuild to see if I can figure out where it is going wrong.
Cheers Jon ________________________________ From: Jon Marshall via slurm-users slurm-users@lists.schedmd.com Sent: 27 August 2025 12:32 To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com; Ole Holm Nielsen Ole.H.Nielsen@fysik.dtu.dk Subject: [slurm-users] Re: slurmrestd unable to find plugin rest_auth/jwt
Hi Ole,
Sorry, missed your reply to this - I did indeed and the machine I am building the packages on definitely has jwt-devel packages installed, as noted in the rpmbuild config.
Basically it looks like it is getting built, and when the program runs it looks up auth_jwt.sohttps://linkprotect.cudasvc.com/url?a=https%3a%2f%2fauth_jwt.so&c=E,1,vzYqcGXmXu56Qol_c6IpWIu4S3GwabXV9He3gz42cfYo8y3I-825045IkkTKZHEwXV65y5A9-iue1UzMkRKs8SXtW9QHUGHZCRwhrTaT5NS229E,&typo=1&ancr_add=1 which definitely exists, but then it craps out saying it can't find it.
Cheers Jon ________________________________ From: Ole Holm Nielsen via slurm-users slurm-users@lists.schedmd.com Sent: 14 August 2025 06:57 To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [slurm-users] Re: slurmrestd unable to find plugin rest_auth/jwt
Hi Jon,
Did you check the software prerequisite minimum versions listed in this page: https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fslurm.schedmd.com%2frest...
IHTH, Ole
On 8/13/2025 4:28 PM, Jon Marshall via slurm-users wrote:
Hi,
I feel like I must have missed something obvious here, so hopefully someone can spot it! We recently updated to Slurm 24.11.6 on our cluster and I made sure to compile --with slurmrestd and --with jwt, as there are a few things I've been meaning to try out with the REST API.
From the rpmbuild config.log I get:
$ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat- linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/ etc/slurm --datadir=/usr/share --includedir=/usr/include --libdir=/usr/ lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/ var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with- systemdsystemunitdir=/usr/lib/systemd/system --enable-pkgconfig --with-jwt -- configure:21020: checking for jwt.h configure:21020: gcc -c -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,- D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc- switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection - fcf-protection -fno-omit-frame-pointer conftest.c >&5 configure:21020: $? = 0 configure:21020: result: yes configure:21028: checking for jwt_add_header in -ljwt configure:21051: gcc -o conftest -DNUMA_VERSION1_COMPATIBILITY -O2 -g - pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,- D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc- switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection - fcf-protection -fno-omit-frame-pointer -Wl,-z,relro -Wl,-z,lazy conftest.c -ljwt -ldl -lrt -lm -lresolv >&5 configure:21051: $? = 0 configure:21061: result: yes -- config.status:1746: creating src/plugins/auth/jwt/Makefile -- config.status:1746: creating src/slurmrestd/plugins/auth/jwt/Makefile -- ac_cv_header_jwt_h=yes -- ac_cv_lib_jwt_jwt_add_header=yes -- JWT_LDFLAGS='-ljwt'
And:
configure:26873: checking whether to compile slurmrestd configure:26904: result: yes -- configure:26915: checking for slurmrestd default port configure:26933: result: 6820
And indeed I get a slurmrestd package out of it, which I've successfully installed with no errors. I've created a JWT key and rolled it out to the controller node, but when I go to check auth types I get:
slurmrestd -a rest_auth/jwt slurmrestd: fatal: Unable to find plugin: rest_auth/jwt
Running an strace on this process shows that it does in fact stat the .so that does in fact exist:
stat("/usr/lib64/slurm/https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fauth_jwt.so&c=E,1,pz...", {st_mode=S_IFREG|0755, st_size=283728, ...}) = 0
And I'm kind of stumped now. I've checked through all the resolved bug tickets to see if I can spot what is wrong but I'm getting nothing, and I can't find anything in the mailing lists either, other than things I've checked. Has anyone else had similar issues, or can someone see the blindingly obvious thing I'm missing?
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-
Jon Marshall -
Ole Holm Nielsen