Hi Josh,

thanks for reply, that's very helpful. I used exact same compilation setup as you did, I could have mentioned that. But this gives extra confidence. So I will just accept current situation and test it as soon as I have gpus available.

Best, Matthias

Am 13.11.24 um 13:58 schrieb Joshua Randall via slurm-users:

Hi Matthias,

Just another user here, but we did notice similar behaviour on our cluster with NVIDIA GPU nodes. For this cluster, we built slurm 24.05.1 deb packages from source ourselves on Ubuntu 22.04 with the `libnvidia- ml-dev` package installed directly from the Ubuntu package archive (using the mk-build-deps / debuild method described here: https:// slurm.schedmd.com/quickstart_admin.html#debuild <https:// slurm.schedmd.com/quickstart_admin.html#debuild>)

In our cluster, the dynamic object dependencies for the gpu_nvml.so shared object file looks the same as yours (they do not show a dependency on  /lib/x86_64-linux-gnu/libnvidia-ml.so.1 though we do have it available):

ubuntu@gpu0:~$ ldd /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so
         linux-vdso.so.1 (0x00007ffe8c3b4000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f76301a9000)
         /lib64/ld-linux-x86-64.so.2 (0x00007f76303ef000)

However, NVML autodetection is working:

ubuntu@gpu0:~$ sudo grep nvml /var/log/slurm/slurmd.log  | tail -n 1
[2024-11-05T16:09:06.359] gpu/nvml: _get_system_gpu_list_nvml: 8 GPU 
system device(s) detected

I can also confirm that NVML library functions are being referenced from gpu_nvml.so (but are undefined therein):

ubuntu@gpu0:~$ objdump -T /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so | 
grep nvmlInit_v2
0000000000000000      D  *UND*  0000000000000000  Base        nvmlInit_v2

It looks like at some point, slurm has moved to a model where the NVML library (libnvidia-ml.so) is autodetected and dlopen'ed prior to being needed by the plugin, so the plugins can now assume that it will be preloaded if available and no longer need to have a shared library dependency on it: https://github.com/SchedMD/slurm/blob/slurm-24-05-1-1/ src/interfaces/gpu.c#L80-L101 <https://github.com/SchedMD/slurm/blob/ slurm-24-05-1-1/src/interfaces/gpu.c#L80-L101>

Cheers,

Josh.

-- Dr. Joshua C. Randall Principal Software Engineer Altos Labs email: jrandall@altoslabs.com mailto:jrandall@altoslabs.com

On Wed, Nov 13, 2024 at 10:21 AM Matthias Leopold via slurm-users <slurm-users@lists.schedmd.com mailto:slurm-users@lists.schedmd.com> wrote:

Hi,

I'm trying to compile Slurm with NVIDIA NVML support, but the result is
unexpected. I get /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so, but when
I do "ldd /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so" there is no
reference to /lib/x86_64-linux-gnu/libnvidia-ml.so.1 (which I would
expect).

~$ ldd  /usr/lib/x86_64-linux-gnu/slurm/gpu_nvml.so
          linux-vdso.so.1 (0x00007ffd9a3f4000)
          libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
(0x00007f0bc2c06000)
          /lib64/ld-linux-x86-64.so.2 (0x00007f0bc2e47000)

/lib/x86_64-linux-gnu/libnvidia-ml.so.1 is present during compilation.
Also I can see that NVML headers where found in config.status (else I
wouldn't get gpu_nvml.so at all to my understanding).

Our old cluster was deployed with NVIDIA deepops (which compiles Slurm
on every node) and also has NVML support. There ldd brings the expected
result

~$ ldd /usr/local/lib/slurm/gpu_nvml.so
...
libnvidia-ml.so.1 => /lib/x86_64-linux-gnu/libnvidia-ml.so.1
(0x00007f3b10120000)
...

I can't test actual functionality with my new binaries because I don't
have a node with GPUs yet.

Am I missing something?

thank you
Matthias


-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
<mailto:slurm-users@lists.schedmd.com>
To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
<mailto:slurm-users-leave@lists.schedmd.com>

Altos Labs UK Limited | England | Company reg 13484917 Registered address: 3rd Floor 1 Ashley Road, Altrincham, Cheshire, United Kingdom, WA14 2DT