[slurm-users] Trying to troubleshoot slurmctld start failure

[Kevin Buckley]

On 2022/10/13 03:42, Sopena Ballesteros Manuel wrote:
> Dear Slurm user community,
> 
> 
> I am new to slurm and trying to start a slurmd and slurmctld on same machine. I started with slurmctld which is having issues.
> 
> 
> slurmctld: ext_sensors/none: init: ExtSensors NONE plugin loaded
> slurmctld: debug:  MPI: Loading all types
> slurmctld: error:  mpi/pmix_v3: init: (null) [0]: mpi_pmix.c:195: pmi/pmix: can not load PMIx library
> slurmctld: error: Couldn't load specified plugin name for mpi/pmix_v3: Plugin init() callback failed
> slurmctld: error: MPI: Cannot create context for mpi/pmix_v3
> slurmctld: debug2: No mpi.conf file (/etc/slurm/mpi.conf)

We don't use PMIx here but this bit in the mpi.conf manpage

   PMIxEnv=<environment variables>
       Comma separated list of environment variables to be set in job
       environments to be used by PMIx. Defaults to not being set.

suggests that you could set a LD_LIBRARY_PATH or similar EnvVar that
might expose your local PMIx library to Slurm jobs, so maybe the
daemons need similar.

Maybe try running the daemon startup with a lookup path set?

Kevin Buckley
-- 
Supercomputing Systems Administrator
Pawsey Supercomputing Centre