[slurm-users] slurmstepd crash 18.03 when using pmi2 interface

[Bas van der Vlies]

Oke if we change:
  * TaskPlugin=task/affinity,task/cgroup

to:
  * TaskPlugin=task/affinity

The pmi2 interface works. Investigating this further

On 31/10/2018 08:26, Bas van der Vlies wrote:
> I am busy with migrating from Torque/Moab to SLURM.
> 
> I have installed slurm 18.03 and trying to run an mpi program woth the 
> pmi2 interface.
> 
> {{{
> ~/mpitest> srun --mpi=list
> srun: MPI types are...
> srun: none
> srun: openmpi
> srun: pmi2
> }}}
> 
> The none and openmpi interface works but the pmi2 interface crashes the 
> slurmstepd. Have I missed some setting or is this a bug?
> 
> {{{
> (gdb) thread apply all bt
> 
> Thread 6 (Thread 0x2b9ce9b8b700 (LWP 21945)):
> #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at 
> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
> #1  0x00002b9ce5c7862b in ?? () from 
> /usr/lib/x86_64-linux-gnu/slurm/libslurmfull.so
> #2  0x00002b9ce6c08494 in start_thread (arg=0x2b9ce9b8b700) at 
> pthread_create.c:333
> #3  0x00002b9ce6f06acf in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> 
> Thread 5 (Thread 0x2b9ce9c8c700 (LWP 21946)):
> #0  0x00002b9ce6efd67d in poll () at ../sysdeps/unix/syscall-template.S:84
> #1  0x00002b9ce5d16cfb in slurm_eio_handle_mainloop () from 
> /usr/lib/x86_64-linux-gnu/slurm/libslurmfull.so
> #2  0x00005631c29f69f6 in ?? ()
> #3  0x00002b9ce6c08494 in start_thread (arg=0x2b9ce9c8c700) at 
> pthread_create.c:333
> #4  0x00002b9ce6f06acf in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> 
> Thread 4 (Thread 0x2b9ceaedb700 (LWP 21948)):
> #0  0x00002b9ce6efd67d in poll () at ../sysdeps/unix/syscall-template.S:84
> #1  0x00002b9cea2a8f52 in ?? () from 
> /usr/lib/x86_64-linux-gnu/slurm//task_cgroup.so
> #2  0x00002b9ce6c08494 in start_thread (arg=0x2b9ceaedb700) at 
> pthread_create.c:333
> #3  0x00002b9ce6f06acf in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> 
> Thread 3 (Thread 0x2b9ceadda700 (LWP 21947)):
> #0  0x00002b9ce6efd67d in poll () at ../sysdeps/unix/syscall-template.S:84
> #1  0x00002b9ce5d16cfb in slurm_eio_handle_mainloop () from 
> /usr/lib/x86_64-linux-gnu/slurm/libslurmfull.so
> #2  0x00002b9ceaac7355 in ?? () from 
> /usr/lib/x86_64-linux-gnu/slurm//mpi_pmi2.so
> #3  0x00002b9ce6c08494 in start_thread (arg=0x2b9ceadda700) at 
> pthread_create.c:333
> #4  0x00002b9ce6f06acf in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> 
> Thread 2 (Thread 0x2b9ce5ae0700 (LWP 21944)):
> #0  pthread_cond_wait@@GLIBC_2.3.2 () at 
> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> #1  0x00002b9ce5c7e65d in ?? () from 
> /usr/lib/x86_64-linux-gnu/slurm/libslurmfull.so
> #2  0x00002b9ce6c08494 in start_thread (arg=0x2b9ce5ae0700) at 
> pthread_create.c:333
> #3  0x00002b9ce6f06acf in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> 
> Thread 1 (Thread 0x2b9ce59dd080 (LWP 21943)):
> #0  __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> #1  0x00002b9ce6e5242a in __GI_abort () at abort.c:89
> #2  0x00002b9ce6e8ec00 in __libc_message (do_abort=do_abort at entry=2, 
> fmt=fmt at entry=0x2b9ce6f83d98 "*** Error in `%s': %s: 0x%s ***\n")
>      at ../sysdeps/posix/libc_fatal.c:175
> #3  0x00002b9ce6e94fc6 in malloc_printerr (action=3, str=0x2b9ce6f8094a 
> "free(): invalid pointer", ptr=<optimized out>,
>      ar_ptr=<optimized out>) at malloc.c:5049
> #4  0x00002b9ce6e9580e in _int_free (av=0x2b9ce71b7b00 <main_arena>, 
> p=0x2b9ce71bba60 <lock>, have_lock=0) at malloc.c:3905
> #5  0x00002b9ce5d1084d in slurm_xfree () from 
> /usr/lib/x86_64-linux-gnu/slurm/libslurmfull.so
> #6  0x00002b9cea2ab0b0 in task_cgroup_devices_create () from 
> /usr/lib/x86_64-linux-gnu/slurm//task_cgroup.so
> #7  0x00002b9cea2a5977 in task_p_pre_setuid () from 
> /usr/lib/x86_64-linux-gnu/slurm//task_cgroup.so
> #8  0x00005631c2a04216 in task_g_pre_setuid ()
> #9  0x00005631c29e713d in ?? ()
> #10 0x00005631c29ec3f4 in job_manager ()
> #11 0x00005631c29e9374 in main ()
> }}}}
> 
> 
> 

-- 
--
Bas van der Vlies
| Operations, Support & Development | SURFsara | Science Park 140 | 1098 
XG  Amsterdam
| T +31 (0) 20 800 1300  | bas.vandervlies at surfsara.nl | www.surfsara.nl |

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3601 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20181101/77cd2a1d/attachment.bin>