<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000'><font face="arial, helvetica, sans-serif"><span style="font-size: 10pt;">Hi all,</span></font><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><br></div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;">we observe a lot of job which keep being in completing state until we kill the sleep process inside the step_extern cgroup.</div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><br></div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;">In these cases what we see on the involved nodes is a defunct slurmd</div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><br></div><div style=""><div style=""><span style="font-size: 13.3333px;"><font face="courier new, courier, monaco, monospace, sans-serif">[root@r113c18s01 ~]# ps --forest -lfe | egrep '[s]leep|[s]lurm'</font></span></div><div style=""><span style="font-size: 13.3333px;"><font face="courier new, courier, monaco, monospace, sans-serif">1 S root 26867 1 0 80 0 - 891256 inet_c Jan23 ? 00:03:48 /usr/sbin/slurmd</font></span></div><div style=""><span style="font-size: 13.3333px;"><font face="courier new, courier, monaco, monospace, sans-serif">1 Z root 25518 26867 0 80 0 - 0 exit 12:59 ? 00:00:00 \_ [slurmd] <defunct></font></span></div><div style=""><span style="font-size: 13.3333px;"><font face="courier new, courier, monaco, monospace, sans-serif">0 S root 25525 1 0 80 0 - 26974 hrtime 12:59 ? 00:00:00 sleep 1000000</font></span></div><div style=""><span style="font-size: 13.3333px;"><font face="courier new, courier, monaco, monospace, sans-serif"><br></font></span></div><div style=""><span style="font-size: 13.3333px;"><font face="courier new, courier, monaco, monospace, sans-serif"><div>[root@r113c18s01 ~]# cat /sys/fs/cgroup/cpuset/slurm/uid_29592/job_62379/step_extern/tasks </div><div>25525</div><div><br></div></font></span></div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;">we see from UNIX accounting logs that the step_extern slurmstepd died immediately</div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><br></div><div style=""><div style="color: rgb(0, 0, 0); font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">[root@r113c18s01 ~]# lastcomm --command slurmstepd | grep D</font></div><div style="color: rgb(0, 0, 0); font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">slurmstepd DX root __ 0.89 secs Tue Jan 30 12:59</font></div><div style="color: rgb(0, 0, 0); font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif"><br></font></div><div style="color: rgb(0, 0, 0); font-size: 10pt;"><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">[root@r113c18s01 ~]# dump-acct /var/account/pacct | grep 'Tue Jan 30 12:59' | grep slurm</font></div><div style="font-size: medium;"><div style=""><span style="font-size: 13.3333px;"><font face="courier new, courier, monaco, monospace, sans-serif">slurmd |v3| 0.00| 0.00| 0.00| 0| 0|3565056.00| 0.00| 25518 26867|Tue Jan 30 12:59:48 2018</font></span></div><div style=""><font style="" face="courier new, courier, monaco, monospace, sans-serif"><span style="font-size: 13.3333px;">slurmstepd |v3| 31.00| 58.00| 93.00| 0| 0|199680.00| 0.00| 25519 1|Tue Jan 30 12:59:49 2018</span></font></div><div style="font-family: "Times New Roman";"><font face="arial, helvetica, sans-serif"><span style="font-size: 13.3333px;"><br></span></font></div></div></div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><span style="font-size: 10pt;">So both the sleep and slurmstepd processes turn to be children of systemd (pid 1).</span></div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><br></div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;">Slurmd reports</div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><br></div><div style="color: rgb(0, 0, 0); font-size: 10pt;"><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">[root@r113c18s01 ~]# journalctl -u slurmd | grep 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 12:59:48 r113c18s01 slurmd[26867]: task_p_slurmd_batch_request: 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 12:59:48 r113c18s01 slurmd[26867]: task/affinity: job 62379 CPU input mask for node: 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 12:59:48 r113c18s01 slurmd[26867]: task/affinity: job 62379 CPU final HW mask for node: 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 12:59:48 r113c18s01 slurmd[26867]: debug: Waiting for job 62379's prolog to complete</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 12:59:48 r113c18s01 slurmd[26867]: debug: [job 62379] attempting to run prolog [/etc/slurm/prolog.d/create_local_tmpdir.sh]</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 12:59:48 r113c18s01 slurmd[26867]: _run_prolog: prolog with lock for job 62379 ran for 0 seconds</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 12:59:49 r113c18s01 slurmd[26867]: debug: _step_connect: connect() failed dir /var/spool/slurmd node r113c18s01 step 62379.4294967295 Connection refused</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 15:11:24 r113c18s01 slurmd[26867]: debug: _step_connect: connect() failed dir /var/spool/slurmd node r113c18s01 step 62379.4294967295 Connection refused</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 15:11:24 r113c18s01 slurmd[26867]: debug: Cleaned up stray socket /var/spool/slurmd/r113c18s01_62379.4294967295</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:00:13 r113c18s01 slurmd[26867]: Job 62379: timeout: sent SIGTERM to 0 active steps</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:00:13 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:00:13 r113c18s01 slurmd[26867]: debug: credential for job 62379 revoked</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:00:13 r113c18s01 slurmd[26867]: debug: Waiting for job 62379's prolog to complete</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:04:33 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:08:39 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:12:57 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:16:55 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:21:02 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:25:10 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:29:11 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:33:21 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:37:24 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:41:26 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:45:28 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:49:35 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-size: 10pt;"><font face="courier new, courier, monaco, monospace, sans-serif">Jan 30 17:53:37 r113c18s01 slurmd[26867]: debug: task_p_slurmd_release_resources: affinity jobid 62379</font></div><div style="font-family: arial, helvetica, sans-serif;"><br></div><div style="font-family: arial, helvetica, sans-serif;">We tried to setup an UnkillableStepProgram to kill the sleep process but the script is not invoked, we guess because</div><div style="font-family: arial, helvetica, sans-serif;">the slurmd is defunct.</div><div style="font-family: arial, helvetica, sans-serif;"><br></div><div style="font-family: arial, helvetica, sans-serif;">Any idea?</div><div style="font-family: arial, helvetica, sans-serif;"><br></div><div style="font-family: arial, helvetica, sans-serif;">Thanks</div><div style="font-family: arial, helvetica, sans-serif;">ale</div><div style="font-family: arial, helvetica, sans-serif;"><br></div><div style="font-family: arial, helvetica, sans-serif;"><br></div></div></div><font face="arial, helvetica, sans-serif"><span style="font-size: 10pt;">-- </span></font><br><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><span name="x"></span>Alessandro Federico <br>HPC System Management Group <br>System & Technology Department <br>CINECA www.cineca.it <br>Via dei Tizii 6, 00185 Rome - Italy <br>phone: +39 06 44486708 <br><br>All work and no play makes Jack a dull boy. <br>All work and no play makes Jack a dull boy. <br>All work and no play makes Jack...<span name="x"></span><br></div></div></div></body></html>