[slurm-users] Is this a bug in slurm array completion logic or expected behaviour

Thu Jan 30 15:52:49 UTC 2020

Hi

I want to run an epilogctld after all parts of an array job have completed
in order to clean up an on demand filesystem created in the prologctld.

First I though I could just assume that I could run the epilog after the
completion of the final job step until I realised that they might not take
the same amount of time and though Slurm must be able to handle this logic
I shall read the docs!

Since the array job docs page says:
"A job which is to be dependent upon an entire job array should specify
itself dependent upon the ArrayJobID."

I thought I would poll this using a horrible bit of bash and sed in my
pro/epilog and dump to a log file

I then submit a synthetic array job with 10 elements where each element
sleeps for a random amount of time (upto 10 seconds)

prolog jobid=511,arraytaskid=2,SLURM_ARRAY_JOB_ID=509
prolog jobid=512,arraytaskid=3,SLURM_ARRAY_JOB_ID=509
prolog jobid=513,arraytaskid=4,SLURM_ARRAY_JOB_ID=509
prolog jobid=514,arraytaskid=5,SLURM_ARRAY_JOB_ID=509
prolog jobid=515,arraytaskid=6,SLURM_ARRAY_JOB_ID=509
prolog jobid=516,arraytaskid=7,SLURM_ARRAY_JOB_ID=509
prolog jobid=517,arraytaskid=8,SLURM_ARRAY_JOB_ID=509
prolog jobid=509,arraytaskid=10,SLURM_ARRAY_JOB_ID=509
prolog jobid=510,arraytaskid=1,SLURM_ARRAY_JOB_ID=509
prolog jobid=518,arraytaskid=9,SLURM_ARRAY_JOB_ID=509
epilog jobid=513,arraytaskid=4,SLURM_ARRAY_JOB_ID=509,JobState(509)=RUNNING
epilog jobid=514,arraytaskid=5,SLURM_ARRAY_JOB_ID=509,JobState(509)=RUNNING
epilog jobid=515,arraytaskid=6,SLURM_ARRAY_JOB_ID=509,JobState(509)=RUNNING
epilog jobid=518,arraytaskid=9,SLURM_ARRAY_JOB_ID=509,JobState(509)=RUNNING
epilog jobid=512,arraytaskid=3,SLURM_ARRAY_JOB_ID=509,JobState(509)=RUNNING
epilog jobid=517,arraytaskid=8,SLURM_ARRAY_JOB_ID=509,JobState(509)=RUNNING
epilog jobid=509,arraytaskid=10,SLURM_ARRAY_JOB_ID=509,JobState(509)=COMPLETING
epilog jobid=516,arraytaskid=7,SLURM_ARRAY_JOB_ID=509,JobState(509)=COMPLETED
epilog jobid=511,arraytaskid=2,SLURM_ARRAY_JOB_ID=509,JobState(509)=COMPLETED
epilog jobid=510,arraytaskid=1,SLURM_ARRAY_JOB_ID=509,JobState(509)=COMPLETED

Slurm seems to think that the array job is complete after the final array
task has completed even thought there are 3 more tasks running. If I was to
tear down the filesystem when the job goes into the completing step as I
was planning to then I would cause an error in tasks 7,2 and 1. This would
also happen if I was using a job dependency like --depend=afterany:509 to
try and do the same thing.

I have repeated it 2x with some timestamps in the pro and epliog scripts to
show that it's not due to a race condition printing, there as a good few
seconds between the completing job and the remaining jobs that are yet to
complete.

prolog 15:42:46.456925345 jobid=530,arraytaskid=1,SLURM_ARRAY_JOB_ID=529
prolog 15:42:46.461794268 jobid=532,arraytaskid=3,SLURM_ARRAY_JOB_ID=529
prolog 15:42:46.460449387 jobid=531,arraytaskid=2,SLURM_ARRAY_JOB_ID=529
prolog 15:42:46.464872996 jobid=533,arraytaskid=4,SLURM_ARRAY_JOB_ID=529
prolog 15:42:46.466919112 jobid=534,arraytaskid=5,SLURM_ARRAY_JOB_ID=529
prolog 15:42:46.469350534 jobid=536,arraytaskid=7,SLURM_ARRAY_JOB_ID=529
prolog 15:42:46.472210879 jobid=537,arraytaskid=8,SLURM_ARRAY_JOB_ID=529
prolog 15:42:46.473099772 jobid=538,arraytaskid=9,SLURM_ARRAY_JOB_ID=529
prolog 15:42:46.476322345 jobid=535,arraytaskid=6,SLURM_ARRAY_JOB_ID=529
prolog 15:42:46.479001241 jobid=529,arraytaskid=10,SLURM_ARRAY_JOB_ID=529
epilog 15:42:48.725625849
jobid=536,arraytaskid=7,SLURM_ARRAY_JOB_ID=529,JobState(529)=RUNNING
epilog 15:42:49.731580409
jobid=529,arraytaskid=10,SLURM_ARRAY_JOB_ID=529,JobState(529)=COMPLETING
epilog 15:42:50.597469868
jobid=532,arraytaskid=3,SLURM_ARRAY_JOB_ID=529,JobState(529)=COMPLETED
epilog 15:42:50.678213697
jobid=533,arraytaskid=4,SLURM_ARRAY_JOB_ID=529,JobState(529)=COMPLETED
epilog 15:42:50.714197275
jobid=534,arraytaskid=5,SLURM_ARRAY_JOB_ID=529,JobState(529)=COMPLETED
epilog 15:42:50.721083470
jobid=531,arraytaskid=2,SLURM_ARRAY_JOB_ID=529,JobState(529)=COMPLETED
epilog 15:42:53.707315733
jobid=530,arraytaskid=1,SLURM_ARRAY_JOB_ID=529,JobState(529)=COMPLETED
epilog 15:42:53.728610476
jobid=538,arraytaskid=9,SLURM_ARRAY_JOB_ID=529,JobState(529)=COMPLETED
epilog 15:42:55.733841379
jobid=535,arraytaskid=6,SLURM_ARRAY_JOB_ID=529,JobState(529)=COMPLETED
epilog 15:42:56.728246980
jobid=537,arraytaskid=8,SLURM_ARRAY_JOB_ID=529,JobState(529)=COMPLETED
prolog 15:43:27.518215088 jobid=541,arraytaskid=2,SLURM_ARRAY_JOB_ID=539
prolog 15:43:27.519780788 jobid=540,arraytaskid=1,SLURM_ARRAY_JOB_ID=539
prolog 15:43:27.520344497 jobid=542,arraytaskid=3,SLURM_ARRAY_JOB_ID=539
prolog 15:43:27.521069023 jobid=543,arraytaskid=4,SLURM_ARRAY_JOB_ID=539
prolog 15:43:27.523453937 jobid=544,arraytaskid=5,SLURM_ARRAY_JOB_ID=539
prolog 15:43:27.526033806 jobid=545,arraytaskid=6,SLURM_ARRAY_JOB_ID=539
prolog 15:43:27.527783240 jobid=546,arraytaskid=7,SLURM_ARRAY_JOB_ID=539
prolog 15:43:27.530369206 jobid=539,arraytaskid=10,SLURM_ARRAY_JOB_ID=539
prolog 15:43:27.534797886 jobid=547,arraytaskid=8,SLURM_ARRAY_JOB_ID=539
prolog 15:43:27.538816566 jobid=548,arraytaskid=9,SLURM_ARRAY_JOB_ID=539
epilog 15:43:29.758028002
jobid=544,arraytaskid=5,SLURM_ARRAY_JOB_ID=539,JobState(539)=RUNNING
epilog 15:43:30.831216132
jobid=540,arraytaskid=1,SLURM_ARRAY_JOB_ID=539,JobState(539)=RUNNING
epilog 15:43:30.833185667
jobid=546,arraytaskid=7,SLURM_ARRAY_JOB_ID=539,JobState(539)=RUNNING
epilog 15:43:31.799390789
jobid=541,arraytaskid=2,SLURM_ARRAY_JOB_ID=539,JobState(539)=RUNNING
epilog 15:43:31.801698014
jobid=543,arraytaskid=4,SLURM_ARRAY_JOB_ID=539,JobState(539)=RUNNING
epilog 15:43:34.837543145
jobid=545,arraytaskid=6,SLURM_ARRAY_JOB_ID=539,JobState(539)=COMPLETING
epilog 15:43:34.839544475
jobid=539,arraytaskid=10,SLURM_ARRAY_JOB_ID=539,JobState(539)=COMPLETING
epilog 15:43:35.667776053
jobid=542,arraytaskid=3,SLURM_ARRAY_JOB_ID=539,JobState(539)=COMPLETED
epilog 15:43:35.835100302
jobid=548,arraytaskid=9,SLURM_ARRAY_JOB_ID=539,JobState(539)=COMPLETED
epilog 15:43:37.827816869
jobid=547,arraytaskid=8,SLURM_ARRAY_JOB_ID=539,JobState(539)=COMPLETED

short of making every task in the array job create a file on completion and
testing for the existence of all files, is there a way to reliably detect
when all tasks in the array have completed?

I'm using a newly installed system based on slurm 18.08.8 as packaged in
the OpenHPC repos.

Thanks

Antony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20200130/077b8aa4/attachment.htm>