Large overhead for --task-epilog
Hi all, I've been experimenting with task prolog/epilog and noticed that task epilogs have significant overhead. The overhead seems to be independent of the epilog script content: $ cat test.sh #!/bin/bash $ time srun --overlap --task-prolog=$PWD/test.sh --ntasks=64 sleep 1 real 0m1.730s user 0m0.211s sys 0m0.009s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=2 sleep 1 real 0m3.530s user 0m0.213s sys 0m0.007s $ time srun --overlap --task-prolog=$PWD/test.sh --task-epilog=$PWD/test.sh --ntasks=64 sleep 1 real 1m5.936s user 0m0.216s sys 0m0.005s This test was on one node, with 64 tasks in the allocation. The overhead seems linked only to number of tasks per node, as adding additional nodes did not increase the overhead. I saw this on $ srun --version slurm 23.11.10-BullSequana.1.2.1 and someone else reproduced similar results with |slurm 25.05.7|. I can live with the overhead, but I am curious as to why it happens and to understand the design of the implementation. Does the task epilog script only run when all tasks finish, or does it run immediately at exit for each task? Are the calls serialised? My use case is a fusermount that I want to mount in userspace at task start and clean up at task finish, one task per node is responsible for the mount/unmount and I want to understand if I have a race condition for the unmount. Any advice very much appreciated, Alan
participants (1)
-
Alan O'Cais