<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Folks,
<div class=""><br class="">
</div>
<div class="">I am trying to figure out how to advise users on starting worker daemons in their allocations using
<font face="Courier" class="">srun</font>. That is, I want to be able to run “<font face="Courier" class="">srun foo</font>”, where
<font face="Courier" class="">foo</font> starts some child process and then exits, and the child process(es) persist and wait for work.</div>
<div class=""><br class="">
</div>
<div class="">Use cases for this include Apache Spark and FUSE mounts. In general, it seems that there are a number of newer computing frameworks that have this model, in particular for the data science space.</div>
<div class=""><br class="">
</div>
<div class="">We are on Slurm 17.02.10 with the <font face="Courier" class="">proctrack/cgroup</font> plugin.</div>
<div class=""><br class="">
</div>
<div class="">I’m using a Python script <font face="Courier" class="">foo.py</font> to test this (included at end of e-mail). After forking, the parent exits immediately, and the child writes the numbers 1 to 10 at one-second intervals to
<font face="Courier" class="">/tmp/foo</font>, then the word “done”, and then exits.</div>
<div class=""><br class="">
</div>
<div class="">Desired behavior in a one-node allocation:</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;" class="">
<div class=""><font face="Courier" class="">$ srun ./foo.py && sleep 12 && cat /tmp/foo</font></div>
<div class=""><font face="Courier" class="">starting cn001.localdomain 79615</font></div>
<div class=""><font face="Courier" class="">0</font></div>
<div class=""><font face="Courier" class="">1</font></div>
<div class=""><font face="Courier" class="">2</font></div>
<div class=""><font face="Courier" class="">3</font></div>
<div class=""><font face="Courier" class="">4</font></div>
<div class=""><font face="Courier" class="">5</font></div>
<div class=""><font face="Courier" class="">6</font></div>
<div class=""><font face="Courier" class="">7</font></div>
<div class=""><font face="Courier" class="">8</font></div>
<div class=""><font face="Courier" class="">9</font></div>
<div class=""><font face="Courier" class="">10</font></div>
<div class=""><font face="Courier" class="">done</font></div>
</blockquote>
</div>
<div class=""><br class="">
</div>
<div class="">Actual behavior:</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class="">
<div class="">
<div class=""><font face="Courier" class="">$ srun ./foo.py && sleep 12 && cat /tmp/foo</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">starting cn001.localdomain 79615</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">0</font></div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">As far as I can tell, what is going on is that when <font face="Courier" class="">
foo.py</font> exits, Slurm concludes that the job step is over and kills the child; see debug log at end of e-mail.</div>
<div class=""><br class="">
</div>
<div class="">I have considered the following:</div>
<div class=""><br class="">
</div>
<div class="">(1) Various command line options, all of which have no effect on this:
<font face="Courier" class="">--kill-on-bad-exit=0</font>, <font face="Courier" class="">
--no-kill</font>, <font face="Courier" class="">--mpi-none</font>, <font face="Courier" class="">
--overcommit</font>, <font face="Courier" class="">--oversubscribe</font>, <font face="Courier" class="">
--wait=0</font>.</div>
<div class=""><br class="">
</div>
<div class="">(2) <font face="Courier" class="">srun --task-prolog=./foo.py true</font></div>
<div class=""><br class="">
</div>
<div class="">Instead of killing <font face="Courier" class="">foo.py</font>’s child, this invocation waits for it to exit. Also, this seems to require a single executable rather than a command line.</div>
<div class=""><br class="">
</div>
<div class="">One can work around the waiting to exit by putting the entire command in the background, but then subsequent
<font face="Courier" class="">srun</font>s wait until the child completes anyway (with the warning “<font face="Courier" class="">Job step creation temporarily disabled, retrying</font>”). --overcommit on the 1st, 2nd, or both
<font face="Courier" class="">srun</font>s has no effect.</div>
<div class=""><br class="">
</div>
<div class="">Recall that for real-world tasks, the child will run indefinitely waiting for work, so we can’t wait for it to finish.</div>
<div class=""><br class="">
</div>
<div class="">(3) <font face="Courier" class="">srun sh -c './foo.py && sleep 15'</font> : same behavior as item 2.</div>
<div class=""><br class="">
</div>
<div class="">(4) Teach Slurm how to deal with the worker daemons somehow.</div>
<div class=""><br class="">
</div>
<div class="">This doesn’t generalize. We want users to be able to bring whatever compute framework they want, without waiting for Slurm support, so they can innovate faster.</div>
<div class=""><br class="">
</div>
<div class="">(5) Put the worker daemons in their own job. For example, one could start the Spark worker daemons in one job, with the Spark coordinator daemon and user work submission in a second one-node job.</div>
<div class=""><br class="">
</div>
<div class="">This doesn’t solve the general use case. For example, in the case of Spark, I’ve a large test suite where starting and stopping a Spark cluster is only one of many tests. For FUSE, which depends on a worker daemon to implement filesystem operations,
the mount is there to serve the needs of the rest of the job script.</div>
<div class=""><br class="">
</div>
<div class="">(6) Change the software to not daemonize. For example, one can start Spark by invoking the .jar files directly, bypassing the daemonizing start script, or in newer versions by setting
<font face="Courier" class="">SPARK_NO_DAEMONIZE=1</font>.</div>
<div class=""><br class="">
</div>
<div class="">This again doesn’t generalize. I need to be able to support imperfect scientific software as it arrives, without hacking or framework-specific workarounds.</div>
<div class=""><br class="">
</div>
<div class="">(7) Don’t launch with <font face="Courier" class="">srun</font>. For example,
<font face="Courier" class="">pdsh</font> can interpret Slurm environment variables and uses SSH to launch tasks on my allocated nodes.</div>
<div class=""><br class="">
</div>
<div class="">This works, and is what I’m doing currently, but it doesn’t scale. One or two dozen SSH processes on the first node of my allocation are fine, but 1000 or 10,000 aren’t. Also, it’s a kludge since
<font face="Courier" class="">srun</font> is specifically provided and optimized to launch jobs in a Slurm cluster.</div>
<div class=""><br class="">
</div>
<div class=""><i class="">My question:</i> Is there any way I can convince Slurm to let a job step’s children keep running beyond the end of the step, and kill them at the end of the job if needed. Or, less preferably, overlap job steps?</div>
<div class=""><br class="">
</div>
<div class="">Much appreciated,</div>
<div class="">Reid</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Appendix 1: <font face="Courier" class="">foo.py</font></div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class="">
<div class="">
<div class=""><font face="Courier" class="">#!/usr/bin/env python3</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""><br class="">
</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""># Try to find a way to run daemons under srun.</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""><br class="">
</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">import os</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">import socket</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">import sys</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">import time</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""><br class="">
</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">print("starting %s %d" % (socket.gethostname(), os.getpid()))</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""><br class="">
</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""># one fork is enough to get killed by Slurm</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">if (os.fork() > 0): sys.exit(0)</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""><br class="">
</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">fp = open("/tmp/foo", "w")</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""><br class="">
</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">fp.truncate()</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">for i in range(10):</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""> fp.write("%d\n" % i)</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""> fp.flush()</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""> time.sleep(1)</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class=""><br class="">
</font></div>
</div>
<div class="">
<div class=""><font face="Courier" class="">fp.write("done\n")</font></div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Appendix 2: error log showing job step cleanup removes the worker daemon</div>
<div class=""><br class="">
</div>
<blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class="">
<div class=""><font face="Courier" class="">slurmstepd: debug level = 6</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: IO handler started pid=62147</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: mpi/pmi2: _tree_listen_readable</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: mpi/pmi2: _task_readable</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: Using gid list sent by slurmd</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: mpi/pmi2: _tree_listen_readable</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: mpi/pmi2: _task_readable</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: mpi/pmi2: _tree_listen_readable</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: mpi/pmi2: _task_readable</font></div>
<div class=""><font face="Courier" class="">slurmstepd: starting 1 tasks</font></div>
<div class=""><font face="Courier" class="">slurmstepd: task 0 (62153) started 2018-08-27T11:03:33</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: Using gid list sent by slurmd</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: mpi/pmi2: _tree_listen_readable</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: mpi/pmi2: _task_readable</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: mpi/pmi2: _tree_listen_readable</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: mpi/pmi2: _task_readable</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: adding task 0 pid 62153 on node 0 to jobacct</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: jobacct_gather_cgroup_cpuacct_attach_task: jobid 206670 stepid 62 taskid 0 max_task_id 0</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: xcgroup_instantiate: cgroup '/sys/fs/cgroup/cpuacct/slurm' already exists</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: xcgroup_instantiate: cgroup '/sys/fs/cgroup/cpuacct/slurm/uid_1001' already exists</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: xcgroup_instantiate: cgroup '/sys/fs/cgroup/cpuacct/slurm/uid_1001/job_206670' already exists</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: jobacct_gather_cgroup_memory_attach_task: jobid 206670 stepid 62 taskid 0 max_task_id 0</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: xcgroup_instantiate: cgroup '/sys/fs/cgroup/memory/slurm' already exists</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: xcgroup_instantiate: cgroup '/sys/fs/cgroup/memory/slurm/uid_1001' already exists</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: xcgroup_instantiate: cgroup '/sys/fs/cgroup/memory/slurm/uid_1001/job_206670' already exists</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: jag_common_poll_data: 62153 mem size 0 290852 time 0.000000(0+0)</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: _get_sys_interface_freq_line: filename = /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_cur_freq</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: cpu 1 freq= 2101000</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: jag_common_poll_data: Task average frequency = 2101000 pid 62153 mem size 0 290852 time 0.000000(0+0)</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: energycounted = 0</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: getjoules_task energy = 0</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: Step 206670.62 memory used:0 limit:251658240 KB</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: Reading cgroup.conf file /etc/slurm/cgroup.conf</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_load: unable to get cgroup '/sys/fs/cgroup/cpuset' entry '/sys/fs/cgroup/cpuset/slurm/system' properties: No such file or directory</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_load: unable to get cgroup '/sys/fs/cgroup/memory' entry '/sys/fs/cgroup/memory/slurm/system' properties: No such file or directory</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: Sending launch resp rc=0</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: mpi type = (null)</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: [job 206670] attempting to run slurm task_prolog [/opt/slurm/task_prolog]</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: Handling REQUEST_STEP_UID</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: Handling REQUEST_SIGNAL_CONTAINER</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: _handle_signal_container for step=206670.62 uid=0 signal=995</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: _set_limit: conf setrlimit RLIMIT_CPU no change in value: 18446744073709551615</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: _set_limit: conf setrlimit RLIMIT_FSIZE no change in value: 18446744073709551615</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: _set_limit: conf setrlimit RLIMIT_DATA no change in value: 18446744073709551615</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: _set_limit: conf setrlimit RLIMIT_STACK no change in value: 18446744073709551615</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: _set_limit: conf setrlimit RLIMIT_CORE no change in value: 16384</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: _set_limit: RLIMIT_RSS : max:inf cur:inf req:257698037760</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: _set_limit: conf setrlimit RLIMIT_RSS succeeded</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: _set_limit: conf setrlimit RLIMIT_NPROC no change in value: 8192</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: _set_limit: conf setrlimit RLIMIT_NOFILE no change in value: 65536</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: Couldn't find SLURM_RLIMIT_MEMLOCK in environment</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: _set_limit: conf setrlimit RLIMIT_AS no change in value: 18446744073709551615</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: Set task rss(245760 MB)</font></div>
<div class=""><font face="Courier" class="">starting fg001.localdomain 62153</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: Step 206670.62 memory used:0 limit:251658240 KB</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: removing task 0 pid 62153 from jobacct</font></div>
<div class=""><font face="Courier" class="">slurmstepd: task 0 (62153) exited with exit code 0.</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: [job 206670] attempting to run slurm task_epilog [/opt/slurm/task_epilog]</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: Using gid list sent by slurmd</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_delete: rmdir(/sys/fs/cgroup/cpuacct/slurm/uid_1001/job_206670/step_62/task_0): Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: jobacct_gather_cgroup_cpuacct_fini: failed to delete /sys/fs/cgroup/cpuacct/slurm/uid_1001/job_206670/step_62/task_0 Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_delete: rmdir(/sys/fs/cgroup/cpuacct/slurm/uid_1001/job_206670/step_62): Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: jobacct_gather_cgroup_cpuacct_fini: failed to delete /sys/fs/cgroup/cpuacct Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_delete: rmdir(/sys/fs/cgroup/cpuacct/slurm/uid_1001/job_206670): Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: jobacct_gather_cgroup_cpuacct_fini: failed to delete /sys/fs/cgroup/cpuacct/slurm/uid_1001/job_206670 Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_delete: rmdir(/sys/fs/cgroup/cpuacct/slurm/uid_1001): Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: jobacct_gather_cgroup_cpuacct_fini: failed to delete /sys/fs/cgroup/cpuacct/slurm/uid_1001 Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_delete: rmdir(/sys/fs/cgroup/memory/slurm/uid_1001/job_206670/step_62/task_0): Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: jobacct_gather_cgroup_memory_fini: failed to delete /sys/fs/cgroup/memory/slurm/uid_1001/job_206670/step_62/task_0 Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_delete: rmdir(/sys/fs/cgroup/memory/slurm/uid_1001/job_206670/step_62): Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: jobacct_gather_cgroup_memory_fini: failed to delete /sys/fs/cgroup/memory/slurm/uid_1001/job_206670/step_62 Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_delete: rmdir(/sys/fs/cgroup/memory/slurm/uid_1001/job_206670): Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: jobacct_gather_cgroup_memory_fini: failed to delete /sys/fs/cgroup/memory/slurm/uid_1001/job_206670 Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_delete: rmdir(/sys/fs/cgroup/memory/slurm/uid_1001): Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: jobacct_gather_cgroup_memory_fini: failed to delete /sys/fs/cgroup/memory/slurm/uid_1001 Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: step_terminate_monitor will run for 60 secs</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: killing process 62158 (inherited_task) with signal 9</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_delete: rmdir(/sys/fs/cgroup/freezer/slurm/uid_1001/job_206670/step_62): Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: _slurm_cgroup_destroy: problem deleting step cgroup path /sys/fs/cgroup/freezer/slurm/uid_1001/job_206670/step_62: Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: killing process 62158 (inherited_task) with signal 9</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_delete: rmdir(/sys/fs/cgroup/freezer/slurm/uid_1001/job_206670): Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: xcgroup_delete: rmdir(/sys/fs/cgroup/freezer/slurm/uid_1001): Device or resource busy</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: step_terminate_monitor_stop signalling condition</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: step_terminate_monitor is stopping</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug2: Sending SIGKILL to pgid 62147</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: Waiting for IO</font></div>
<div class=""><font face="Courier" class="">slurmstepd: debug: Closing debug channel</font></div>
</blockquote>
<div class=""><br class="">
</div>
</body>
</html>