[slurm-users] run bash script in spank plugin

[Yair Yarom]

I'm also in favor of epilog scripts, though it really depends on what
you are eventually trying to achieve.

Also, I'm not sure I understand what you meant by the slurm job
sleeping for 6 seconds and rebooting. You did want it to reboot, no?
The 4 "missing" seconds might be the time difference between the spank
and the job starting times.

If you still require the spank plugin and it doesn't work, I can
suggest trying to remove the '&' from the exec, and the wait() from
the end, though I'm not sure it'll matter. You could monitor the node
directly and see if the spank plugin script is inside a slurm
controlled cgroup. In which case, you'd have to move it out of it.


On Mon, Jun 4, 2018 at 6:37 PM, Brian Andrus <toomuchit at gmail.com> wrote:
> Seems like there are better approaches.
>
> In this situation, I would use an epilogue script and give sudo access to
> the script. Check out https://slurm.schedmd.com/prolog_epilog.html
>
> That would likely be much easier and fit into the methodology slurm uses.
>
> Brian Andrus
> Firstspot, Inc.
>
>
> On 6/4/2018 8:11 AM, Tueur Volvo wrote:
>
> I would like to run a bash script or binary executable as root (even if the
> user who started the job doesn't have root rights) at the end of a job if I
> put an option in my spank plugin
>
> 2018-06-04 16:36 GMT+02:00 John Hearns <hearnsj at googlemail.com>:
>>
>> That kinnddd....  of...  defeats...  the.... purpose....  of a job
>> scheduler.
>> I am very sure that you know why you need this and you have a good reason
>> for doing it.  Over to others on the list, sorry.
>>
>> On 4 June 2018 at 16:15, Tueur Volvo <huitre39 at gmail.com> wrote:
>>>
>>> no I don't have dependency treated.
>>>
>>> during the job, I would like to run a program on the machine running the
>>> job
>>> but I'd like the program to keep running even after the job ends.
>>>
>>> 2018-06-04 15:30 GMT+02:00 John Hearns <hearnsj at googlemail.com>:
>>>>
>>>> Tueur what are you trying to achieve here?  The example you give is
>>>> touch /tmp/newfile.txt'
>>>> I think you are trying to send a signal to another process. Could this
>>>> be 'Hey - the job has finished and there is a new file for you to process'
>>>> If that is so, there may be better ways to do this. If you have a
>>>> post-processing step, then you can submit a job whihc depends on the main
>>>> job.
>>>> https://hpc.nih.gov/docs/job_dependencies.html
>>>>
>>>> On 4 June 2018 at 15:20, Tueur Volvo <huitre39 at gmail.com> wrote:
>>>>>
>>>>> thanks for your answer, i try some solution but it's not work
>>>>>
>>>>> i try to add setsid and setpgrp for isolate my new process but slurm
>>>>> job sleep 6secondes and reboot my machine (i test with reboot command, but
>>>>> we can make other bash command, it's just example)
>>>>>
>>>>>         pid_t cpid; //process id's and process groups
>>>>>
>>>>>         cpid = fork();
>>>>>
>>>>>         if( cpid == 0 ){
>>>>>             setsid();
>>>>>             setpgrp();
>>>>>             execl("/bin/sh", "sh", "-c", "sleep 10; reboot1&", NULL);
>>>>>
>>>>>         }
>>>>>         wait(NULL);
>>>>>
>>>>>
>>>>> maybe i have a error in my code ?
>>>>>
>>>>> 2018-05-31 9:37 GMT+02:00 Yair Yarom <irush at cs.huji.ac.il>:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm not sure how slurm/spank handles child processes but this might be
>>>>>> intentional. So there might be some issues if this were to work.
>>>>>>
>>>>>> You can try instead of calling system(), to use fork() + exec(). If
>>>>>> that still doesn't work, try calling setsid() before the exec(). I can
>>>>>> think of situations where your process might still get killed, e.g. if
>>>>>> slurm (or even systemd) kills all subprocesses of the "job", by
>>>>>> looking at the cgroup. If that's the case, you'll need to move it to
>>>>>> another cgroup in addition/instead of setsid().
>>>>>>
>>>>>>     Yair.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, May 30, 2018 at 5:16 PM, Tueur Volvo <huitre39 at gmail.com>
>>>>>> wrote:
>>>>>> > Hello i have question, how run in background bash script in spank
>>>>>> > plugin ?
>>>>>> >
>>>>>> > in my spank plugin in function : slurm_spank_task_init_privileged
>>>>>> >
>>>>>> > i want to run this script :
>>>>>> >
>>>>>> > system("nohup bash -c 'sleep 10 ; touch /tmp/newfile.txt' &");
>>>>>> >
>>>>>> > i want to run in independant process this bash script, i don't want
>>>>>> > wait 10
>>>>>> > seconde in my slurm plugin
>>>>>> >
>>>>>> > i have this code :
>>>>>> > int slurm_spank_task_init_privileged (spank_t sp, int ac, char **av)
>>>>>> > {
>>>>>> >
>>>>>> > system("nohup bash -c 'sleep 10 ; touch /tmp/newfile.txt' &");
>>>>>> >
>>>>>> > return 0;
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > actualy it's not work, when slurm ending to run my job, he kill my
>>>>>> > nohup
>>>>>> > command
>>>>>> >
>>>>>> > if i  had in my c code sleep 12, my bash script work
>>>>>> >
>>>>>> >
>>>>>> > int slurm_spank_task_init_privileged (spank_t sp, int ac, char **av)
>>>>>> > {
>>>>>> >
>>>>>> > system("nohup bash -c 'sleep 10 ; touch /tmp/newfile.txt' &");
>>>>>> >
>>>>>> > sleep(12);
>>>>>> >
>>>>>> > return 0;
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > but i don't want to wait, i want to run my bash script in
>>>>>> > independant
>>>>>> > process
>>>>>> >
>>>>>> > thanks for advance for your help
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>>
>>>
>>
>
>