[slurm-users] How to debug a prolog script?

[Davide DelVento]

Finally I found some time available when I could do the job without
disrupting my users.

It turned out to be both the permissions issue as discussed here, and
the fact that the slurm.conf needs the fully qualified path of the
prolog script.

So that is solved, but sadly my problem is not solved as I will
describe in another thread.

On Sun, Sep 18, 2022 at 11:57 PM Bjørn-Helge Mevik
<b.h.mevik at usit.uio.no> wrote:
>
> Davide DelVento <davide.quantum at gmail.com> writes:
>
> >> I'm curious: What kind of disruption did it cause for your production
> >> jobs?
> >
> > All jobs failed and went in pending/held with "launch failed requeued
> > held" status, all nodes where the jobs were scheduled went draining.
> >
> > The logs only said "error: validate_node_specs: Prolog or job env
> > setup failure on node xxxx, draining the node". I guess if they said
> > "-bash: /path/to/prolog: Permission denied" I would have caught the
> > problem myself.
>
> But that is not a problem caused by having things like
>
> exec &> /root/prolog_slurmd.$$
>
> in the script, as you indicated.  It is a problem caused by the prolog
> script file not being executable.
>
> > In hindsight it is obvious, but I don't think even the documentation
> > mentions that, does it? After all you can execute a file with a
> > non-executable with with "sh filename", so I made the incorrect
> > assumption that slurm would have invoked the prolog that way.
>
> Slurm prologs can be written in any language - we used to have perl
> prolog scripts. :)
>
> --
> Regards,
> Bjørn-Helge Mevik, dr. scient,
> Department for Research Computing, University of Oslo
>