[slurm-users] License management and invoking scontrol in the prolog

[Davide DelVento]

No, I never moved that file from the Linux cluster (and I do the
editing with VIM which warns me of that possible issue).

On Wed, Sep 7, 2022 at 11:19 AM Brian Andrus <toomuchit at gmail.com> wrote:
>
> Possibly way off base, but did you happen to do any of the editing in
> Windows? Maybe running into the cr/lf issue for how windows saves text
> files?
>
> Brian Andrus
>
> On 9/7/2022 5:21 AM, Davide DelVento wrote:
> > Thanks Ole, your wiki page sheds some light on this mystery.
> > Very frustrating that even the simple example provided in the release
> > fails, and it fails at the most basic logging functionality.
> >
> > Note that "my" job_submit.lua is now the unmodified, slurm-provided
> > one.... and that the luac command returns nothing in my case (this is
> > Lua 5.3.4) so syntax seems correct?
> >
> > Yet the logs report the problem I mentioned rather than the actual
> > content that the plugin is attempting to log.
> >
> > On Wed, Sep 7, 2022 at 2:13 AM Ole Holm Nielsen
> > <Ole.H.Nielsen at fysik.dtu.dk> wrote:
> >> Hi Davide,
> >>
> >> I suggest that you check your job_submit.lua script with the LUA compiler:
> >>
> >> luac -p /etc/slurm/job_submit.lua
> >>
> >> I have written some more details in my Wiki page
> >> https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#job-submit-plugins
> >>
> >> Best regards,
> >> Ole
> >>
> >> On 9/7/22 01:51, Davide DelVento wrote:
> >>> Thanks again to both of you.
> >>>
> >>> I actually did not build Slurm myself, otherwise I'd keep extensive
> >>> logs of what I did. Other people did, so I don't know. However, I get
> >>> the same grep'ing results as yours.
> >>>
> >>> Looking at the logs reveals some info, but it's cryptic.
> >>>
> >>> [2022-09-06T17:33:56.513] debug3: job_submit/lua:
> >>> slurm_lua_loadscript: skipping loading Lua script:
> >>> /opt/slurm/job_submit.lua
> >>> [2022-09-06T17:33:56.513] error: job_submit/lua:
> >>> /opt/slurm/job_submit.lua: [string "slurm.user_msg
> >>> (string.format(table.unpack({...})))"]:1: bad argument #2 to 'format'
> >>> (no value)
> >>>
> >>> As you can see, there is no line number and there is nothing like
> >>> user_msg in this code. There is indeed an "unpack" which is used in
> >>> the SchedMD-defined logging helper function which has a comment
> >>> "Implicit definition of arg was removed in Lua 5.2" and that's where I
> >>> speculate the error occurs.
> >>>
> >>> I should stress, this is with their own example, not my code. I guess
> >>> I could forgo the logging and move forward, but that won't probably
> >>> lead me very far.
> >>>
> >>> I am contemplating submitting a github issue about it? I did check
> >>> that the version of the job_submit.lua I have is the same currently in
> >>> the repo at https://github.com/SchedMD/slurm/blob/master/etc/job_submit.lua.example
> >>>
> >>> On Thu, Sep 1, 2022 at 11:55 PM Ole Holm Nielsen
> >>> <Ole.H.Nielsen at fysik.dtu.dk> wrote:
> >>>> Did you install all prerequiste packages (including lua) on the server
> >>>> where you built the Slurm packages?
> >>>>
> >>>> On my system I get:
> >>>>
> >>>> $ strings `which slurmctld ` | grep HAVE_LUA
> >>>> HAVE_LUA 1
> >>>>
> >>>> /Ole
> >>>>
> >>>> https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#install-prerequisites
> >>>>
> >>>> On 9/2/22 05:15, Davide DelVento wrote:
> >>>>> Thanks.
> >>>>>
> >>>>> I did try a lua script as soon as I got your first email, but that
> >>>>> never worked (yes, I enabled it in slurm.conf and ran "scontrol
> >>>>> reconfigure" after). Slurm simply acted as if there was no job_submit script.
> >>>>>
> >>>>> After various tests, all unsuccessful, today I found that link which I
> >>>>> mentioned saying that lua might not be compiled in, hence all my most
> >>>>> recent messages of this thread.
> >>>>>
> >>>>> That file is indeed there, so that's good news that I don't need to recompile.
> >>>>> However I'm puzzled on what might be missing...
> >>>>>
> >>>>>
> >>>>> On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus <toomuchit at gmail.com> wrote:
> >>>>>> lua is the language you can use with the job_submit plugin.
> >>>>>>
> >>>>>> I was showing a quick way to see that job_submit capability is indeed in
> >>>>>> there.
> >>>>>>
> >>>>>> You can see if lua support is there by looking for the job_submit_lua.so
> >>>>>> file is there.
> >>>>>> It would be part of the slurm rpm (not the slurm-slurmctl rpm)
> >>>>>>
> >>>>>> Usually it would be found at /usr/lib64/slurm/job_submit_lua.so
> >>>>>>
> >>>>>> If that is there, you should be good with trying out a job_submit lua
> >>>>>> script.
> >>>>>>
> >>>>>> Brian Andrus
> >>>>>>
> >>>>>> On 9/1/2022 1:24 PM, Davide DelVento wrote:
> >>>>>>> Thanks again, Brian, indeed that grep returns many hits, but none of
> >>>>>>> them includes lua, i.e.
> >>>>>>>
> >>>>>>>      strings `which slurmctld ` | grep -i job_submit | grep -i lua
> >>>>>>>
> >>>>>>> returns nothing. So I should use the C rather than the more convenient
> >>>>>>> lua interface, unless I recompile or am I missing something?
> >>>>>>>
> >>>>>>> On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus <toomuchit at gmail.com> wrote:
> >>>>>>>> I would be surprised if it were compiled without the support. However,
> >>>>>>>> you could check and run something like:
> >>>>>>>>
> >>>>>>>> strings /sbin/slurmctld | grep job_submit
> >>>>>>>>
> >>>>>>>> (or where ever your slurmctld binary is). There should be quite a few
> >>>>>>>> lines with that in it.
> >>>>>>>>
> >>>>>>>> Brian Andrus
> >>>>>>>>
> >>>>>>>> On 9/1/2022 10:54 AM, Davide DelVento wrote:
> >>>>>>>>> Thanks Brian for the suggestion, which I am now exploring.
> >>>>>>>>>
> >>>>>>>>> The documentation is a bit cryptic for me, but exploring a few things
> >>>>>>>>> and checking https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/
> >>>>>>>>> I suspect my slurm install (provided by cluster vendor) was not
> >>>>>>>>> compiled with the lua plugin installed. Do you know how to verify if
> >>>>>>>>> that is the case or if it's something else? I don't see a way to show
> >>>>>>>>> if the plugin is actually being "seen" by slurm, and I suspect it's
> >>>>>>>>> not.
> >>>>>>>>>
> >>>>>>>>> Does anyone else have other suggestions or comment on either the
> >>>>>>>>> plugin or the prolog workaround?
> >>>>>>>>>
> >>>>>>>>> Thanks!
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus <toomuchit at gmail.com> wrote:
> >>>>>>>>>> Not sure if you can do all the things you intend, but the job_submit
> >>>>>>>>>> script is precisely where you want to check submission options.
> >>>>>>>>>>
> >>>>>>>>>> https://slurm.schedmd.com/job_submit_plugins.html
> >>>>>>>>>>
> >>>>>>>>>> Brian Andrus
> >>>>>>>>>>
> >>>>>>>>>> On 8/30/2022 12:58 PM, Davide DelVento wrote:
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> I would like to soft-enforce license utilization only when the -L is
> >>>>>>>>>>> set. My idea: check in the prolog if the license was requested and
> >>>>>>>>>>> only if it were, set the environmental variables needed for the
> >>>>>>>>>>> license.
> >>>>>>>>>>>
> >>>>>>>>>>> I looked at all environmental variables set by slurm and did not find
> >>>>>>>>>>> any related to the license as I was hoping.
> >>>>>>>>>>>
> >>>>>>>>>>> As a workaround, I could check
> >>>>>>>>>>>
> >>>>>>>>>>> scontrol show job $SLURM_JOB_ID | grep License
> >>>>>>>>>>>
> >>>>>>>>>>> and that would work, but (as discussed in other messages in this list)
> >>>>>>>>>>> the documentation at https://slurm.schedmd.com/prolog_epilog.html say
> >>>>>>>>>>>
> >>>>>>>>>>>> Prolog and Epilog scripts should be designed to be as short as possible
> >>>>>>>>>>>> and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr,
> >>>>>>>>>>>> etc). [...] Slurm commands in these scripts can potentially lead to performance
> >>>>>>>>>>>> issues and should not be used.
> >>>>>>>>>>> This is a bit of a concern, since the prolog would be invoked for
> >>>>>>>>>>> every job on the cluster, and it's a prolog (rather than the epilogue
> >>>>>>>>>>> like discussed in earlier messages).
> >>>>>>>>>>>
> >>>>>>>>>>> So two questions:
> >>>>>>>>>>>
> >>>>>>>>>>> 1) is there a better workaround to check in the prolog if the current
> >>>>>>>>>>> job requested a license and/or
> >>>>>>>>>>> 2) would this kind of use of scontrol be okay or is indeed a concern
> >>>>>>>>>>>
>