[slurm-users] remote license

Fri Sep 16 22:06:56 UTC 2022

Hi Brian,

>From your response, I speculate that my wording sounded harsh or
unrespectful. That was not my intention and therefore I sincerely
apologize for it.

In fact my perplexity is certainly due to my ignorance (as it must be
very clear by the number and "quality" of queries that I am posting on
this mailing list). It seemed to me that what is currently available
is a special edge case, whereas the simpler one is not covered, so I
was (perhaps still am) convinced that I must have misunderstood how
things work. Perhaps my "am I missing something?" sounded rhetorical
rather than sincere which was the spirit I wrote it with. Sorry about
that.

I find it a nice coincidence that you suggested paying SchedMD for
this, because after the clarifications which I am trying to get in
this thread, I thought to ask my management to do exactly that!!!

For the local license, are you suggesting to programmatically change
slurm.conf and reconfigure e.g. in a cron?

Thanks a lot for your help and have a great weekend

Davide

On Fri, Sep 16, 2022 at 11:52 AM Brian Andrus <toomuchit at gmail.com> wrote:
>
> Davide,
>
> I'll not engage on this. If you want a feature, pay SchedMD for support
> and they will prioritize it and work on it.  You are already using a
> very impressive bit of software for free.
>
> As far as local license updates, yes, you can do the local license and
> reconfigure regularly. Feel free to do that. It is not something that
> scales well, but it looks like you have a rather beginner cluster that
> would never be impacted by such choices.
>
> Brian Andrus
>
>
> On 9/16/2022 10:00 AM, Davide DelVento wrote:
> > Thanks Brian.
> >
> > I am still perplexed. What is a database to install, administer,
> > patch, update, could break, be down, etc buying us? I see limited use
> > cases, e.g. a license server which does not provide the license
> > count/use in a parsable way, and that someone wants to use with
> > multiple SLURM installations (if it's on a single one, the local
> > license is perfect). Wouldn't it much, much easier for everybody if
> > one could specify a script (the bullet 1. you mentioned) inside SLURM,
> > and use the license server ITSELF as the authoritative source of
> > license count? Sure, it won't be perfect, e.g. race conditions in
> > license acquisition can still cause failures, but the database won't
> > be fixing that....
> > I must be missing something....
> >
> > Alternatively, can one update the license count of local license with
> > a scontrol command, rather than changing the slurm.conf and
> > reconfigure? That could make what I say possible....
> >
> > Thanks
> >
> > On Fri, Sep 16, 2022 at 9:25 AM Brian Andrus <toomuchit at gmail.com> wrote:
> >> Davide,
> >>
> >> You have it pretty correct. While the database itself is not part of the
> >> slurm suite, slurmdbd (which would access the database) is.
> >>
> >> As far as writing something that keeps things updated, I'm sure many
> >> have done this. However, it would be unique to your installation. The
> >> specific number of licenses, naming them, what license server is being
> >> used, etc.
> >> All of that could easily be a few lines in a script that you have in a
> >> cron job or other trigger (eg prolog/epilog). You would just:
> >>
> >> 1) Read/parse current licenses/use (eg: if you are using flexlm, lmutil
> >> lmstat output)
> >> 2) Update the database (sacctmgr command)
> >>
> >> As you can see, that 1st step would be highly dependent on you and your
> >> environment. The 2nd step would be dependent on what things you are
> >> tracking within that.
> >>
> >> Brian Andrus
> >>
> >>
> >> On 9/16/2022 5:01 AM, Davide DelVento wrote:
> >>> So if I understand correctly, this "remote database" is something that
> >>> is neither part of slurm itself, nor part of the license server per
> >>> se, correct?
> >>>
> >>> Regarding the "if you got creative", has anybody on this list done
> >>> that already? I can't believe I'm the first one wanting this feature!
> >>> Matching the number in that database with the actual number the
> >>> license server knows would be extremely helpful! We use various
> >>> license servers (for various licensed software), so each one of them
> >>> would be useful. I can probably script/develop one of these myself,
> >>> but I am not sure I've got the time...
> >>>
> >>> Thanks!
> >>>
> >>> On Thu, Sep 15, 2022 at 6:04 PM Brian Andrus <toomuchit at gmail.com> wrote:
> >>>> So if you follow the links to: https://slurm.schedmd.com/licenses.html
> >>>> you should see the difference.
> >>>>
> >>>> Local licenses are just a counter that is setup in slurm.conf
> >>>> Remote liceneses are a counter in a database (the database is "remote"),
> >>>> so you can change/update it dynamically. So, you could change their
> >>>> allocation with a sacctmgr command. It is especially useful when you are
> >>>> managing multiple clusters that share licenses. You can allocate that a
> >>>> certain number are allowed by each cluster and change that if needed.
> >>>>
> >>>> If you got creative, you could keep the license count that is in the
> >>>> database updated to match the number free from flexlm to stop license
> >>>> starvation due to users outside slurm using them up so they really
> >>>> aren't available to slurm.
> >>>>
> >>>> Brian Andrus
> >>>>
> >>>>
> >>>> On 9/15/2022 3:34 PM, Davide DelVento wrote:
> >>>>> I am a bit confused by remote licenses.
> >>>>>
> >>>>> https://lists.schedmd.com/pipermail/slurm-users/2020-September/006049.html
> >>>>> (which is only 2 years old) claims that they are just a counter, so
> >>>>> like local licenses. Then why call them remote?
> >>>>>
> >>>>> Only a few days after, this
> >>>>> https://lists.schedmd.com/pipermail/slurm-users/2020-September/006081.html
> >>>>> appeared to imply (but not clearly stated) that the remote license are
> >>>>> not simply a counter, but then it's not clear how they are different.
> >>>>>
> >>>>> The current documentation (and attempts to run the "add resource"
> >>>>> command) says that one must use the license count, which seems to
> >>>>> imply they are just a simple counter (but then what do they need the
> >>>>> server for)?
> >>>>>
> >>>>> So what is what?
> >>>>>
> >>>>> In my cursory past experience with this, it seemed that it were
> >>>>> possible to query a license server (at least some of them) to get the
> >>>>> actual number of available licenses and schedule (or let jobs pending)
> >>>>> accordingly. Which would be very helpful for the not-too-uncommon
> >>>>> situation in which the same license server provides licenses for both
> >>>>> the HPC cluster and other non-slurm-controlled resources, such a
> >>>>> user's workstations. Was that impression wrong, or perhaps somebody
> >>>>> scripted it in some way? If the latter, does anybody know if those
> >>>>> scripts are publicly available anywhere?
> >>>>>
> >>>>> Thanks
> >>>>>
>