[slurm-users] remote license

Fri Sep 16 22:45:36 UTC 2022

Davide,

No worries. I like to see folks engage SchedMD, especially if they are 
seeing value in all the work that has gone into creating the software 
that runs the majority of fastest clusters on the planet.

To your question: yes, you can update slurm.conf and do 'scontrol 
reconfigure' to change the number of local licenses available. You could 
do that fairly regularly. I am not sure what it would do if you lowered 
the number to something smaller than the total in use. It should allow 
jobs to continue and drain down.

It is a much simpler way to achieve your goal, however, it affects the 
system more to run 'scontrol reconfigure' than it does to use the remote 
license method.
For a large scale system, that impact is very noticeable. For a small 
cluster, it is negligible. Thus, it may well be the best method for you 
and your environment. If you find yourself working on a large cluster 
sometime in your career, I would not recommend using it there.

Brian Andrus

On 9/16/2022 3:06 PM, Davide DelVento wrote:
> Hi Brian,
>
>  From your response, I speculate that my wording sounded harsh or
> unrespectful. That was not my intention and therefore I sincerely
> apologize for it.
>
> In fact my perplexity is certainly due to my ignorance (as it must be
> very clear by the number and "quality" of queries that I am posting on
> this mailing list). It seemed to me that what is currently available
> is a special edge case, whereas the simpler one is not covered, so I
> was (perhaps still am) convinced that I must have misunderstood how
> things work. Perhaps my "am I missing something?" sounded rhetorical
> rather than sincere which was the spirit I wrote it with. Sorry about
> that.
>
> I find it a nice coincidence that you suggested paying SchedMD for
> this, because after the clarifications which I am trying to get in
> this thread, I thought to ask my management to do exactly that!!!
>
> For the local license, are you suggesting to programmatically change
> slurm.conf and reconfigure e.g. in a cron?
>
> Thanks a lot for your help and have a great weekend
>
> Davide
>
>
>
>
>
>
> On Fri, Sep 16, 2022 at 11:52 AM Brian Andrus <toomuchit at gmail.com> wrote:
>> Davide,
>>
>> I'll not engage on this. If you want a feature, pay SchedMD for support
>> and they will prioritize it and work on it.  You are already using a
>> very impressive bit of software for free.
>>
>> As far as local license updates, yes, you can do the local license and
>> reconfigure regularly. Feel free to do that. It is not something that
>> scales well, but it looks like you have a rather beginner cluster that
>> would never be impacted by such choices.
>>
>> Brian Andrus
>>
>>
>> On 9/16/2022 10:00 AM, Davide DelVento wrote:
>>> Thanks Brian.
>>>
>>> I am still perplexed. What is a database to install, administer,
>>> patch, update, could break, be down, etc buying us? I see limited use
>>> cases, e.g. a license server which does not provide the license
>>> count/use in a parsable way, and that someone wants to use with
>>> multiple SLURM installations (if it's on a single one, the local
>>> license is perfect). Wouldn't it much, much easier for everybody if
>>> one could specify a script (the bullet 1. you mentioned) inside SLURM,
>>> and use the license server ITSELF as the authoritative source of
>>> license count? Sure, it won't be perfect, e.g. race conditions in
>>> license acquisition can still cause failures, but the database won't
>>> be fixing that....
>>> I must be missing something....
>>>
>>> Alternatively, can one update the license count of local license with
>>> a scontrol command, rather than changing the slurm.conf and
>>> reconfigure? That could make what I say possible....
>>>
>>> Thanks
>>>
>>> On Fri, Sep 16, 2022 at 9:25 AM Brian Andrus <toomuchit at gmail.com> wrote:
>>>> Davide,
>>>>
>>>> You have it pretty correct. While the database itself is not part of the
>>>> slurm suite, slurmdbd (which would access the database) is.
>>>>
>>>> As far as writing something that keeps things updated, I'm sure many
>>>> have done this. However, it would be unique to your installation. The
>>>> specific number of licenses, naming them, what license server is being
>>>> used, etc.
>>>> All of that could easily be a few lines in a script that you have in a
>>>> cron job or other trigger (eg prolog/epilog). You would just:
>>>>
>>>> 1) Read/parse current licenses/use (eg: if you are using flexlm, lmutil
>>>> lmstat output)
>>>> 2) Update the database (sacctmgr command)
>>>>
>>>> As you can see, that 1st step would be highly dependent on you and your
>>>> environment. The 2nd step would be dependent on what things you are
>>>> tracking within that.
>>>>
>>>> Brian Andrus
>>>>
>>>>
>>>> On 9/16/2022 5:01 AM, Davide DelVento wrote:
>>>>> So if I understand correctly, this "remote database" is something that
>>>>> is neither part of slurm itself, nor part of the license server per
>>>>> se, correct?
>>>>>
>>>>> Regarding the "if you got creative", has anybody on this list done
>>>>> that already? I can't believe I'm the first one wanting this feature!
>>>>> Matching the number in that database with the actual number the
>>>>> license server knows would be extremely helpful! We use various
>>>>> license servers (for various licensed software), so each one of them
>>>>> would be useful. I can probably script/develop one of these myself,
>>>>> but I am not sure I've got the time...
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Thu, Sep 15, 2022 at 6:04 PM Brian Andrus <toomuchit at gmail.com> wrote:
>>>>>> So if you follow the links to: https://slurm.schedmd.com/licenses.html
>>>>>> you should see the difference.
>>>>>>
>>>>>> Local licenses are just a counter that is setup in slurm.conf
>>>>>> Remote liceneses are a counter in a database (the database is "remote"),
>>>>>> so you can change/update it dynamically. So, you could change their
>>>>>> allocation with a sacctmgr command. It is especially useful when you are
>>>>>> managing multiple clusters that share licenses. You can allocate that a
>>>>>> certain number are allowed by each cluster and change that if needed.
>>>>>>
>>>>>> If you got creative, you could keep the license count that is in the
>>>>>> database updated to match the number free from flexlm to stop license
>>>>>> starvation due to users outside slurm using them up so they really
>>>>>> aren't available to slurm.
>>>>>>
>>>>>> Brian Andrus
>>>>>>
>>>>>>
>>>>>> On 9/15/2022 3:34 PM, Davide DelVento wrote:
>>>>>>> I am a bit confused by remote licenses.
>>>>>>>
>>>>>>> https://lists.schedmd.com/pipermail/slurm-users/2020-September/006049.html
>>>>>>> (which is only 2 years old) claims that they are just a counter, so
>>>>>>> like local licenses. Then why call them remote?
>>>>>>>
>>>>>>> Only a few days after, this
>>>>>>> https://lists.schedmd.com/pipermail/slurm-users/2020-September/006081.html
>>>>>>> appeared to imply (but not clearly stated) that the remote license are
>>>>>>> not simply a counter, but then it's not clear how they are different.
>>>>>>>
>>>>>>> The current documentation (and attempts to run the "add resource"
>>>>>>> command) says that one must use the license count, which seems to
>>>>>>> imply they are just a simple counter (but then what do they need the
>>>>>>> server for)?
>>>>>>>
>>>>>>> So what is what?
>>>>>>>
>>>>>>> In my cursory past experience with this, it seemed that it were
>>>>>>> possible to query a license server (at least some of them) to get the
>>>>>>> actual number of available licenses and schedule (or let jobs pending)
>>>>>>> accordingly. Which would be very helpful for the not-too-uncommon
>>>>>>> situation in which the same license server provides licenses for both
>>>>>>> the HPC cluster and other non-slurm-controlled resources, such a
>>>>>>> user's workstations. Was that impression wrong, or perhaps somebody
>>>>>>> scripted it in some way? If the latter, does anybody know if those
>>>>>>> scripts are publicly available anywhere?
>>>>>>>
>>>>>>> Thanks
>>>>>>>