[slurm-users] Working with local licenses

[Loris Bennett]

Hi Alexey,

"Tager, Alexey" <alexeyt at amazon.com> writes:

> Hi Loris,
>
> I just need better understanding details of your solution 
> Are you talking about dynamic licenses, right?

I am talking about a limited pool of licences for which one licence is
consumed for each user-node pair.

> Should I add license ttt to database by sacctmgr with "Count" and "% Allocated",
> and also create unlimited reservation of same license ttt and update it with "free licenses"?

We didn't add anything to the database.  We added the total number of
licences to the slurm.conf using the 'Licenses' option.  We then just
calculated the number of free licences by parsing the output of the
licence manager and then updated the reservation with 'scontrol' with a
cron job.

> How I define resource as "unlimited" ?

This approach does not use a 'resource' in the sense of sacctmgr.
Instead it relies on licences within a reservation created with

  duration=infinite

> Will sacctmgr see "Count" updated and scontrol see "Total" and "Used" accordingly ?

Probably not, since this approach, as I mention above, does not use a
sacctmrg resource.

Cheers,

Loris

> Thank you
> Alexey
>
>
>
> -----Original Message-----
> From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Loris Bennett
> Sent: Monday, September 28, 2020 10:23 AM
> To: Slurm User Community List <slurm-users at lists.schedmd.com>
> Subject: RE: [EXTERNAL] [slurm-users] Working with local licenses
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> "Tager, Alexey" <alexeyt at amazon.com> writes:
>
>> Hi Ahmet,
>>
>> I agree, it isn't so good solution
>
> We used to use a solution which also wasn't very good, but perhaps slightly
>> better than restarting slurmctld every minute.  We created an unlimited
>> reservation and then updated the reservation via a cron job with the current
>> number of free licences.  Obviously the users then had to specify the
>> reservation as well as the licence.
>
> Cheers,
>
> Loris
>
>> Thanks
>> Alexey
>>
>> -----Original Message-----
>> From: mercan <ahmet.mercan at uhem.itu.edu.tr>
>> Sent: Friday, September 25, 2020 11:50 AM
>> To: Tager, Alexey <alexeyt at amazon.com>; Slurm User Community List 
>> <slurm-users at lists.schedmd.com>
>> Subject: RE: [EXTERNAL] [slurm-users] Working with local licenses
>>
>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>
>>
>>
>> Hi;
>>
>> But, restarting in every minute, is not a good solution.
>>
>> Ahmet M.
>>
>>
>> On 25.09.2020 11:25, Tager, Alexey wrote:
>>> Hi Ahmet,
>>>
>>> We tried remote licenses, but encountered following issues, which lead us to using of local licenses.
>>> - only low case while inserting by sacctmgr
>>>   - dead locks and duplicate records
>>> - direct insert is working and case sensitive, but scontrol doesn't 
>>> see change until slurmctld restart
>>>
>>>
>>> Regards
>>> Alexey
>>>
>>> -----Original Message-----
>>> From: mercan <ahmet.mercan at uhem.itu.edu.tr>
>>> Sent: Friday, September 25, 2020 11:16 AM
>>> To: Slurm User Community List <slurm-users at lists.schedmd.com>; Tager, 
>>> Alexey <alexeyt at amazon.com>
>>> Subject: RE: [EXTERNAL] [slurm-users] Working with local licenses
>>>
>>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>>
>>>
>>>
>>> Hi;
>>>
>>> You don't need to modify slurm.conf and reconfigure. There is a remote/dynamic licenses feature:
>>>
>>> https://slurm.schedmd.com/licenses.html
>>>
>>> You can add licenses using scontrol command such as:
>>>
>>> sacctmgr add resource name=matlab count=50 server=rlm_host \
>>>     servertype=rlm type=license
>>>
>>> Regards;
>>>
>>> Ahmet M.
>>>
>>>
>>> On 25.09.2020 11:09, Tager, Alexey wrote:
>>>> Hello,
>>>>
>>>> We use  local licenses by updating slurm.conf with script,
>>>>
>>>> which talking with FlexLM
>>>>
>>>> We execute “scontrol reconfigure” every 1 min after file change in 
>>>> crontab
>>>>
>>>> Users get following error when run slurm commands (srun,scontrol
>>>> etc.) during
>>>>
>>>> crontab execution, as result srun failed on license request
>>>>
>>>> Any option to prevent such behavior ?
>>>>
>>>> [September 24, 2020, 7:18 PM] Tager, Alexey: scontrol: error: Parse 
>>>> error in file /tools/slurm/etc/slurm_licenses.conf line 10: "Licenses="
>>>>
>>>> scontrol: error: "Include" failed in file 
>>>> /tools/slurm/etc/slurm.conf line 13
>>>>
>>>> scontrol: fatal: Unable to process configuration file
>>>>
>>>> /Alexey Tager/
>>>>
> --
> Dr. Loris Bennett (Mr.)
> ZEDAT, Freie Universität Berlin         Email loris.bennett at fu-berlin.de