[slurm-users] slurm-users Digest, Vol 57, Issue 26

[gphipps]

Thanks, I will look into that. 

Cheers
Geoff


    Message: 1
    Date: Tue, 19 Jul 2022 21:38:56 -0700
    From: Christopher Samuel <chris at csamuel.org>
    To: slurm-users at lists.schedmd.com
    Subject: Re: [slurm-users] Rate-limiting sbatch and srun
    Message-ID: <6f7850ba-8b66-47a0-7312-8c222f1ff969 at csamuel.org>
    Content-Type: text/plain; charset=UTF-8; format=flowed

    On 7/18/22 3:45 pm, gphipps wrote:

    > Everyone so often one of our users accidentally writes a ?fork-bomb? 
    > that submits thousands of sbatch and srun requests per second. It is a 
    > giant DDOS attack on our scheduler. Is there a way of rate limiting 
    > these requests before they reach the daemon?

    Yes there is, you can use the Slurm cli_filter to do this.

    https://slurm.schedmd.com/cli_filter_plugins.html

    If you use the lua plugin you can write what you need in that; though of 
    course it would need careful thought as you would need somewhere to 
    store state on the node (writeable by users), a way of counting the 
    frequency of the RPCs and introducing increasing delays (up to a point) 
    if it's out of control and then decaying that delay time down when the 
    RPCs from that user cease/decrease.

    All the best,
    Chris
    -- 
    Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA




    End of slurm-users Digest, Vol 57, Issue 26
    *******************************************