[slurm-users] Rate-limiting sbatch and srun

[Christopher Samuel]

On 7/18/22 3:45 pm, gphipps wrote:

> Everyone so often one of our users accidentally writes a “fork-bomb” 
> that submits thousands of sbatch and srun requests per second. It is a 
> giant DDOS attack on our scheduler. Is there a way of rate limiting 
> these requests before they reach the daemon?

Yes there is, you can use the Slurm cli_filter to do this.

https://slurm.schedmd.com/cli_filter_plugins.html

If you use the lua plugin you can write what you need in that; though of 
course it would need careful thought as you would need somewhere to 
store state on the node (writeable by users), a way of counting the 
frequency of the RPCs and introducing increasing delays (up to a point) 
if it's out of control and then decaying that delay time down when the 
RPCs from that user cease/decrease.

All the best,
Chris
-- 
Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA