<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Every cluster I've ever managed has this issue. Once cgroup support arrived in Linux, the path we took (on CentOS 6) was to use the 'cgconfig' and 'cgred' services on the login node(s) to setup containers for regular users and quarantine them therein. The config left 4 CPU cores unused by regular users (cpuset config), and allowed them to use up to 100% of the 16 cores granted but yield cycles as other users demand (cpu config). The config also keeps a minor amount of RAM unused by regular users, and limits each regular user to a couple GB.<div class=""><br class=""></div><div class="">The cgrules.conf works on first-match, so at the top we make sure root and sysadmins don't have any limits. Support staff get the overall limits for regular users, and everyone else who's not a daemon user, etc, gets a personal cgroup with the most stringent limits.</div><div class=""><br class=""><div class=""><br class=""><div class=""><br class=""></div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class="">/etc/cgconfig.conf:</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class="">mount {</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>cpuset<span class="Apple-tab-span" style="white-space:pre"> </span>= /cgroup/cpuset;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>cpu<span class="Apple-tab-span" style="white-space:pre"> </span>= /cgroup/cpu;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>#cpuacct<span class="Apple-tab-span" style="white-space:pre"> </span>= /cgroup/cpuacct;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>memory<span class="Apple-tab-span" style="white-space:pre"> </span>= /cgroup/memory;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>#devices<span class="Apple-tab-span" style="white-space:pre"> </span>= /cgroup/devices;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>#freezer<span class="Apple-tab-span" style="white-space:pre"> </span>= /cgroup/freezer;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>#net_cls<span class="Apple-tab-span" style="white-space:pre"> </span>= /cgroup/net_cls;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>#blkio<span class="Apple-tab-span" style="white-space:pre"> </span>= /cgroup/blkio;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class="">}</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""><br class=""></font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class="">group regular_users {</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> cpu {</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> cpu.shares=100;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> }</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> cpuset {</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> cpuset.cpus=4-19;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> cpuset.mems=0-1;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> }</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> memory {</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> memory.limit_in_bytes=48G;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> memory.soft_limit_in_bytes=48G;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> memory.memsw.limit_in_bytes=60G;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> }</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class="">}</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""><br class=""></font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class="">template regular_users/%U {</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> cpu {</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> cpu.shares=100;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> }</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> cpuset {</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> cpuset.cpus=4-19;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> cpuset.mems=0-1;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> }</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> memory {</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> memory.limit_in_bytes=4G;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> memory.soft_limit_in_bytes=2G;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> memory.memsw.limit_in_bytes=6G;</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class=""> }</font></div></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class="">}</font></div><div class=""><font size="1" class=""><br class=""></font></div><div class=""><font size="1" class=""><br class=""></font></div></blockquote><span style="font-size: x-small;" class="">/etc/cgrules.conf</span><br class=""><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><font size="1" class="">#<br class=""># Include an explicit rule for root, otherwise commands with<br class=""># the setuid bit set on them will inherit the original user's<br class=""># gid and probably wind up under @everyone:<br class="">#<br class="">root<span class="Apple-tab-span" style="white-space:pre"> </span>cpuset,cpu,memory<span class="Apple-tab-span" style="white-space:pre"> </span>/<br class=""><br class="">#<br class=""># sysadmin<br class="">#<br class="">user1<span class="Apple-tab-span" style="white-space:pre"> </span>cpuset,cpu,memory<span class="Apple-tab-span" style="white-space:pre"> </span>/<br class="">user2<span class="Apple-tab-span" style="white-space:pre"> </span>cpuset,cpu,memory<span class="Apple-tab-span" style="white-space:pre"> </span>/<br class=""><br class="">#<br class=""># sysstaff<br class="">#<br class="">user3<span class="Apple-tab-span" style="white-space:pre"> </span>cpuset,cpu,memory<span class="Apple-tab-span" style="white-space:pre"> </span>regular_users/<br class="">user4<span class="Apple-tab-span" style="white-space:pre"> </span>cpuset,cpu,memory<span class="Apple-tab-span" style="white-space:pre"> </span>regular_users/<br class=""><br class="">#<br class=""># workgroups:<br class="">#<br class="">@everyone<span class="Apple-tab-span" style="white-space:pre"> </span>cpuset,cpu,memory<span class="Apple-tab-span" style="white-space:pre"> </span>regular_users/%U/<br class="">@group1<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>cpuset,cpu,memory<span class="Apple-tab-span" style="white-space:pre"> </span>regular_users/%U/<br class="">@group2<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>cpuset,cpu,memory<span class="Apple-tab-span" style="white-space:pre"> </span>regular_users/%U/</font></div><div class=""><font size="1" class=""> :</font></div></blockquote></blockquote><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class="">On Feb 15, 2018, at 10:11 AM, Manuel RodrÃguez Pascual <<a href="mailto:manuel.rodriguez.pascual@gmail.com" class="">manuel.rodriguez.pascual@gmail.com</a>> wrote:<br class=""><br class="">Hi all, <br class=""><br class="">Although this is not strictly related to Slurm, maybe you can recommend me some actions to deal with a particular user. <br class=""><br class="">On our small cluster, currently there are no limits to run applications in the frontend. This is sometimes really useful for some users, for example to have scripts monitoring the execution of jobs and taking decisions depending on the partial results.<br class=""><br class="">However, we have this user that keeps abusing this system: when the job queue is long and there is a significant time wait, he sometimes runs his jobs on the frontend, resulting on a CPU load of 100% and some delays on using it for the things it is supposed to serve (user login, monitoring and so). <br class=""><br class="">Have you faced the same issue? Is there any solution? I am thinking about using ulimit to limit the execution time of this jobs in the frontend to 5 minutes or so. This however does not look so elegant as other users can perform the sabe abuse on the future, and he should also be able to run low cpu-consuming jobs for a longer period. However I am not an experienced sysadmin so I am completely open to suggestions or different ways of facing this issue.<br class=""><br class="">Any thoughts?<br class=""><br class="">cheers, <br class=""><br class=""><br class=""><br class=""><br class="">Manuel<br class=""></blockquote><br class=""><div class=""><br class="">::::::::::::::::::::::::::::::::::::::::::::::::::::::<br class="">Jeffrey T. Frey, Ph.D.<br class="">Systems Programmer V / HPC Management<br class="">Network & Systems Services / College of Engineering<br class="">University of Delaware, Newark DE 19716<br class="">Office: (302) 831-6034 Mobile: (302) 419-4976<br class="">::::::::::::::::::::::::::::::::::::::::::::::::::::::<br class=""><br class=""><br class=""><br class=""></div><br class=""></div></div></div></body></html>