<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-CA" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">I’ve setup SLURM to enable pre-emption so that high-priority jobs can take-over resources from lower-priority jobs. As we use a lot of expensive EDA software, we want to get the best use of these expensive licenses. The software all uses
the FlexLM license manager, and when a job is suspended using SIGTSTP and later resumed with SIGCONT, it releases and then gets the license again allowing another job to use it.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I wrote a simple BASH script to test this behavior with SLURM:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">#!/bin/bash<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">function suspendJob () {<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""> echo "INFO: Job Suspended"<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">}<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">function resumeJob () {<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""> echo "INFO: Job Resumed"<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">}<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">function terminateJob () {<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""> echo "INFO: Job Terminating..."<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">}<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">trap suspendJob SIGTSTP<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">trap resumeJob SIGCONT<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">trap terminateJob SIGTERM<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">echo "Burning some compute now...."<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New"">yes > /dev/null<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal">When I configure SLURM to use:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> ProctrackType=protrack/pgid<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This works as expected when I manually SUSPEND/RESUME/CANCEL a job with each of the corresponding messages appearing in the SLURM StdOut file.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When I change SLURM to use CGROUPS:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> ProctrackType=protrack/cgroup<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">No messages appear at all in the SLURM StdOut file indicated that the cgroup was thrown into freezer without any signals being sent. Is this expected behavior and are there ways to “fix” this so that it behaves the same way as using Process
Groups?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Maybe this is a moot point since SLURM still shows the License being Used under “scontrol show license” even if a job is suspended, but I figure that problem might be solvable…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Michael<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>