<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Sry, if this is a double post, but I have the impression my first email was discarded, because I was not yet subscribed to the list - it did not show up in the list archive.<br>
</p>
<br>
<div style="color: rgb(46, 52, 54);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Klein, Dennis<br>
<b>Sent:</b> Tuesday, February 25, 2020 4:56 PM<br>
<b>To:</b> slurm-users@schedmd.com<br>
<b>Subject:</b> Question on how to make slurm aware of a CVMFS revision</font>
<div> </div>
</div>
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p>Hi,</p>
<p><br>
</p>
<p>our slurm worker nodes mount several read-only software repositories via the cvmfs filesystem [1]. Each repository is versioned and each cvmfs mountpoint automatically switches to serving the newest revision eventually. But this may take a while. A slurm
job might depend on a certain revision of such a cvmfs repository. One naive way to express this dependency is to block at the beginning of the job script until the required cvmfs revision is available. A much better way would be to make the slurm scheduler
aware of the job's dependency on a given cvmfs revision and delay scheduling of the job until enough worker nodes satisfying the requirements are available.</p>
<p><br>
</p>
<p>What would be the best slurm facility to achieve the above?</p>
<p><br>
</p>
<p>It appears to me that a no_consume GRES could be a candidate for my use case (the repository revision is a positive integer that increases steadily and the job's dependency is satisfied if current_revision >= required_revision):</p>
<ul style="margin-bottom:0px; margin-top:0px">
<li>Can I use a GRES AutoDetect library to define new no_consume GRES resources per worker node (without having to statically define a list of GRES a-priori)?</li><li>Can I (and if yes, how can I) update the GRES count dynamically (The idea would be to monitor the revision changes on all cvmfs mountpoints with a simple daemon process on each worker node which then notifies slurm on a revision change)?<br>
</li></ul>
<p><br>
</p>
<p>Or is there an entirely different approach necessary?</p>
<p><br>
</p>
<p>Your comments or pointers to docs or source code are very much appreciated!</p>
<p><br>
</p>
<p>Best,</p>
<p>Dennis<br>
</p>
<p><br>
</p>
<p>[1]: <a href="http://cernvm.cern.ch/portal/filesystem" class="OWAAutoLink">http://cernvm.cern.ch/portal/filesystem</a><br>
</p>
</div>
</div>
</div>
</div>
</body>
</html>