[slurm-users] Question on how to make slurm aware of a CVMFS revision

[Klein, Dennis]

Sry, if this is a double post, but I have the impression my first email was discarded, because I was not yet subscribed to the list - it did not show up in the list archive.

________________________________
From: Klein, Dennis
Sent: Tuesday, February 25, 2020 4:56 PM
To: slurm-users at schedmd.com
Subject: Question on how to make slurm aware of a CVMFS revision


Hi,


our slurm worker nodes mount several read-only software repositories via the cvmfs filesystem [1]. Each repository is versioned and each cvmfs mountpoint automatically switches to serving the newest revision eventually. But this may take a while. A slurm job might depend on a certain revision of such a cvmfs repository. One naive way to express this dependency is to block at the beginning of the job script until the required cvmfs revision is available. A much better way would be to make the slurm scheduler aware of the job's dependency on a given cvmfs revision and delay scheduling of the job until enough worker nodes satisfying the requirements are available.


What would be the best slurm facility to achieve the above?


It appears to me that a no_consume GRES could be a candidate for my use case (the repository revision is a positive integer that increases steadily and the job's dependency is satisfied if current_revision >= required_revision):

  *   Can I use a GRES AutoDetect library to define new no_consume GRES resources per worker node (without having to statically define a list of GRES a-priori)?
  *   Can I (and if yes, how can I) update the GRES count dynamically (The idea would be to monitor the revision changes on all cvmfs mountpoints with a simple daemon process on each worker node which then notifies slurm on a revision change)?


Or is there an entirely different approach necessary?


Your comments or pointers to docs or source code are very much appreciated!


Best,

Dennis


[1]: http://cernvm.cern.ch/portal/filesystem
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20200226/cd46ff20/attachment.htm>