[slurm-users] Slurm Feature Poll

Wed Aug 28 15:13:14 UTC 2019

Hi Paul,

I submitted the poll - thanks! For bug #7609, while I'd be happier with a built in slurm solution, you may find that our jobscript archiver implementation would work nicely for you. It is very high-performing and has no effect on the scheduler, or db performance. 

The solution is a multithreaded c++ program which starts 1 thread for each /var/spool/slurm/hash.N directory. Each thread subscribes to inotify filesystem change events and when a new job directory shows up under hash.N, the program copies the jobscript file, and environment file to a local archive directory, at the same time creating user based ACLs on the files/dirs for security. We then have a cron that moves the jobscripts to a NFS share from which users can grab their jobscripts if desired. For our model, we wanted only the admins, and the user that submitted the script to have access to the jobscripts. Thus the reason for the ACLs on the files/dirs. 

We tried a slurmctld_prolog solution initially to archive jobs, but impacted scheduler performance dramatically.

We have been very happy with it. Check it out, if you find it useful let me know!

https://github.com/nauhpc/job_archive

If you have any questions, please let me know!

Best,
Chris

—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167

On 8/28/19, 7:25 AM, "slurm-users on behalf of Paul Edmon" <slurm-users-bounces at lists.schedmd.com on behalf of pedmon at cfa.harvard.edu> wrote:

    We have several pending feature requests to SchedMD regarding different 
    features we would like to see, as I am sure many other groups have.  We 
    were curious if anyone else in the community is interested in these 
    features and if your group would be interested in talking with us 
    (Harvard FAS Research Computing) about getting these implemented 
    (possibly through some coalition or collaboration).  Please let us know 
    which features you are interested in the poll below and then please list 
    your University or organization and contact person.  If you don't want 
    to send your preferred contact info just post your organization and we 
    will reach out to you.  Thanks in advance.

    https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fforms.gle%2FLGWLuu9b3bRcihHs7&data=02%7C01%7Cchris.coffey%40nau.edu%7C53f4b79669d1455b7a5b08d72bc3a322%7C27d49e9f89e14aa099a3d35b57b2ba03%7C0%7C0%7C637025991574652242&sdata=8yOILmYvqUwCVvZNpiHmZdSdAMgVyduGRqbuStXvANQ%3D&reserved=0

    -Paul Edmon-

    FAS Research Computing

    Harvard University