[slurm-users] SLURM heterogeneous jobs, a little help needed plz

[Loris Bennett]

Hi Ryan,

Ryan Novosielski <novosirj at rutgers.edu> writes:

>> On Mar 21, 2019, at 11:26 AM, Prentice Bisbal <pbisbal at pppl.gov> wrote:
>> On 3/20/19 1:58 PM, Christopher Samuel wrote:
>>> On 3/20/19 4:20 AM, Frava wrote:
>>> 
>>>> Hi Chris, thank you for the reply.
>>>> The team that manages that cluster is not very fond of upgrading SLURM, which I understand.
>> 
>> As a system admin who manages clusters myself, I don't understand this. Our
>> job is to provide and maintain resources for our users. Part of that
>> maintenance is to provide updates for security, performance, and functionality
>> (new features) reasons. HPC has always been a leading-edge kind if field, so I
>> feel this is even more important for HPC admins.
>> 
>> Yes, there can be issues caused by updates, but those can be with proper
>> planning: Have a plan to do the actual upgrade, have a plan to test for
>> issues, and have a plan to revert to an earlier version if issues are
>> discovered. This is work, but it's really not all that much work, and this is
>> exactly the work we are being paid to do as cluster admins.
>> 
>> From my own experience, I find *not* updating in a timely manner is actually
>> more problematic and more work than keep on top of updates. For example, where
>> I work now, we still haven't upgraded to CentOS 7, and as a result, many basic
>> libraries are older than what many of the open-source apps my users need
>> require. As a result, I don't just have to install application X, I often have
>> to install up-to-date versions of basic libraries like libreadline, libcurses,
>> zlib, etc. And then there are the security concerns...

Chris, maybe you should look at EasyBuild
(https://easybuild.readthedocs.io/en/latest/).  That way you can install
all the dependencies (such as zlib) as modules and be pretty much
independent of the ancient packages your distro may provide (other
software-building frameworks are available).

>> Okay, rant over. I'm sorry. It just bothers me when I hear fellow system
>> admins aren't "very fond" of things that I think are a core responsbility of
>> our jobs. I take a lot of pride on my job.
>
> All of those things take time, depending on where you work (not necessarily
> speaking about my current employer/employment situation), you may be ordered to
> do something else with that time. If so, all bets are off. Planned updates where
> sufficient testing time is not allotted moves the associated work from planned
> work to unplanned emergency (something broken, etc.), and in some cases from
> business hours to off hours, generate lots of support queries, etc.
>
> I’ve never seen a paycheck signed by “Best Practices”.

It may be true that some employers prioritise the wrong things, but in
my experience, Slurm is pretty easy and quick to update.  It may seem a
little scary (people often seem to worry erroneously about loosing
everything in the queue), but we started with version 2.2.4 in 2012 and
have always updated regularly.  We have both slurmctld and slurmdbd on
one machine, which is often advised against, but I have only ever had
one problem, which I was able to solve by using a backup of the spool
directory.  Our last cluster only hit around 2.5 million jobs after
around 6 years, so database conversion was never an issue.  For sites
with a higher-throughput things may be different, but I would hope that
at those places, the managers would know the importance of planned
updates and testing.

Cheers,

Loris

-- 
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin         Email loris.bennett at fu-berlin.de