[slurm-users] Slurm reservation for migrating user home directories

[Ole Holm Nielsen]

On 4/16/21 4:21 PM, Ole Holm Nielsen wrote:
> I'm thinking of a reservation something like this:
> 
> scontrol create reservation starttime=...  duration=12:00:00 
> ReservationName=migrate_physics nodes=ALL Accounts=-physics

For the record:  The idea of creating a Slurm reservation for excluding 
specified accounts from running jobs seems to be a viable one.  The 
question is being tracked in https://bugs.schedmd.com/show_bug.cgi?id=11404

The correct way to make such a reservation is actually to add several flags:

$ scontrol create reservation reservationname=exclude_account 
starttime=13:40:00 duration=30:00 flags=ignore_jobs,magnetic,flex 
nodes=ALL accounts=-sub1

Caveat: This will result in all Pending jobs getting an incorrect 
Reason=(ReqNodeNotAvail, Reserved for maintenance).  It seems that jobs 
from other accounts are starting correctly, however, so this does achieve 
the goal, but probably also causes confusion among users!

SchedMD is looking at a way to enhance a future Slurm version so that the 
incorrect Reason doesn't appear


>> On 16/04/2021 14.23, Ole Holm Nielsen wrote:
>>> I need to migrate several sets of user home directories from an old NFS 
>>> file server to a new NFS file server.  Each group of users belong to 
>>> specific Slurm accounts organized in a hierarchical tree.
>>>
>>> I want to make the migration while the cluster is in full production 
>>> mode for all the other accounts (the terms "service window" or 
>>> "downtime" don't exist for me :-)
>>>
>>> My idea is to make a Slurm reservation so that the accounts in question 
>>> will have zero jobs running during the reservation, and I also need to 
>>> kick users off the login nodes.  During the reservation I can rsync the 
>>> home directories from the old NFS server to the new NFS server and 
>>> update the NFS automounter links.
>>>
>>> Question:  Does anyone have experiences with this type of scenario? Any 
>>> good ideas or suggestions for other methods for data migration?

/Ole