[slurm-users] Slurm reservation for migrating user home directories

[Ole Holm Nielsen]

Hi Niels Carl,

On 16-04-2021 14:41, Niels Carl Hansen wrote:
> For each account do
>       sacctmgr modify account name=<accountname> set GrpJobs=0
> 
> After sync'ing, resume with
>       sacctmgr modify account name=<accountname> set GrpJobs=-1

Yes, but this would block all jobs from <accountname> immediately.  If 
this account had a week-long job running, it couldn't run any shorter 
jobs until after the migration.

That's why I'm thinking that a system reservation excluding all jobs 
from <accountname> could be created several weeks in advance, so that 
<accountname> jobs could keep starting and running until they would be 
blocked by the reservation.

I'm thinking of a reservation something like this:

scontrol create reservation starttime=...  duration=12:00:00 
ReservationName=migrate_physics nodes=ALL Accounts=-physics

Would this work as expected?

Best regards,
Ole

> On 16/04/2021 14.23, Ole Holm Nielsen wrote:
>> I need to migrate several sets of user home directories from an old 
>> NFS file server to a new NFS file server.  Each group of users belong 
>> to specific Slurm accounts organized in a hierarchical tree.
>>
>> I want to make the migration while the cluster is in full production 
>> mode for all the other accounts (the terms "service window" or 
>> "downtime" don't exist for me :-)
>>
>> My idea is to make a Slurm reservation so that the accounts in 
>> question will have zero jobs running during the reservation, and I 
>> also need to kick users off the login nodes.  During the reservation I 
>> can rsync the home directories from the old NFS server to the new NFS 
>> server and update the NFS automounter links.
>>
>> Question:  Does anyone have experiences with this type of scenario? 
>> Any good ideas or suggestions for other methods for data migration?