[slurm-users] Slurm reservation for migrating user home directories
Ole Holm Nielsen
Ole.H.Nielsen at fysik.dtu.dk
Fri Apr 16 14:21:07 UTC 2021
Hi Niels Carl,
On 16-04-2021 14:41, Niels Carl Hansen wrote:
> For each account do
> sacctmgr modify account name=<accountname> set GrpJobs=0
> After sync'ing, resume with
> sacctmgr modify account name=<accountname> set GrpJobs=-1
Yes, but this would block all jobs from <accountname> immediately. If
this account had a week-long job running, it couldn't run any shorter
jobs until after the migration.
That's why I'm thinking that a system reservation excluding all jobs
from <accountname> could be created several weeks in advance, so that
<accountname> jobs could keep starting and running until they would be
blocked by the reservation.
I'm thinking of a reservation something like this:
scontrol create reservation starttime=... duration=12:00:00
ReservationName=migrate_physics nodes=ALL Accounts=-physics
Would this work as expected?
> On 16/04/2021 14.23, Ole Holm Nielsen wrote:
>> I need to migrate several sets of user home directories from an old
>> NFS file server to a new NFS file server. Each group of users belong
>> to specific Slurm accounts organized in a hierarchical tree.
>> I want to make the migration while the cluster is in full production
>> mode for all the other accounts (the terms "service window" or
>> "downtime" don't exist for me :-)
>> My idea is to make a Slurm reservation so that the accounts in
>> question will have zero jobs running during the reservation, and I
>> also need to kick users off the login nodes. During the reservation I
>> can rsync the home directories from the old NFS server to the new NFS
>> server and update the NFS automounter links.
>> Question: Does anyone have experiences with this type of scenario?
>> Any good ideas or suggestions for other methods for data migration?
More information about the slurm-users