[slurm-users] Flag OVERLAP in advanced reservation

[Gizo Nanava]

Hello,

I observe strange behavior of advanced reservations having OVERLAP in their flag's list.

If I create two advanced reservations on different set of nodes and a particular username
is configured to only have an access to one with the flag OVERLAP,  then the username can
also run jobs on nodes in other reservation, which is reserved for other users.

In this example below, the username user1 should not have rights to start jobs on enos-n014:
 
> scontrol show reservations phd
ReservationName=phd StartTime=2023-07-23T08:00:00 EndTime=2023-07-23T20:00:00 Duration=12:00:00
   Nodes=phd-n[001-032] NodeCnt=32 CoreCnt=2048 Features=(null) PartitionName=phd Flags=FLEX,OVERLAP,SPEC_NODES,PART_NODES,MAGNETIC                                                                                             
   TRES=cpu=2048
   Users=user1 Groups=(null) Accounts=(null) Licenses=(null) State=ACTIVE BurstBuffer=(null) Watts=n/a
   MaxStartDelay=(null)

> scontrol show reservations test
ReservationName=phd StartTime=2023-07-23T08:00:00 EndTime=2023-07-23T20:00:00 Duration=12:00:00
   Nodes=enos-n014 NodeCnt=1 CoreCnt=32 Features=(null) PartitionName=(null) Flags=FLEX,SPEC_NODES,MAGNETIC
   TRES=cpu=16
   Users=user2 Groups=(null) Accounts=(null) Licenses=(null) State=ACTIVE BurstBuffer=(null) Watts=n/a
   MaxStartDelay=(null)

user1 at login01:~$ salloc --nodelist=enos-n014
..
salloc: Pending job allocation 1816915
..
salloc: Nodes enos-n014 are ready for job
user1 at enos-n014:~$

It is interesting that once the job starts on the "wrong" node, enos-n014 in the above example, it has the field Reservation equal to phd.

> scontrol show job 1816915 | grep -i reser
Reservation=phd

If I remove OVERLAP (or user1) from the phd reservation, then user1 cannot run jobs on enos-n014.

We run SLURM 22.05.9

Any suggestion?. I would appreciate any help.

Thank you.
Gizo