Hi Fritz, "Ratnasamy, Fritz via slurm-users" <slurm-users@lists.schedmd.com> writes:
Hello,
We have a recurring demand where some users would like to use idle GPU resources on periods where the usage is low. Their normal GPU/qos limits prevent this, which is expected as part of the fair share scheduler principles.
Fairshare itself does not necessarily prevent users from using idle resources. If there is no competition for the resources, then anyone should be able to use them, regardless of the resources that person has used recently. Fairshare just decides the priority when there is competition. We have several users who continually submit jobs and have no almost shares, but their jobs still run, because there are always periods during which resources become idle.
A pattern we're considering is the preempt option on slurm to have these jobs being pre-empted when another user is submitting his jobs within his quota. The jobs that were submitted earlier by a faculty member taking advantage of available resources on top of his regular qos, will be preempted however, there is no suspend option for gpu jobs. There is only requeue, which kills the jobs to make the gpu available. I believe you can use suspend for cpu/mem without killing the job. I know that would require the users to set up checkpoints and re-start their jobs. The goal is to let heavy users fill otherwise idle nodes over low usage time without impacting smaller users when activity picks up. Has anyone implemented something similar? I am in general against preemption because that might lead to conflict between users. Thanks,
For me, the main argument against pre-emption is that computing time may be lost, if users are unable to checkpoint their jobs. Regarding possible conflict between users, it should be possible to avoid the by expectation management. We have some resources which were bought by individual research groups and to which members of those research groups have privileged access via a corresponding QOS. Everyone else can also access the resources, but only via a 'scavenger' QOS, which has shorter run-times and other restrictions. The 'scavenger' QOS is fairly heavily used, particularly for GPU resources, and these jobs can be pre-empted, but we haven't had any conflicts so far. Cheers, Loris -- Dr. Loris Bennett (Herr/Mr) FUB-IT, Freie Universität Berlin