<div dir="ltr"><div>Thanks Ole for pointing to necessary documents.</div><div><br></div><div>So Do you mean to say I need to configure the "billing" parameter instead of "GrpTRESMins" . </div><div><br></div><div>>></div><div>[testuser100@login04 3072]$ scontrol show assoc users=testuser100 accounts=testfac3_imf flags=assoc|grep GrpTRESMins<br> GrpTRESMins=cpu=0(88),mem=N(380538),energy=N(0),node=N(9),billing=N(176),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0)<br></div><div>>></div><div><br></div><div>Regards</div><div>Hemanta</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 19, 2022 at 2:01 PM <<a href="mailto:slurm-users-request@lists.schedmd.com">slurm-users-request@lists.schedmd.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send slurm-users mailing list submissions to<br>
<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users" rel="noreferrer" target="_blank">https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:slurm-users-request@lists.schedmd.com" target="_blank">slurm-users-request@lists.schedmd.com</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:slurm-users-owner@lists.schedmd.com" target="_blank">slurm-users-owner@lists.schedmd.com</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of slurm-users digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: slurm-users Digest, Vol 60, Issue 19 (Hemanta Sahu)<br>
2. Re: slurm-users Digest, Vol 60, Issue 19 (Ole Holm Nielsen)<br>
3. GPU utilization of running jobs (Vecerka Daniel)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 18 Oct 2022 19:39:39 +0530<br>
From: Hemanta Sahu <<a href="mailto:hemantaku.sahu@gmail.com" target="_blank">hemantaku.sahu@gmail.com</a>><br>
To: <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br>
Subject: Re: [slurm-users] slurm-users Digest, Vol 60, Issue 19<br>
Message-ID:<br>
<CAH5Hmwf4jRXYFYbwwBe=+<a href="mailto:aRtrzWo22ycuy_akGch06W_OPo5qg@mail.gmail.com" target="_blank">aRtrzWo22ycuy_akGch06W_OPo5qg@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi Ole,<br>
<br>
I confirm that the Slurm database has been configured and "<br>
AccountingStorageEnforce" parameters have been set.<br>
<br>
>><br>
[admin2@login01 ~]$ scontrol show config | grep AccountingStorageEnforce<br>
AccountingStorageEnforce = associations,limits,qos,safe<br>
>><br>
<br>
My Question : If I have multiple users under a slurm Account and I<br>
want to limit user xxx to max 1000 CPU core-minutes and user yyy to<br>
max 2000 CPU core-minutes for all past,present and future jobs , what<br>
would be the best way to achieve this ?<br>
<br>
<br>
Thanks<br>
<br>
Hemanta<br>
<br>
<br>
On Tue, Oct 18, 2022 at 5:31 PM <<a href="mailto:slurm-users-request@lists.schedmd.com" target="_blank">slurm-users-request@lists.schedmd.com</a>><br>
wrote:<br>
<br>
> Send slurm-users mailing list submissions to<br>
> <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br>
><br>
> To subscribe or unsubscribe via the World Wide Web, visit<br>
> <a href="https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users" rel="noreferrer" target="_blank">https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users</a><br>
> or, via email, send a message with subject or body 'help' to<br>
> <a href="mailto:slurm-users-request@lists.schedmd.com" target="_blank">slurm-users-request@lists.schedmd.com</a><br>
><br>
> You can reach the person managing the list at<br>
> <a href="mailto:slurm-users-owner@lists.schedmd.com" target="_blank">slurm-users-owner@lists.schedmd.com</a><br>
><br>
> When replying, please edit your Subject line so it is more specific<br>
> than "Re: Contents of slurm-users digest..."<br>
><br>
><br>
> Today's Topics:<br>
><br>
> 1. How to implement resource restriction for diffrent slurm<br>
> users under same slurm account (Hemanta Sahu)<br>
> 2. Re: How to implement resource restriction for diffrent slurm<br>
> users under same slurm account (Ole Holm Nielsen)<br>
><br>
><br>
> ----------------------------------------------------------------------<br>
><br>
> Message: 1<br>
> Date: Mon, 17 Oct 2022 20:21:59 +0530<br>
> From: Hemanta Sahu <<a href="mailto:hemantaku.sahu@gmail.com" target="_blank">hemantaku.sahu@gmail.com</a>><br>
> To: <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br>
> Subject: [slurm-users] How to implement resource restriction for<br>
> diffrent slurm users under same slurm account<br>
> Message-ID:<br>
> <CAH5HmweLsQ7uUkH=D=<br>
> <a href="mailto:6XqbzrxLXsGDvPftSBt6T5snqbqQckXQ@mail.gmail.com" target="_blank">6XqbzrxLXsGDvPftSBt6T5snqbqQckXQ@mail.gmail.com</a>><br>
> Content-Type: text/plain; charset="utf-8"<br>
><br>
> Dear All,<br>
><br>
> I want to implement resource restriction for different slurm users<br>
> under<br>
> the same slurm account by setting "GrpTRESMins" Flag. For testing purposes<br>
> I set the "GrpTRESMins=cpu=0" and submitted the Job.<br>
><br>
> I expect the job submission should fail but it is not happening. The jobs<br>
> are still going to Q and running.<br>
><br>
> Kindly help me if I am missing anything obvious. Command outputs given<br>
> below for reference.<br>
><br>
><br>
> >><br>
> [testfac3@login04 export_bin]$ sacctmgr modify user name=testuser100<br>
> Account=testfac3_imf set GrpTRESMins=cpu=0<br>
> Modified user associations...<br>
> C = param-shakti A = testfac3_imf U = testuser100<br>
> Would you like to commit changes? (You have 30 seconds to decide)<br>
> (N/y): y<br>
><br>
> [testuser100@login04 ~]$ sacctmgr show assoc where Account=testfac3_imf<br>
> user=testuser100 format=Account%15,User%15,GrpTRESMins,QOS%30<br>
> Account User GrpTRESMins<br>
> QOS<br>
> --------------- --------------- -------------<br>
> ------------------------------<br>
> testfac3_imf testuser100 cpu=0<br>
> testfac3_imf<br>
><br>
> [testuser100@login04 testuser100]$ sacctmgr show qos sacctmgr show qos<br>
> testfac3_imf<br>
><br>
> format=Name%20,MaxWall,Flags%20,GrpTRESMins%20,MaxSubmitJobsPerUser,MaxSubmitJobsPeraccount,GrpTRESRunMin,Priority<br>
> Name MaxWall Flags GrpTRESMins<br>
> MaxSubmitPU MaxSubmitPA GrpTRESRunMin Priority<br>
> -------------------- ----------- -------------------- --------------------<br>
> ----------- ----------- ------------- ----------<br>
> testfac3_imf 3-00:00:00 DenyOnLimit,NoDecay cpu=210000000<br>
> 100 500 10000<br>
> [testuser100@login04 testuser100]$<br>
><br>
> [testuser100@login04 testuser100]$ scontrol show job 949622|grep JobState<br>
> JobState=COMPLETED Reason=None Dependency=(null)<br>
> [testuser100@login04 testuser100]$<br>
><br>
><br>
> [testuser100@login04 testuser100]$ cat testjob.sh<br>
> #!/bin/bash<br>
> #SBATCH -J testjob # name of the job<br>
> #SBATCH -p standard # name of the partition: available<br>
> options "standard,standard-low,gpu,gpu-low,hm"<br>
> #SBATCH -n 2 # no of processes<br>
> #SBATCH -q testfac3_imf<br>
> #SBATCH -A testfac3_imf<br>
> #SBATCH -t 01:00:00 # walltime in HH:MM:SS, Max value<br>
> 72:00:00<br>
> #list of modules you want to use, for example<br>
> module load compiler/intel-mpi/mpi-2020-v4 compiler/intel/2020.4.304<br>
><br>
> #name of the executable<br>
> exe="uname -n"<br>
><br>
> #run the application<br>
> mpirun -n $SLURM_NTASKS $exe<br>
><br>
> [testuser100@login04 testuser100]$ sbatch testjob.sh<br>
> Submitted batch job 949622<br>
><br>
> [testuser100@login04 testuser100]$ squeue<br>
> JOBID PARTITION NAME USER ST TIME NODES<br>
> NODELIST(REASON)<br>
> 949622 standard testjob- testuser R 0:04 2<br>
> cn[304-305]<br>
> >><br>
><br>
> Thanks in advance<br>
><br>
> Best Regards<br>
> Hemanta<br>
> -------------- next part --------------<br>
> An HTML attachment was scrubbed...<br>
> URL: <<br>
> <a href="http://lists.schedmd.com/pipermail/slurm-users/attachments/20221017/7b137ddb/attachment-0001.htm" rel="noreferrer" target="_blank">http://lists.schedmd.com/pipermail/slurm-users/attachments/20221017/7b137ddb/attachment-0001.htm</a><br>
> ><br>
><br>
> ------------------------------<br>
><br>
> Message: 2<br>
> Date: Tue, 18 Oct 2022 07:41:29 +0200<br>
> From: Ole Holm Nielsen <<a href="mailto:Ole.H.Nielsen@fysik.dtu.dk" target="_blank">Ole.H.Nielsen@fysik.dtu.dk</a>><br>
> To: <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
> Subject: Re: [slurm-users] How to implement resource restriction for<br>
> diffrent slurm users under same slurm account<br>
> Message-ID: <<a href="mailto:e780481e-9314-9a3e-790b-40c08e3f66bd@fysik.dtu.dk" target="_blank">e780481e-9314-9a3e-790b-40c08e3f66bd@fysik.dtu.dk</a>><br>
> Content-Type: text/plain; charset="UTF-8"; format=flowed<br>
><br>
> On 10/17/22 16:51, Hemanta Sahu wrote:<br>
> > ? ? ?I want to implement resource restriction for different slurm users<br>
> > under<br>
> > the same slurm account by setting ?"GrpTRESMins" Flag. For testing<br>
> > purposes I set the "GrpTRESMins=cpu=0" ?and submitted the Job.<br>
> ><br>
> > ? I expect the job submission should fail but it is not happening. The<br>
> > jobs are still going to Q and running.<br>
> ><br>
> > Kindly help me if I am missing anything obvious. Command outputs given<br>
> > below for reference.<br>
><br>
> Job submission should not fail due to resource limits.<br>
><br>
> Read the slurm.conf manual page to make sure you have set this parameter<br>
> correctly, for example:<br>
><br>
> $ scontrol show config | grep AccountingStorageEnforce<br>
> AccountingStorageEnforce = associations,limits,qos,safe<br>
><br>
> You should also read this documentation:<br>
> <a href="https://slurm.schedmd.com/resource_limits.html" rel="noreferrer" target="_blank">https://slurm.schedmd.com/resource_limits.html</a><br>
><br>
> I assume that you have configured a Slurm database?<br>
><br>
> /Ole<br>
><br>
><br>
><br>
> End of slurm-users Digest, Vol 60, Issue 19<br>
> *******************************************<br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.schedmd.com/pipermail/slurm-users/attachments/20221018/662f720c/attachment-0001.htm" rel="noreferrer" target="_blank">http://lists.schedmd.com/pipermail/slurm-users/attachments/20221018/662f720c/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 19 Oct 2022 08:22:22 +0200<br>
From: Ole Holm Nielsen <<a href="mailto:Ole.H.Nielsen@fysik.dtu.dk" target="_blank">Ole.H.Nielsen@fysik.dtu.dk</a>><br>
To: <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
Subject: Re: [slurm-users] slurm-users Digest, Vol 60, Issue 19<br>
Message-ID: <<a href="mailto:76e0cc7c-8ec0-140f-beef-d82498ad63ef@fysik.dtu.dk" target="_blank">76e0cc7c-8ec0-140f-beef-d82498ad63ef@fysik.dtu.dk</a>><br>
Content-Type: text/plain; charset="UTF-8"; format=flowed<br>
<br>
Hi Hemanta,<br>
<br>
On 10/18/22 16:09, Hemanta Sahu wrote:<br>
> Hi Ole,<br>
> <br>
> I confirm that the Slurm database has been configured and " <br>
> AccountingStorageEnforce"?parameters have been set.<br>
> <br>
> >><br>
> [admin2@login01 ~]$ ?scontrol show config | grep AccountingStorageEnforce<br>
> AccountingStorageEnforce = associations,limits,qos,safe<br>
> >><br>
> <br>
> My Question : If I have multiple users under a slurm Account and I want to <br>
> limit user xxx to max 1000 CPU core-minutes and user yyy to max 2000 CPUcore-minutes for all past,present <br>
> and future jobs , what would be the best way to achieve this ?<br>
<br>
Slurm works with a "Associations", see some information in<br>
<a href="https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_accounting/#associations" rel="noreferrer" target="_blank">https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_accounting/#associations</a><br>
<br>
Under each account in the hierarchy you must define the different users <br>
that you want. They must have different associations (under the same <br>
account), and you can give them different limits. See this nice tutorial: <br>
<a href="https://slurm.schedmd.com/slurm_ug_2012/SUG_Oct2012_DBD.pdf" rel="noreferrer" target="_blank">https://slurm.schedmd.com/slurm_ug_2012/SUG_Oct2012_DBD.pdf</a><br>
<br>
You can only put limits on future jobs, not on past and running jobs. If <br>
you want to make limits for all of history, you need to setup "banking", <br>
see <a href="https://slurm.schedmd.com/SLUG19/Banking.pdf" rel="noreferrer" target="_blank">https://slurm.schedmd.com/SLUG19/Banking.pdf</a><br>
<br>
/Ole<br>
<br>
<br>
<br>
<br>
<br>
------------------------------<br><br>
</blockquote></div></div>