<div dir="ltr"><div dir="ltr">Hi Juergen,<br></div><div dir="ltr"><br></div><div> Thanks for the guidance.</div><div><br></div><div dir="ltr"> >> is PrivateData also set in your slurmdbd.conf?</div><div dir="ltr"><br></div><div>No. it is not set in slurmdbd.conf. I will set and verify.</div><div><br></div><div>Thanks</div><div>Hemanta</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 20, 2021 at 2:02 PM <<a href="mailto:slurm-users-request@lists.schedmd.com">slurm-users-request@lists.schedmd.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send slurm-users mailing list submissions to<br>
<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users" rel="noreferrer" target="_blank">https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:slurm-users-request@lists.schedmd.com" target="_blank">slurm-users-request@lists.schedmd.com</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:slurm-users-owner@lists.schedmd.com" target="_blank">slurm-users-owner@lists.schedmd.com</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of slurm-users digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: PrivateData does not filter the billing info "scontrol<br>
show assoc_mgr flags=qos" (Juergen Salk)<br>
2. Preemption not working for jobs in higher priority partition<br>
(Russell Jones)<br>
3. GPU jobs not running correctly (Andrey Malyutin)<br>
4. Re: GPU jobs not running correctly (Fulcomer, Samuel)<br>
5. jobs stuck in "CG" state (Durai Arasan)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Thu, 19 Aug 2021 22:51:57 +0200<br>
From: Juergen Salk <<a href="mailto:juergen.salk@uni-ulm.de" target="_blank">juergen.salk@uni-ulm.de</a>><br>
To: Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
Subject: Re: [slurm-users] PrivateData does not filter the billing<br>
info "scontrol show assoc_mgr flags=qos"<br>
Message-ID: <<a href="mailto:20210819205157.GA1331266@qualle.rz.uni-ulm.de" target="_blank">20210819205157.GA1331266@qualle.rz.uni-ulm.de</a>><br>
Content-Type: text/plain; charset=us-ascii<br>
<br>
Hi Hemanta,<br>
<br>
is PrivateData also set in your slurmdbd.conf? <br>
<br>
Best regards<br>
Juergen<br>
<br>
<br>
<br>
* Hemanta Sahu <<a href="mailto:hemantaku.sahu@gmail.com" target="_blank">hemantaku.sahu@gmail.com</a>> [210818 15:04]:<br>
> I am still searching for a solution for this .<br>
> <br>
> On Fri, Aug 7, 2020 at 1:15 PM Hemanta Sahu <<a href="mailto:hemantaku.sahu@gmail.com" target="_blank">hemantaku.sahu@gmail.com</a>><br>
> wrote:<br>
> <br>
> > Hi All,<br>
> ><br>
> > I have configured in our test cluster "PrivateData" parameter in<br>
> > "slurm.conf" as below.<br>
> ><br>
> > >><br>
> > [testuser1@centos7vm01 ~]$ cat /etc/slurm/clurm.conf|less<br>
> ><br>
> > PrivateData=accounts,jobs,reservations,usage,users,events,partitions,nodes<br>
> > MCSPlugin=mcs/user<br>
> > MCSParameters=enforced,select,privatedata<br>
> > >><br>
> ><br>
> > The command "scontrol show assoc_mgr flags=Association" filetrs the<br>
> > relvant information for the user.<br>
> > But "scontrol show assoc_mgr flags=qos" did not filter anything rather it<br>
> > show the information about all QOS<br>
> > to the normal users who even don't have privilege of Slurm Operator/slurm<br>
> > Administaror.Basically I want to Hide the billing details to users who are<br>
> > not co-ordinator for a particular account<br>
> ><br>
> > Appreciate any help or guidance.<br>
> ><br>
> > >><br>
> > [testuser1@centos7vm01 ~]$ scontrol show assoc_mgr flags=qos|egrep<br>
> > "QOS|GrpTRESMins"<br>
> > QOS Records<br>
> > QOS=normal(1)<br>
> ><br>
> > GrpTRESMins=cpu=N(0),mem=N(78),energy=N(0),node=N(0),billing=N(0),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0)<br>
> > QOS=testfac1(7)<br>
> ><br>
> > GrpTRESMins=cpu=N(0),mem=N(143),energy=N(0),node=N(0),billing=6000000(0),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0)<br>
> > QOS=cdac_fac1(10)<br>
> ><br>
> > GrpTRESMins=cpu=N(10),mem=N(163830),energy=N(0),node=N(4),billing=10000000(11),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0)<br>
> > QOS=iitkgp_fac1(11)<br>
> ><br>
> > GrpTRESMins=cpu=N(0),mem=N(20899),energy=N(0),node=N(0),billing=10000000(0),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0)<br>
> > QOS=iitkgp_faculty(13)<br>
> ><br>
> > GrpTRESMins=cpu=N(92),mem=N(379873),energy=N(0),node=N(35),billing=N(175),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0)<br>
> ><br>
> ><br>
> > [testuser1@centos7vm01 ~]$ scontrol show assoc_mgr flags=Association|grep<br>
> > GrpTRESMins<br>
> ><br>
> > GrpTRESMins=cpu=N(0),mem=N(0),energy=N(0),node=N(0),billing=N(0),fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0)<br>
> > [testuser1@centos7vm01 ~]$<br>
> > >><br>
> ><br>
> > Regards,<br>
> > Hemanta<br>
> ><br>
> > Hemanta Kumar Sahu<br>
> > Senior System Engineer<br>
> > CCDS,JC Bose Annexe<br>
> > Phone:03222-304604/Ext:84604<br>
> > I I T Kharagpur-721302<br>
> > E-Mail: <a href="mailto:hksahu@iitkgp.ac.in" target="_blank">hksahu@iitkgp.ac.in</a><br>
> > <a href="mailto:hemantaku.sahu@gmail.com" target="_blank">hemantaku.sahu@gmail.com</a><br>
> ><br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Thu, 19 Aug 2021 16:49:05 -0500<br>
From: Russell Jones <<a href="mailto:arjones85@gmail.com" target="_blank">arjones85@gmail.com</a>><br>
To: Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
Subject: [slurm-users] Preemption not working for jobs in higher<br>
priority partition<br>
Message-ID:<br>
<CABb1d=hx54=<a href="mailto:jb9UC%2Bzpf3JAe%2BV5f0wdPMAQD1KU0UEKPDNkRfA@mail.gmail.com" target="_blank">jb9UC+zpf3JAe+V5f0wdPMAQD1KU0UEKPDNkRfA@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi all,<br>
<br>
I could use some help to understand why preemption is not working for me<br>
properly. I have a job blocking other jobs that doesn't make sense to me.<br>
Any assistance is appreciated, thank you!<br>
<br>
<br>
I have two partitions defined in slurm, a day time and a night time<br>
pariition:<br>
<br>
Day partition - PriorityTier of 5, always Up. Limited resources under this<br>
QOS.<br>
Night partition - PriorityTier of 5 during night time, during day time set<br>
to Down and PriorityTier changed to 1. Jobs can be submitted to night queue<br>
for an unlimited QOS as long as resources are available.<br>
<br>
The thought here is jobs can continue to run in the night partition, even<br>
during the day time, until resources are requested from the day partition.<br>
Jobs would then be requeued/canceled in the night partition to<br>
satisfy those requirements.<br>
<br>
<br>
<br>
Current output of "scontrol show part" :<br>
<br>
PartitionName=day<br>
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL<br>
AllocNodes=ALL Default=NO QoS=part_day<br>
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0<br>
Hidden=NO<br>
MaxNodes=UNLIMITED MaxTime=1-00:00:00 MinNodes=0 LLN=NO<br>
MaxCPUsPerNode=UNLIMITED<br>
Nodes=cluster-r1n[01-13],cluster-r2n[01-08]<br>
PriorityJobFactor=1 PriorityTier=5 RootOnly=NO ReqResv=NO<br>
OverSubscribe=NO<br>
OverTimeLimit=NONE PreemptMode=REQUEUE<br>
State=UP TotalCPUs=336 TotalNodes=21 SelectTypeParameters=NONE<br>
JobDefaults=(null)<br>
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED<br>
<br>
<br>
PartitionName=night<br>
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL<br>
AllocNodes=ALL Default=NO QoS=part_night<br>
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0<br>
Hidden=NO<br>
MaxNodes=22 MaxTime=7-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED<br>
Nodes=cluster-r1n[01-13],cluster-r2n[01-08]<br>
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO<br>
OverSubscribe=NO<br>
OverTimeLimit=NONE PreemptMode=REQUEUE<br>
State=DOWN TotalCPUs=336 TotalNodes=21 SelectTypeParameters=NONE<br>
JobDefaults=(null)<br>
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED<br>
<br>
<br>
<br>
<br>
I currently have a job in the night partition that is blocking jobs in the<br>
day partition, even though the day partition has a PriorityTier of 5, and<br>
night partition is Down with a PriorityTier of 1.<br>
<br>
My current slurm.conf preemption settings are:<br>
<br>
PreemptMode=REQUEUE<br>
PreemptType=preempt/partition_prio<br>
<br>
<br>
<br>
The blocking job's scontrol show job output is:<br>
<br>
JobId=105713 JobName=jobname<br>
Priority=1986 Nice=0 Account=xxx QOS=normal<br>
JobState=RUNNING Reason=None Dependency=(null)<br>
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0<br>
RunTime=17:49:39 TimeLimit=7-00:00:00 TimeMin=N/A<br>
SubmitTime=2021-08-18T22:36:36 EligibleTime=2021-08-18T22:36:36<br>
AccrueTime=2021-08-18T22:36:36<br>
StartTime=2021-08-18T22:36:39 EndTime=2021-08-25T22:36:39 Deadline=N/A<br>
PreemptEligibleTime=2021-08-18T22:36:39 PreemptTime=None<br>
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2021-08-18T22:36:39<br>
Partition=night AllocNode:Sid=cluster-1:1341505<br>
ReqNodeList=(null) ExcNodeList=(null)<br>
NodeList=cluster-r1n[12-13],cluster-r2n[04-06]<br>
BatchHost=cluster-r1n12<br>
NumNodes=5 NumCPUs=80 NumTasks=5 CPUs/Task=1 ReqB:S:C:T=0:0:*:*<br>
TRES=cpu=80,node=5,billing=80,gres/gpu=20<br>
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*<br>
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br>
Features=(null) DelayBoot=00:00:00<br>
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)<br>
<br>
<br>
<br>
The job that is being blocked:<br>
<br>
JobId=105876 JobName=bash<br>
Priority=2103 Nice=0 Account=xxx QOS=normal<br>
JobState=PENDING<br>
Reason=Nodes_required_for_job_are_DOWN,_DRAINED_or_reserved_for_jobs_in_higher_priority_partitions<br>
Dependency=(null)<br>
Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0<br>
RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A<br>
SubmitTime=2021-08-19T16:19:23 EligibleTime=2021-08-19T16:19:23<br>
AccrueTime=2021-08-19T16:19:23<br>
StartTime=Unknown EndTime=Unknown Deadline=N/A<br>
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2021-08-19T16:26:43<br>
Partition=day AllocNode:Sid=cluster-1:2776451<br>
ReqNodeList=(null) ExcNodeList=(null)<br>
NodeList=(null)<br>
NumNodes=3 NumCPUs=40 NumTasks=40 CPUs/Task=1 ReqB:S:C:T=0:0:*:*<br>
TRES=cpu=40,node=1,billing=40<br>
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*<br>
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br>
Features=(null) DelayBoot=00:00:00<br>
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)<br>
<br>
<br>
<br>
Why is the day job not preempting the night job?<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.schedmd.com/pipermail/slurm-users/attachments/20210819/bdecefbc/attachment-0001.htm" rel="noreferrer" target="_blank">http://lists.schedmd.com/pipermail/slurm-users/attachments/20210819/bdecefbc/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Thu, 19 Aug 2021 17:35:29 -0700<br>
From: Andrey Malyutin <<a href="mailto:malyutinag@gmail.com" target="_blank">malyutinag@gmail.com</a>><br>
To: <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br>
Subject: [slurm-users] GPU jobs not running correctly<br>
Message-ID:<br>
<CAGiFTXK6cT=<a href="mailto:MRV2FUEwCCpbvuwTfeoRsmjcJao9ULtfVtuefKA@mail.gmail.com" target="_blank">MRV2FUEwCCpbvuwTfeoRsmjcJao9ULtfVtuefKA@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hello,<br>
<br>
We are in the process of finishing up the setup of a cluster with 3 nodes,<br>
4 GPUs each. One node has RTX3090s and the other 2 have RTX6000s.Any job<br>
asking for 1 GPU in the submission script will wait to run on the 3090<br>
node, no matter resource availability. Same job requesting 2 or more GPUs<br>
will run on any node. I don't even know where to begin troubleshooting this<br>
issue; entries for the 3 nodes are effectively identical in slurm.conf. Any<br>
help would be appreciated. (If helpful - this cluster is used for<br>
structural biology, with cryosparc and relion packages).<br>
<br>
Thank you,<br>
Andrey<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.schedmd.com/pipermail/slurm-users/attachments/20210819/10e1b1b7/attachment-0001.htm" rel="noreferrer" target="_blank">http://lists.schedmd.com/pipermail/slurm-users/attachments/20210819/10e1b1b7/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Thu, 19 Aug 2021 21:05:28 -0400<br>
From: "Fulcomer, Samuel" <<a href="mailto:samuel_fulcomer@brown.edu" target="_blank">samuel_fulcomer@brown.edu</a>><br>
To: Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
Subject: Re: [slurm-users] GPU jobs not running correctly<br>
Message-ID:<br>
<CAOORAuFa+ahMxY--8=a1dVu4cPGUuVSojEDv=Sxg6kfaJLi=<a href="mailto:Zw@mail.gmail.com" target="_blank">Zw@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
What SLURM version are you running?<br>
<br>
What are the #SLURM directives in the batch script? (or the sbatch<br>
arguments)<br>
<br>
When the single GPU jobs are pending, what's the output of 'scontrol show<br>
job JOBID'?<br>
<br>
What are the node definitions in slurm.conf, and the lines in gres.conf?<br>
<br>
Are the nodes all the same host platform (motherboard)?<br>
<br>
We have P100s, TitanVs, Titan RTXs, Quadro RTX 6000s, 3090s, V100s, DGX 1s,<br>
A6000s, and A40s, with a mix of single and dual-root platforms, and haven't<br>
seen this problem with SLURM 20.02.6 or earlier versions.<br>
<br>
On Thu, Aug 19, 2021 at 8:38 PM Andrey Malyutin <<a href="mailto:malyutinag@gmail.com" target="_blank">malyutinag@gmail.com</a>><br>
wrote:<br>
<br>
> Hello,<br>
><br>
> We are in the process of finishing up the setup of a cluster with 3 nodes,<br>
> 4 GPUs each. One node has RTX3090s and the other 2 have RTX6000s.Any job<br>
> asking for 1 GPU in the submission script will wait to run on the 3090<br>
> node, no matter resource availability. Same job requesting 2 or more GPUs<br>
> will run on any node. I don't even know where to begin troubleshooting this<br>
> issue; entries for the 3 nodes are effectively identical in slurm.conf. Any<br>
> help would be appreciated. (If helpful - this cluster is used for<br>
> structural biology, with cryosparc and relion packages).<br>
><br>
> Thank you,<br>
> Andrey<br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.schedmd.com/pipermail/slurm-users/attachments/20210819/4e2636a0/attachment-0001.htm" rel="noreferrer" target="_blank">http://lists.schedmd.com/pipermail/slurm-users/attachments/20210819/4e2636a0/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
Message: 5<br>
Date: Fri, 20 Aug 2021 10:31:40 +0200<br>
From: Durai Arasan <<a href="mailto:arasan.durai@gmail.com" target="_blank">arasan.durai@gmail.com</a>><br>
To: Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
Subject: [slurm-users] jobs stuck in "CG" state<br>
Message-ID:<br>
<CA+WZHCZsT4OiL9p3i9BfYArERYzqhyM9eNrYH=<a href="mailto:cR7cWLEPwcEw@mail.gmail.com" target="_blank">cR7cWLEPwcEw@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hello!<br>
<br>
We have a huge number of jobs stuck in CG state from a user who probably<br>
wrote code with bad I/O. "scancel" does not make them go away. Is there a<br>
way for admins to get rid of these jobs without draining and rebooting the<br>
nodes. I read somewhere that killing the respective slurmstepd process will<br>
do the job. Is this possible? Any other solutions? Also are there any<br>
parameters in slurm.conf one can set to manage such situations better?<br>
<br>
Best,<br>
Durai<br>
MPI T?bingen<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.schedmd.com/pipermail/slurm-users/attachments/20210820/f34971c1/attachment.htm" rel="noreferrer" target="_blank">http://lists.schedmd.com/pipermail/slurm-users/attachments/20210820/f34971c1/attachment.htm</a>><br>
<br>
End of slurm-users Digest, Vol 46, Issue 20<br>
*******************************************<br>
</blockquote></div></div>