<div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:monospace">Hi</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">It might be useful to see the configuration of the partition and how the QOS is set up... but at first blush I suspect you may need to set OverPartQOS (<a href="https://slurm.schedmd.com/resource_limits.html">https://slurm.schedmd.com/resource_limits.html</a>) to get the QOS limit to take precedence over the limit in the partition. However, the "reason" should be different if that were the case.</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">Look at that, maybe send the QOS and partition config.</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace"> - Michael</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace"><br></div><input name="virtru-metadata" type="hidden" value="{"email-policy":{"state":"closed","expirationUnit":"days","disableCopyPaste":false,"disablePrint":false,"disableForwarding":false,"enableNoauth":false,"expires":false,"isManaged":false},"attachments":{},"compose-id":"10","compose-window":{"secure":false}}"></div></div><br><div class="gmail_quote" style=""><div dir="ltr" class="gmail_attr">On Tue, Mar 5, 2019 at 7:40 AM Matthew BETTINGER <<a href="mailto:matthew.bettinger@external.total.com">matthew.bettinger@external.total.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hey slurm gurus. We have been trying to enable slurm QOS on a cray system here off and on for quite a while but can never get it working. Every time we try to enable QOS we disrupt the cluster and users and have to fall back. I'm not sure what we are doing wrong. We run a pretty open system here since we are a research group but there are time where we need to let a user run a job to exceed a partition limit. In lieu of using QOs the only other way we have figured out how to do this is create a new partition and push out the modified slurm.conf. It's a hassle.<br>
<br>
I'm not sure what information is needed exactly to troubleshoot this but I understand to enable QOS we need to enable this line in slurm.conf<br>
<br>
AccountingStorageEnforce=limits,qos<br>
<br>
Every time we attempt this no one can submit a job, slurm says waiting on resources I believe.<br>
<br>
We have accounting enabled and everyone is a member of the default qos group "normal". <br>
<br>
Configuration data as of 2019-03-05T09:36:19<br>
AccountingStorageBackupHost = (null)<br>
AccountingStorageEnforce = none<br>
AccountingStorageHost = hickory-1<br>
AccountingStorageLoc = N/A<br>
AccountingStoragePort = 6819<br>
AccountingStorageTRES = cpu,mem,energy,node,bb/cray,gres/craynetwork,gres/gpu<br>
AccountingStorageType = accounting_storage/slurmdbd<br>
AccountingStorageUser = N/A<br>
AccountingStoreJobComment = Yes<br>
AcctGatherEnergyType = acct_gather_energy/rapl<br>
AcctGatherFilesystemType = acct_gather_filesystem/none<br>
AcctGatherInfinibandType = acct_gather_infiniband/none<br>
AcctGatherNodeFreq = 30 sec<br>
AcctGatherProfileType = acct_gather_profile/none<br>
AllowSpecResourcesUsage = 1<br>
AuthInfo = (null)<br>
AuthType = auth/munge<br>
BackupAddr = hickory-2<br>
BackupController = hickory-2<br>
BatchStartTimeout = 10 sec<br>
BOOT_TIME = 2019-03-04T16:11:55<br>
BurstBufferType = burst_buffer/cray<br>
CacheGroups = 0<br>
CheckpointType = checkpoint/none<br>
ChosLoc = (null)<br>
ClusterName = hickory<br>
CompleteWait = 0 sec<br>
ControlAddr = hickory-1<br>
ControlMachine = hickory-1<br>
CoreSpecPlugin = cray<br>
CpuFreqDef = Performance<br>
CpuFreqGovernors = Performance,OnDemand<br>
CryptoType = crypto/munge<br>
DebugFlags = (null)<br>
DefMemPerNode = UNLIMITED<br>
DisableRootJobs = No<br>
EioTimeout = 60<br>
EnforcePartLimits = NO<br>
Epilog = (null)<br>
EpilogMsgTime = 2000 usec<br>
EpilogSlurmctld = (null)<br>
ExtSensorsType = ext_sensors/none<br>
ExtSensorsFreq = 0 sec<br>
FairShareDampeningFactor = 1<br>
FastSchedule = 0<br>
FirstJobId = 1<br>
GetEnvTimeout = 2 sec<br>
GresTypes = gpu,craynetwork<br>
GroupUpdateForce = 1<br>
GroupUpdateTime = 600 sec<br>
HASH_VAL = Match<br>
HealthCheckInterval = 0 sec<br>
HealthCheckNodeState = ANY<br>
HealthCheckProgram = (null)<br>
InactiveLimit = 0 sec<br>
JobAcctGatherFrequency = 30<br>
JobAcctGatherType = jobacct_gather/linux<br>
JobAcctGatherParams = (null)<br>
JobCheckpointDir = /var/slurm/checkpoint<br>
JobCompHost = localhost<br>
JobCompLoc = /var/log/slurm_jobcomp.log<br>
JobCompPort = 0<br>
JobCompType = jobcomp/none<br>
JobCompUser = root<br>
JobContainerType = job_container/cncu<br>
JobCredentialPrivateKey = (null)<br>
JobCredentialPublicCertificate = (null)<br>
JobFileAppend = 0<br>
JobRequeue = 1<br>
JobSubmitPlugins = cray<br>
KeepAliveTime = SYSTEM_DEFAULT<br>
KillOnBadExit = 1<br>
KillWait = 30 sec<br>
LaunchParameters = (null)<br>
LaunchType = launch/slurm<br>
Layouts =<br>
Licenses = (null)<br>
LicensesUsed = (null)<br>
MailDomain = (null)<br>
MailProg = /bin/mail<br>
MaxArraySize = 1001<br>
MaxJobCount = 10000<br>
MaxJobId = 67043328<br>
MaxMemPerCPU = 128450<br>
MaxStepCount = 40000<br>
MaxTasksPerNode = 512<br>
MCSPlugin = mcs/none<br>
MCSParameters = (null)<br>
MemLimitEnforce = Yes<br>
MessageTimeout = 10 sec<br>
MinJobAge = 300 sec<br>
MpiDefault = none<br>
MpiParams = ports=20000-32767<br>
MsgAggregationParams = (null)<br>
NEXT_JOB_ID = 244342<br>
NodeFeaturesPlugins = (null)<br>
OverTimeLimit = 0 min<br>
PluginDir = /opt/slurm/17.02.6/lib64/slurm<br>
PlugStackConfig = /etc/opt/slurm/plugstack.conf<br>
PowerParameters = (null)<br>
PowerPlugin =<br>
PreemptMode = OFF<br>
PreemptType = preempt/none<br>
PriorityParameters = (null)<br>
PriorityDecayHalfLife = 7-00:00:00<br>
PriorityCalcPeriod = 00:05:00<br>
PriorityFavorSmall = No<br>
PriorityFlags =<br>
PriorityMaxAge = 7-00:00:00<br>
PriorityUsageResetPeriod = NONE<br>
PriorityType = priority/multifactor<br>
PriorityWeightAge = 0<br>
PriorityWeightFairShare = 0<br>
PriorityWeightJobSize = 0<br>
PriorityWeightPartition = 0<br>
PriorityWeightQOS = 0<br>
PriorityWeightTRES = (null)<br>
PrivateData = none<br>
ProctrackType = proctrack/cray<br>
Prolog = (null)<br>
PrologEpilogTimeout = 65534<br>
PrologSlurmctld = (null)<br>
PrologFlags = (null)<br>
PropagatePrioProcess = 0<br>
PropagateResourceLimits = (null)<br>
PropagateResourceLimitsExcept = AS<br>
RebootProgram = (null)<br>
ReconfigFlags = (null)<br>
RequeueExit = (null)<br>
RequeueExitHold = (null)<br>
ResumeProgram = (null)<br>
ResumeRate = 300 nodes/min<br>
ResumeTimeout = 60 sec<br>
ResvEpilog = (null)<br>
ResvOverRun = 0 min<br>
ResvProlog = (null)<br>
ReturnToService = 2<br>
RoutePlugin = route/default<br>
SallocDefaultCommand = (null)<br>
SbcastParameters = (null)<br>
SchedulerParameters = (null)<br>
SchedulerTimeSlice = 30 sec<br>
SchedulerType = sched/backfill<br>
SelectType = select/cray<br>
SelectTypeParameters = CR_CORE_MEMORY,OTHER_CONS_RES,NHC_ABSOLUTELY_NO<br>
SlurmUser = root(0)<br>
SlurmctldDebug = info<br>
SlurmctldLogFile = /var/spool/slurm/slurmctld.log<br>
SlurmctldPort = 6817<br>
SlurmctldTimeout = 120 sec<br>
SlurmdDebug = info<br>
SlurmdLogFile = /var/spool/slurmd/%h.log<br>
SlurmdPidFile = /var/spool/slurmd/slurmd.pid<br>
SlurmdPlugstack = (null)<br>
SlurmdPort = 6818<br>
SlurmdSpoolDir = /var/spool/slurmd<br>
SlurmdTimeout = 300 sec<br>
SlurmdUser = root(0)<br>
SlurmSchedLogFile = (null)<br>
SlurmSchedLogLevel = 0<br>
SlurmctldPidFile = /var/spool/slurm/slurmctld.pid<br>
SlurmctldPlugstack = (null)<br>
SLURM_CONF = /etc/opt/slurm/slurm.conf<br>
SLURM_VERSION = 17.02.6<br>
SrunEpilog = (null)<br>
SrunPortRange = 0-0<br>
SrunProlog = (null)<br>
StateSaveLocation = /apps/cluster/hickory/slurm/<br>
SuspendExcNodes = (null)<br>
SuspendExcParts = (null)<br>
SuspendProgram = (null)<br>
SuspendRate = 60 nodes/min<br>
SuspendTime = NONE<br>
SuspendTimeout = 30 sec<br>
SwitchType = switch/cray<br>
TaskEpilog = (null)<br>
TaskPlugin = task/cray,task/affinity,task/cgroup<br>
TaskPluginParam = (null type)<br>
TaskProlog = (null)<br>
TCPTimeout = 2 sec<br>
TmpFS = /tmp<br>
TopologyParam = (null)<br>
TopologyPlugin = topology/none<br>
TrackWCKey = No<br>
TreeWidth = 50<br>
UsePam = 0<br>
UnkillableStepProgram = (null)<br>
UnkillableStepTimeout = 60 sec<br>
VSizeFactor = 0 percent<br>
WaitTime = 0 sec<br>
<br>
Slurmctld(primary/backup) at hickory-1/hickory-2 are UP/UP<br>
<br>
<br>
<br>
</blockquote></div></div>