<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hey SLURM user group,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We're seeing intermittent issues with timeouts contacting the controller / slurmctld using sbatch in a non-interactive / non-blocking fashion, even with MessageTimeout increased to 60s. Our slurmctld does not have a slurmdbd configured,
and CPU load / memory usage is barely registering. We're not seeing latency in storage I/O on the controller. In this specific cluster, we are running less than 50 nodes, and are not submitting more than 10's of jobs per second.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any ideas on how we can troubleshoot this issue?<o:p></o:p></p>
<p class="MsoNormal">We have done packet captures on both the controller and the submission nodes, and see no obvious issues with the network.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What do you have your MessageTimeout set to?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># sinfo --version<o:p></o:p></p>
<p class="MsoNormal">slurm 18.08.9<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># scontrol | show config<o:p></o:p></p>
<p class="MsoPlainText">Configuration data as of 2023-11-17T10:14:25 AccountingStorageBackupHost = (null) AccountingStorageEnforce = none<o:p></o:p></p>
<p class="MsoPlainText">AccountingStorageHost = localhost<o:p></o:p></p>
<p class="MsoPlainText">AccountingStorageLoc = <<REDACTED>>/accounting.log<o:p></o:p></p>
<p class="MsoPlainText">AccountingStoragePort = 0<o:p></o:p></p>
<p class="MsoPlainText">AccountingStorageTRES = cpu,mem,energy,node,billing,fs/disk,vmem,pages<o:p></o:p></p>
<p class="MsoPlainText">AccountingStorageType = accounting_storage/filetxt<o:p></o:p></p>
<p class="MsoPlainText">AccountingStorageUser = root<o:p></o:p></p>
<p class="MsoPlainText">AccountingStoreJobComment = Yes<o:p></o:p></p>
<p class="MsoPlainText">AcctGatherEnergyType = acct_gather_energy/none<o:p></o:p></p>
<p class="MsoPlainText">AcctGatherFilesystemType = acct_gather_filesystem/none AcctGatherInterconnectType = acct_gather_interconnect/none<o:p></o:p></p>
<p class="MsoPlainText">AcctGatherNodeFreq = 10 sec<o:p></o:p></p>
<p class="MsoPlainText">AcctGatherProfileType = acct_gather_profile/none<o:p></o:p></p>
<p class="MsoPlainText">AllowSpecResourcesUsage = 0<o:p></o:p></p>
<p class="MsoPlainText">AuthInfo = (null)<o:p></o:p></p>
<p class="MsoPlainText">AuthType = auth/none<o:p></o:p></p>
<p class="MsoPlainText">BatchStartTimeout = 240 sec<o:p></o:p></p>
<p class="MsoPlainText">BOOT_TIME = 2023-11-15T14:24:09<o:p></o:p></p>
<p class="MsoPlainText">BurstBufferType = (null)<o:p></o:p></p>
<p class="MsoPlainText">CheckpointType = checkpoint/none<o:p></o:p></p>
<p class="MsoPlainText">ClusterName = <<REDACTED>>/<o:p></o:p></p>
<p class="MsoPlainText">CommunicationParameters = (null)<o:p></o:p></p>
<p class="MsoPlainText">CompleteWait = 0 sec<o:p></o:p></p>
<p class="MsoPlainText">CoreSpecPlugin = core_spec/none<o:p></o:p></p>
<p class="MsoPlainText">CpuFreqDef = Unknown<o:p></o:p></p>
<p class="MsoPlainText">CpuFreqGovernors = Performance,OnDemand,UserSpace<o:p></o:p></p>
<p class="MsoPlainText">CryptoType = crypto/openssl<o:p></o:p></p>
<p class="MsoPlainText">DebugFlags = (null)<o:p></o:p></p>
<p class="MsoPlainText">DefMemPerCPU = 1024<o:p></o:p></p>
<p class="MsoPlainText">DisableRootJobs = No<o:p></o:p></p>
<p class="MsoPlainText">EioTimeout = 60<o:p></o:p></p>
<p class="MsoPlainText">EnforcePartLimits = NO<o:p></o:p></p>
<p class="MsoPlainText">Epilog = (null)<o:p></o:p></p>
<p class="MsoPlainText">EpilogMsgTime = 2000 usec<o:p></o:p></p>
<p class="MsoPlainText">EpilogSlurmctld = (null)<o:p></o:p></p>
<p class="MsoPlainText">ExtSensorsType = ext_sensors/none<o:p></o:p></p>
<p class="MsoPlainText">ExtSensorsFreq = 0 sec<o:p></o:p></p>
<p class="MsoPlainText">FairShareDampeningFactor = 1<o:p></o:p></p>
<p class="MsoPlainText">FastSchedule = 1<o:p></o:p></p>
<p class="MsoPlainText">FederationParameters = (null)<o:p></o:p></p>
<p class="MsoPlainText">FirstJobId = 1<o:p></o:p></p>
<p class="MsoPlainText">GetEnvTimeout = 2 sec<o:p></o:p></p>
<p class="MsoPlainText">GresTypes = (null)<o:p></o:p></p>
<p class="MsoPlainText">GroupUpdateForce = 1<o:p></o:p></p>
<p class="MsoPlainText">GroupUpdateTime = 600 sec<o:p></o:p></p>
<p class="MsoPlainText">HASH_VAL = Match<o:p></o:p></p>
<p class="MsoPlainText">HealthCheckInterval = 60 sec<o:p></o:p></p>
<p class="MsoPlainText">HealthCheckNodeState = ANY<o:p></o:p></p>
<p class="MsoPlainText">HealthCheckProgram = /bin/ls<o:p></o:p></p>
<p class="MsoPlainText">InactiveLimit = 3600 sec<o:p></o:p></p>
<p class="MsoPlainText">JobAcctGatherFrequency = task=5<o:p></o:p></p>
<p class="MsoPlainText">JobAcctGatherType = jobacct_gather/none<o:p></o:p></p>
<p class="MsoPlainText">JobAcctGatherParams = NoOverMemoryKill<o:p></o:p></p>
<p class="MsoPlainText">JobCheckpointDir = <<REDACTED>>/checkpoint<o:p></o:p></p>
<p class="MsoPlainText">JobCompHost = localhost<o:p></o:p></p>
<p class="MsoPlainText">JobCompLoc = <<REDACTED>>/job_completions.log<o:p></o:p></p>
<p class="MsoPlainText">JobCompPort = 0<o:p></o:p></p>
<p class="MsoPlainText">JobCompType = jobcomp/filetxt<o:p></o:p></p>
<p class="MsoPlainText">JobCompUser = root<o:p></o:p></p>
<p class="MsoPlainText">JobContainerType = job_container/none<o:p></o:p></p>
<p class="MsoPlainText">JobCredentialPrivateKey = /etc/slurm/pki/slurm.key<o:p></o:p></p>
<p class="MsoPlainText">JobCredentialPublicCertificate = /etc/slurm/pki/slurm.crt<o:p></o:p></p>
<p class="MsoPlainText">JobDefaults = (null)<o:p></o:p></p>
<p class="MsoPlainText">JobFileAppend = 1<o:p></o:p></p>
<p class="MsoPlainText">JobRequeue = 1<o:p></o:p></p>
<p class="MsoPlainText">JobSubmitPlugins = (null)<o:p></o:p></p>
<p class="MsoPlainText">KeepAliveTime = SYSTEM_DEFAULT<o:p></o:p></p>
<p class="MsoPlainText">KillOnBadExit = 1<o:p></o:p></p>
<p class="MsoPlainText">KillWait = 60 sec<o:p></o:p></p>
<p class="MsoPlainText">LaunchParameters = slurmstepd_memlock<o:p></o:p></p>
<p class="MsoPlainText">LaunchType = launch/slurm<o:p></o:p></p>
<p class="MsoPlainText">Layouts = <o:p></o:p></p>
<p class="MsoPlainText">Licenses = <<REDACTED>><o:p></o:p></p>
<p class="MsoPlainText">LicensesUsed = <<REDACTED>><o:p></o:p></p>
<p class="MsoPlainText">LogTimeFormat = iso8601_ms<o:p></o:p></p>
<p class="MsoPlainText">MailDomain = <<REDACTED>><o:p></o:p></p>
<p class="MsoPlainText">MailProg = /bin/mail<o:p></o:p></p>
<p class="MsoPlainText">MaxArraySize = 1001<o:p></o:p></p>
<p class="MsoPlainText">MaxJobCount = 10000<o:p></o:p></p>
<p class="MsoPlainText">MaxJobId = 67043328<o:p></o:p></p>
<p class="MsoPlainText">MaxMemPerNode = UNLIMITED<o:p></o:p></p>
<p class="MsoPlainText">MaxStepCount = 40000<o:p></o:p></p>
<p class="MsoPlainText">MaxTasksPerNode = 512<o:p></o:p></p>
<p class="MsoPlainText">MCSPlugin = mcs/none<o:p></o:p></p>
<p class="MsoPlainText">MCSParameters = (null)<o:p></o:p></p>
<p class="MsoPlainText">MemLimitEnforce = No<o:p></o:p></p>
<p class="MsoPlainText">MessageTimeout = 60 sec<o:p></o:p></p>
<p class="MsoPlainText">MinJobAge = 600 sec<o:p></o:p></p>
<p class="MsoPlainText">MpiDefault = none<o:p></o:p></p>
<p class="MsoPlainText">MpiParams = (null)<o:p></o:p></p>
<p class="MsoPlainText">MsgAggregationParams = (null)<o:p></o:p></p>
<p class="MsoPlainText">NEXT_JOB_ID = 7262059<o:p></o:p></p>
<p class="MsoPlainText">NodeFeaturesPlugins = (null)<o:p></o:p></p>
<p class="MsoPlainText">OverTimeLimit = 10 min<o:p></o:p></p>
<p class="MsoPlainText">PluginDir = <<REDACTED>>/slurm<o:p></o:p></p>
<p class="MsoPlainText">PlugStackConfig = /etc/slurm/plugstack.conf<o:p></o:p></p>
<p class="MsoPlainText">PowerParameters = (null)<o:p></o:p></p>
<p class="MsoPlainText">PowerPlugin = <o:p></o:p></p>
<p class="MsoPlainText">PreemptMode = OFF<o:p></o:p></p>
<p class="MsoPlainText">PreemptType = preempt/none<o:p></o:p></p>
<p class="MsoPlainText">PriorityParameters = (null)<o:p></o:p></p>
<p class="MsoPlainText">PriorityDecayHalfLife = 7-00:00:00<o:p></o:p></p>
<p class="MsoPlainText">PriorityCalcPeriod = 00:05:00<o:p></o:p></p>
<p class="MsoPlainText">PriorityFavorSmall = Yes<o:p></o:p></p>
<p class="MsoPlainText">PriorityFlags = <o:p></o:p></p>
<p class="MsoPlainText">PriorityMaxAge = 1-00:00:00<o:p></o:p></p>
<p class="MsoPlainText">PriorityUsageResetPeriod = NONE<o:p></o:p></p>
<p class="MsoPlainText">PriorityType = priority/multifactor<o:p></o:p></p>
<p class="MsoPlainText">PriorityWeightAge = 1000<o:p></o:p></p>
<p class="MsoPlainText">PriorityWeightFairShare = 0<o:p></o:p></p>
<p class="MsoPlainText">PriorityWeightJobSize = 500<o:p></o:p></p>
<p class="MsoPlainText">PriorityWeightPartition = 1000<o:p></o:p></p>
<p class="MsoPlainText">PriorityWeightQOS = 0<o:p></o:p></p>
<p class="MsoPlainText">PriorityWeightTRES = (null)<o:p></o:p></p>
<p class="MsoPlainText">PrivateData = none<o:p></o:p></p>
<p class="MsoPlainText">ProctrackType = proctrack/linuxproc<o:p></o:p></p>
<p class="MsoPlainText">Prolog = (null)<o:p></o:p></p>
<p class="MsoPlainText">PrologEpilogTimeout = 2700<o:p></o:p></p>
<p class="MsoPlainText">PrologSlurmctld = (null)<o:p></o:p></p>
<p class="MsoPlainText">PrologFlags = (null)<o:p></o:p></p>
<p class="MsoPlainText">PropagatePrioProcess = 0<o:p></o:p></p>
<p class="MsoPlainText">PropagateResourceLimits = NONE<o:p></o:p></p>
<p class="MsoPlainText">PropagateResourceLimitsExcept = (null)<o:p></o:p></p>
<p class="MsoPlainText">RebootProgram = (null)<o:p></o:p></p>
<p class="MsoPlainText">ReconfigFlags = (null)<o:p></o:p></p>
<p class="MsoPlainText">RequeueExit = (null)<o:p></o:p></p>
<p class="MsoPlainText">RequeueExitHold = (null)<o:p></o:p></p>
<p class="MsoPlainText">ResumeFailProgram = (null)<o:p></o:p></p>
<p class="MsoPlainText">ResumeProgram = (null)<o:p></o:p></p>
<p class="MsoPlainText">ResumeRate = 300 nodes/min<o:p></o:p></p>
<p class="MsoPlainText">ResumeTimeout = 60 sec<o:p></o:p></p>
<p class="MsoPlainText">ResvEpilog = (null)<o:p></o:p></p>
<p class="MsoPlainText">ResvOverRun = 0 min<o:p></o:p></p>
<p class="MsoPlainText">ResvProlog = (null)<o:p></o:p></p>
<p class="MsoPlainText">ReturnToService = 2<o:p></o:p></p>
<p class="MsoPlainText">RoutePlugin = route/default<o:p></o:p></p>
<p class="MsoPlainText">SallocDefaultCommand = (null)<o:p></o:p></p>
<p class="MsoPlainText">SbcastParameters = (null)<o:p></o:p></p>
<p class="MsoPlainText">SchedulerParameters = nohold_on_prolog_fail<o:p></o:p></p>
<p class="MsoPlainText">SchedulerTimeSlice = 30 sec<o:p></o:p></p>
<p class="MsoPlainText">SchedulerType = sched/backfill<o:p></o:p></p>
<p class="MsoPlainText">SelectType = select/cons_res<o:p></o:p></p>
<p class="MsoPlainText">SelectTypeParameters = CR_CPU_MEMORY,CR_LLN<o:p></o:p></p>
<p class="MsoPlainText">SlurmUser = slurm(64030)<o:p></o:p></p>
<p class="MsoPlainText">SlurmctldAddr = (null)<o:p></o:p></p>
<p class="MsoPlainText">SlurmctldDebug = info<o:p></o:p></p>
<p class="MsoPlainText">SlurmctldHost[0] = <<REDACTED>><o:p></o:p></p>
<p class="MsoPlainText">SlurmctldLogFile = <<REDACTED>>/slurmctld.log<o:p></o:p></p>
<p class="MsoPlainText">SlurmctldPort = 6817<o:p></o:p></p>
<p class="MsoPlainText">SlurmctldSyslogDebug = debug2<o:p></o:p></p>
<p class="MsoPlainText">SlurmctldPrimaryOffProg = (null)<o:p></o:p></p>
<p class="MsoPlainText">SlurmctldPrimaryOnProg = (null)<o:p></o:p></p>
<p class="MsoPlainText">SlurmctldTimeout = 240 sec<o:p></o:p></p>
<p class="MsoPlainText">SlurmctldParameters = (null)<o:p></o:p></p>
<p class="MsoPlainText">SlurmdDebug = info<o:p></o:p></p>
<p class="MsoPlainText">SlurmdLogFile = /var/log/slurm/slurmd.log<o:p></o:p></p>
<p class="MsoPlainText">SlurmdParameters = (null)<o:p></o:p></p>
<p class="MsoPlainText">SlurmdPidFile = /run/slurmd.pid<o:p></o:p></p>
<p class="MsoPlainText">SlurmdPort = 6818<o:p></o:p></p>
<p class="MsoPlainText">SlurmdSpoolDir = /var/lib/slurm/slurmd<o:p></o:p></p>
<p class="MsoPlainText">SlurmdSyslogDebug = error<o:p></o:p></p>
<p class="MsoPlainText">SlurmdTimeout = 300 sec<o:p></o:p></p>
<p class="MsoPlainText">SlurmdUser = root(0)<o:p></o:p></p>
<p class="MsoPlainText">SlurmSchedLogFile = <<REDACTED>>/slurmsched.log<o:p></o:p></p>
<p class="MsoPlainText">SlurmSchedLogLevel = 3<o:p></o:p></p>
<p class="MsoPlainText">SlurmctldPidFile = /run/slurmctld.pid<o:p></o:p></p>
<p class="MsoPlainText">SlurmctldPlugstack = (null)<o:p></o:p></p>
<p class="MsoPlainText">SLURM_CONF = /etc/slurm/slurm.conf<o:p></o:p></p>
<p class="MsoPlainText">SLURM_VERSION = 18.08.9<o:p></o:p></p>
<p class="MsoPlainText">SrunEpilog = (null)<o:p></o:p></p>
<p class="MsoPlainText">SrunPortRange = 0-0<o:p></o:p></p>
<p class="MsoPlainText">SrunProlog = (null)<o:p></o:p></p>
<p class="MsoPlainText">StateSaveLocation = <<REDACTED>>/slurmctld<o:p></o:p></p>
<p class="MsoPlainText">SuspendExcNodes = (null)<o:p></o:p></p>
<p class="MsoPlainText">SuspendExcParts = (null)<o:p></o:p></p>
<p class="MsoPlainText">SuspendProgram = (null)<o:p></o:p></p>
<p class="MsoPlainText">SuspendRate = 60 nodes/min<o:p></o:p></p>
<p class="MsoPlainText">SuspendTime = NONE<o:p></o:p></p>
<p class="MsoPlainText">SuspendTimeout = 30 sec<o:p></o:p></p>
<p class="MsoPlainText">SwitchType = switch/none<o:p></o:p></p>
<p class="MsoPlainText">TaskEpilog = (null)<o:p></o:p></p>
<p class="MsoPlainText">TaskPlugin = task/none<o:p></o:p></p>
<p class="MsoPlainText">TaskPluginParam = (null type)<o:p></o:p></p>
<p class="MsoPlainText">TaskProlog = (null)<o:p></o:p></p>
<p class="MsoPlainText">TCPTimeout = 60 sec<o:p></o:p></p>
<p class="MsoPlainText">TmpFS = /tmp<o:p></o:p></p>
<p class="MsoPlainText">TopologyParam = (null)<o:p></o:p></p>
<p class="MsoPlainText">TopologyPlugin = topology/none<o:p></o:p></p>
<p class="MsoPlainText">TrackWCKey = No<o:p></o:p></p>
<p class="MsoPlainText">TreeWidth = 50<o:p></o:p></p>
<p class="MsoPlainText">UsePam = 0<o:p></o:p></p>
<p class="MsoPlainText">UnkillableStepProgram = (null)<o:p></o:p></p>
<p class="MsoPlainText">UnkillableStepTimeout = 240 sec<o:p></o:p></p>
<p class="MsoPlainText">VSizeFactor = 0 percent<o:p></o:p></p>
<p class="MsoPlainText">WaitTime = 300 sec<o:p></o:p></p>
<p class="MsoPlainText">X11Parameters = (null)<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Slurmctld(primary) at <<REDACTED>> is UP<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt">Anthony Altemara<o:p></o:p></span></b></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt">IT Infrastructure Associate Director<o:p></o:p></span></i></p>
<p class="MsoNormal"><span style="color:black"><img width="192" height="31" style="width:2.0in;height:.3229in" id="Picture_x0020_1" src="cid:image001.png@01DA193C.5E72E090" alt="cid1603022056*image003.png@01D8384E.C479F0C0"></span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F3864"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt">Office: +1 919.491.2220<span style="color:black"><o:p></o:p></span></span></p>
<p class="MsoNormal"><b><u><span style="font-size:9.0pt;color:#3333FF"><a href="mailto:Anthony.Altemara@Q2LabSolutions.com"><span style="color:blue">Anthony.Altemara@Q2LabSolutions.com</span></a></span></u></b><b><span style="font-size:9.0pt;color:#3333FF">
</span></b><span style="font-size:9.0pt">| <b><u><span style="color:#0563C1"><a href="http://www.q2labsolutions.com/"><span style="color:blue">www.Q2LabSolutions.com</span></a><o:p></o:p></span></u></b></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<br>
________________________________________<br>
<span style="font-size:9pt; font-family: 'Calibri','times new roman','garamond',serif; color:#928E8E;"><b>IMPORTANT</b> - PLEASE READ: This electronic message, including its attachments, is CONFIDENTIAL and may contain PROPRIETARY or LEGALLY PRIVILEGED or PROTECTED
information and is intended for the authorized recipient of the sender. If you are not the intended recipient, you are hereby notified that any use, disclosure, copying, or distribution of this message or any of the information included in it is unauthorized
and strictly prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and permanently delete this message and its attachments, along with any copies thereof, from all locations received (e.g., computer, mobile
device, etc.). To the extent permitted by law, we may monitor electronic communications for the purposes of ensuring compliance with our legal and regulatory obligations and internal policies. We may also collect email traffic headers for analyzing patterns
of network traffic and managing client relationships. For further information see our
<a href="https://www.iqvia.com/about-us/privacy/privacy-policy">privacy-policy</a>. Thank you.
</span>
<p></p>
</body>
</html>