<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Progress on getting slurmd to start under cgroupv2 <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Issue: slurmd 22.05.6 will not start when using cgroupv2 <o:p>
</o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Expected result: even after reboot slurmd will start up without needing to manually add lines to /sys/fs/cgroup files.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When started as service the error is:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"># systemctl status slurmd<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">* slurmd.service - Slurm node daemon<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"> Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"> Drop-In: /etc/systemd/system/slurmd.service.d<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"> `-extendUnit.conf<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"> Active: failed (Result: exit-code) since Tue 2023-07-11 10:29:23 EDT; 2s ago<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"> Process: 11395 ExecStart=/usr/sbin/slurmd -D -s $SLURMD_OPTIONS (code=exited, status=1/FAILURE)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">Main PID: 11395 (code=exited, status=1/FAILURE)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">Jul 11 10:29:23 g1803jles01.ll.unc.edu systemd[1]: Started Slurm node daemon.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">Jul 11 10:29:23 g1803jles01.ll.unc.edu slurmd[11395]: slurmd: slurmd version 22.05.6 started<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">Jul 11 10:29:23 g1803jles01.ll.unc.edu systemd[1]: slurmd.service: Main process exited, code=exited, status=1/FAILURE<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">Jul 11 10:29:23 g1803jles01.ll.unc.edu systemd[1]: slurmd.service: Failed with result 'exit-code'.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When started at the command line the output is:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"># slurmd -D -vvv 2>&1 |egrep error<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: Controller cpuset is not enabled!<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: Controller cpu is not enabled!<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: Controller cpuset is not enabled!<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: Controller cpu is not enabled!<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: Controller cpuset is not enabled!<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: Controller cpu is not enabled!<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: Controller cpuset is not enabled!<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: Controller cpu is not enabled!<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: cpu cgroup controller is not available.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: There's an issue initializing memory or cpu controller<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: Couldn't load specified plugin name for jobacct_gather/cgroup: Plugin init() callback failed<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">slurmd: error: cannot create jobacct_gather context for jobacct_gather/cgroup<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Steps to mitigate the issue:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">While the following steps do not solve the issue, they do get the system in a state such that slurmd will start, at least until next reboot. The re-install slurm-slurmd is a one-time step to ensure that local service modifications are
out of the picture. <b><i>Currently, even after reboot the cgroup echo steps are necessary at a minimum.</i></b><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">#!/bin/bash<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">/usr/bin/dnf -y reinstall slurm-slurmd<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">systemctl daemon-reload<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">/usr/bin/pkill -f '/usr/sbin/slurmstepd infinity'<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">systemctl enable slurmd<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">systemctl stop dcismeng.service && \<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><b><i><span style="font-size:9.0pt;font-family:Consolas">/usr/bin/echo +cpu +cpuset +memory >> /sys/fs/cgroup/cgroup.subtree_control && \<o:p></o:p></span></i></b></p>
<p class="MsoNormal" style="margin-left:.5in"><b><i><span style="font-size:9.0pt;font-family:Consolas">/usr/bin/echo +cpu +cpuset +memory >> /sys/fs/cgroup/system.slice/cgroup.subtree_control && \<o:p></o:p></span></i></b></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">systemctl start slurmd && \
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"> echo 'run this: systemctl start dcismeng'<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal">Environment:<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas"> <o:p>
</o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"># scontrol show config
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">Configuration data as of 2023-07-11T10:39:48<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AccountingStorageBackupHost = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AccountingStorageEnforce = associations,limits,qos,safe<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AccountingStorageHost = m1006<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AccountingStorageExternalHost = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AccountingStorageParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AccountingStoragePort = 6819<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AccountingStorageTRES = cpu,mem,energy,node,billing,fs/disk,vmem,pages,gres/gpu<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AccountingStorageType = accounting_storage/slurmdbd<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AccountingStorageUser = N/A<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AccountingStoreFlags = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AcctGatherEnergyType = acct_gather_energy/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AcctGatherFilesystemType = acct_gather_filesystem/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AcctGatherInterconnectType = acct_gather_interconnect/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AcctGatherNodeFreq = 0 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AcctGatherProfileType = acct_gather_profile/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AllowSpecResourcesUsage = No<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AuthAltTypes = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AuthAltParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AuthInfo = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AuthType = auth/munge<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">BatchStartTimeout = 10 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">BcastExclude = /lib,/usr/lib,/lib64,/usr/lib64<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">BcastParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">BOOT_TIME = 2023-07-11T10:04:31<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">BurstBufferType = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">CliFilterPlugins = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ClusterName = ASlurmCluster<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">CommunicationParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">CompleteWait = 0 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">CoreSpecPlugin = core_spec/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">CpuFreqDef = Unknown<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">CpuFreqGovernors = OnDemand,Performance,UserSpace<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">CredType = cred/munge<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">DebugFlags = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">DefMemPerNode = UNLIMITED<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">DependencyParameters = kill_invalid_depend<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">DisableRootJobs = No<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">EioTimeout = 60<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">EnforcePartLimits = ANY<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">Epilog = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">EpilogMsgTime = 2000 usec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">EpilogSlurmctld = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ExtSensorsType = ext_sensors/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ExtSensorsFreq = 0 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">FairShareDampeningFactor = 1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">FederationParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">FirstJobId = 1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">GetEnvTimeout = 2 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">GresTypes = gpu<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">GpuFreqDef = high,memory=high<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">GroupUpdateForce = 1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">GroupUpdateTime = 600 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">HASH_VAL = Match<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">HealthCheckInterval = 0 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">HealthCheckNodeState = ANY<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">HealthCheckProgram = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">InactiveLimit = 65533 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">InteractiveStepOptions = --interactive --preserve-env --pty $SHELL<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobAcctGatherFrequency = task=15<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobAcctGatherType = jobacct_gather/cgroup<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobAcctGatherParams = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobCompHost = localhost<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobCompLoc = /var/log/slurm_jobcomp.log<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobCompPort = 0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobCompType = jobcomp/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobCompUser = root<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobContainerType = job_container/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobCredentialPrivateKey = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobCredentialPublicCertificate = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobDefaults = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobFileAppend = 0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobRequeue = 1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">JobSubmitPlugins = lua<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">KillOnBadExit = 0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">KillWait = 30 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">LaunchParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">LaunchType = launch/slurm<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">Licenses = mplus:1,nonmem:32<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">LogTimeFormat = iso8601_ms<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MailDomain = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MailProg = /bin/mail<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MaxArraySize = 90001<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MaxDBDMsgs = 701360<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MaxJobCount = 350000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MaxJobId = 67043328<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MaxMemPerNode = UNLIMITED<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MaxNodeCount = 340<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MaxStepCount = 40000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MaxTasksPerNode = 512<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MCSPlugin = mcs/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MCSParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MessageTimeout = 60 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MinJobAge = 300 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MpiDefault = none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MpiParams = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">NEXT_JOB_ID = 12286313<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">NodeFeaturesPlugins = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">OverTimeLimit = 0 min<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PluginDir = /usr/lib64/slurm<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PlugStackConfig = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PowerParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PowerPlugin =
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PreemptMode = OFF<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PreemptType = preempt/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PreemptExemptTime = 00:00:00<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PrEpParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PrEpPlugins = prep/script<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PrioritySiteFactorParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PrioritySiteFactorPlugin = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityDecayHalfLife = 14-00:00:00<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityCalcPeriod = 00:05:00<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityFavorSmall = No<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityFlags = SMALL_RELATIVE_TO_TIME,CALCULATE_RUNNING,MAX_TRES<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityMaxAge = 60-00:00:00<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityUsageResetPeriod = NONE<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityType = priority/multifactor<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityWeightAge = 10000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityWeightAssoc = 0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityWeightFairShare = 10000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityWeightJobSize = 1000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityWeightPartition = 1000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityWeightQOS = 1000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PriorityWeightTRES = CPU=1000,Mem=4000,GRES/gpu=3000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PrivateData = none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ProctrackType = proctrack/cgroup<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">Prolog = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PrologEpilogTimeout = 65534<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PrologSlurmctld = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PrologFlags = Alloc,Contain,X11<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PropagatePrioProcess = 0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PropagateResourceLimits = ALL<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">PropagateResourceLimitsExcept = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">RebootProgram = /usr/sbin/reboot<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ReconfigFlags = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">RequeueExit = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">RequeueExitHold = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ResumeFailProgram = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ResumeProgram = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ResumeRate = 300 nodes/min<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ResumeTimeout = 60 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ResvEpilog = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ResvOverRun = 0 min<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ResvProlog = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ReturnToService = 2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">RoutePlugin = route/default<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SchedulerParameters = batch_sched_delay=10,bf_continue,bf_max_job_part=1000,bf_max_job_test=10000,bf_max_job_user=100,bf_resolution=300,bf_window=10080,bf_yield_interval=1000000,default_queue_depth=1000,partition_job_depth=600,sched_min_interval=20000000,defer,max_rpc_cnt=80<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SchedulerTimeSlice = 30 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SchedulerType = sched/backfill<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ScronParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SelectType = select/cons_tres<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SelectTypeParameters = CR_CPU_MEMORY<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmUser = slurm(47)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldAddr = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldDebug = info<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldHost[0] = ASlurmCluster-sched(x.x.x.x)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldLogFile = /data/slurm/slurmctld.log<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldPort = 6820-6824<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldSyslogDebug = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldPrimaryOffProg = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldPrimaryOnProg = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldTimeout = 6000 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmdDebug = info<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmdLogFile = /var/log/slurm/slurmd.log<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmdParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmdPidFile = /var/run/slurmd.pid<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmdPort = 6818<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmdSpoolDir = /var/spool/slurmd<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmdSyslogDebug = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmdTimeout = 600 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmdUser = root(0)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmSchedLogFile = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmSchedLogLevel = 0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldPidFile = /var/run/slurmctld.pid<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SlurmctldPlugstack = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SLURM_CONF = /etc/slurm/slurm.conf<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SLURM_VERSION = 22.05.6<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SrunEpilog = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SrunPortRange = 0-0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SrunProlog = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">StateSaveLocation = /data/slurm/slurmctld<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SuspendExcNodes = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SuspendExcParts = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SuspendProgram = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SuspendRate = 60 nodes/min<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SuspendTime = INFINITE<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SuspendTimeout = 30 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SwitchParameters = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">SwitchType = switch/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">TaskEpilog = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">TaskPlugin = cgroup,affinity<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">TaskPluginParam = (null type)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">TaskProlog = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">TCPTimeout = 2 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">TmpFS = /tmp<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">TopologyParam = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">TopologyPlugin = topology/none<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">TrackWCKey = No<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">TreeWidth = 50<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">UsePam = No<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">UnkillableStepProgram = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">UnkillableStepTimeout = 600 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">VSizeFactor = 0 percent<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">WaitTime = 0 sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">X11Parameters = home_xauthority<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">Cgroup Support Configuration:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AllowedKmemSpace = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AllowedRAMSpace = 100.0%<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">AllowedSwapSpace = 1.0%<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">CgroupAutomount = yes<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">CgroupMountpoint = /sys/fs/cgroup<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">CgroupPlugin = cgroup/v2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ConstrainCores = yes<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ConstrainDevices = yes<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ConstrainKmemSpace = no<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ConstrainRAMSpace = yes<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">ConstrainSwapSpace = yes<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">IgnoreSystemd = no<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">IgnoreSystemdOnFailure = no<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MaxKmemPercent = 100.0%<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MaxRAMPercent = 100.0%<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MaxSwapPercent = 100.0%<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MemorySwappiness = (null)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MinKmemSpace = 30 MB<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">MinRAMSpace = 30 MB<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas">Slurmctld(primary) at ASlurmCluster-sched is UP<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas"><o:p> </o:p></span></p>
</div>
</body>
</html>