<div dir="ltr"><div>Kirk,</div><div>MailProg=/usr/bin/sendmail</div><div><br></div><div>MailProg should be the program used to SEND mail ie.  /bin/mail  not sendmail</div><div>If I am not wrong int he jargon MailProg is a MUA not an MTA  (sendmail is an MTA)</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, 18 Oct 2018 at 19:01, Kirk Main <<a href="mailto:kjmain@ncsu.edu">kjmain@ncsu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div dir="ltr"><div>Hi all,</div><div><br></div><div>I'm a new administrator to Slurm and I've just got my new cluster up and running. We started getting a lot of "Socket timed out on send/recv operation" errors when submitting jobs, and also if you try to "squeue" while others are submitting jobs. The job does eventually run after about a minute, but the entire system feels very sluggish and obviously this isn't normal. Not sure whats going on here...</div><div><br></div><div>Head nodema-vm-slurm01 and ma-vm-slurm02 are virtual machines running on a Hyper-V host with a common NFS share between all of the worker and head nodes. Head nodes have 8CPU/8GB running on Ubuntu 16.04. All network interconnect is 10GBE.</div><div><br></div><div>slurmctld Log Snippet:</div><div>





<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 11:08:57 ma-vm-slurm01 slurmctld[1603]: validate_node_specs: Node ma-pm-hpc03 unexpectedly rebooted boot_time=1539616111 last response=1539615694</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 12:40:21 ma-vm-slurm01 slurmctld[1603]: _slurm_rpc_submit_batch_job: JobId=476 InitPrio=4294901364 usec=33061624</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 12:40:36 ma-vm-slurm01 slurmctld[1603]: email msg to <a href="mailto:mrgaddy@ncsu.edu" target="_blank">mrgaddy@ncsu.edu</a>: Slurm Job_id=476 Name=StochUnifOpt_Chu_3D_forCluster.m.job Began, Queued time 00:00:48</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 12:40:36 ma-vm-slurm01 slurmctld[1603]: sched: Allocate JobId=476 NodeList=ma-pm-hpc17 #CPUs=8 Partition=math-hpc</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p2" style="margin:0px;color:rgb(180,36,25);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s2" style="color:rgb(242,242,242)">Oct 15 12:40:45 ma-vm-slurm01 slurmctld[16956]: </span><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">error: Failed to exec /usr/bin/sendmail: No such file or directory</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 13:12:00 ma-vm-slurm01 slurmctld[1603]: _slurm_rpc_submit_batch_job: JobId=477 InitPrio=4294901363 usec=75836582</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 13:12:23 ma-vm-slurm01 slurmctld[1603]: email msg to <a href="mailto:mrgaddy@ncsu.edu" target="_blank">mrgaddy@ncsu.edu</a>: Slurm Job_id=477 Name=StochUnifOpt_Chu_3D_forCluster.m.job Began, Queued time 00:01:39</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 13:12:23 ma-vm-slurm01 slurmctld[1603]: sched: Allocate JobId=477 NodeList=ma-pm-hpc17 #CPUs=8 Partition=math-hpc</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p2" style="margin:0px;color:rgb(180,36,25);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s2" style="color:rgb(242,242,242)">Oct 15 13:12:34 ma-vm-slurm01 slurmctld[18952]: </span><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">error: Failed to exec /usr/bin/sendmail: No such file or directory</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 13:13:58 ma-vm-slurm01 slurmctld[1603]: sched: _slurm_rpc_allocate_resources JobId=478 NodeList=ma-pm-hpc17 usec=12600182</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p2" style="margin:0px;color:rgb(180,36,25);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s2" style="color:rgb(242,242,242)">Oct 15 13:13:58 ma-vm-slurm01 slurmctld[1603]: </span><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">error: Job allocate response msg send failure, killing JobId=478</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 13:13:58 ma-vm-slurm01 slurmctld[1603]: _job_complete: JobId=478 WTERMSIG 15</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 13:13:58 ma-vm-slurm01 slurmctld[1603]: _job_complete: JobId=478 done</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 13:36:58 ma-vm-slurm01 slurmctld[1603]: _job_complete: JobId=476 WEXITSTATUS 0</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 13:36:58 ma-vm-slurm01 slurmctld[1603]: email msg to <a href="mailto:mrgaddy@ncsu.edu" target="_blank">mrgaddy@ncsu.edu</a>: Slurm Job_id=476 Name=StochUnifOpt_Chu_3D_forCluster.m.job Ended, Run time 00:56:22, COMPLETED, ExitCode 0</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Oct 15 13:36:58 ma-vm-slurm01 slurmctld[1603]: _job_complete: JobId=476 done</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p2" style="margin:0px;color:rgb(180,36,25);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s2" style="color:rgb(242,242,242)">Oct 15 13:37:03 ma-vm-slurm01 slurmctld[19285]: </span><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">error: Failed to exec /usr/bin/sendmail: No such file or directory</span></p></div><div><br></div><div>slurm.conf:</div><div>





<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style=""># slurm.conf file generated by configurator.html.</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style=""># See the slurm.conf man page for more information.</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">ClusterName=math-hpc</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">ControlMachine=ma-vm-slurm01</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#ControlAddr=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">BackupController=ma-vm-slurm02</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#BackupAddr=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SlurmUser=slurm</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SlurmdUser=root</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SlurmctldPort=6817</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SlurmdPort=6818</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">AuthType=auth/munge</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#JobCredentialPrivateKey=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#JobCredentialPublicCertificate=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">MailProg=/usr/bin/sendmail</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">StateSaveLocation=/mnt/HpcStor/etc/slurm/state</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SlurmdSpoolDir=/var/spool/slurmd.spool</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SwitchType=switch/none</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">MpiDefault=none</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">MpiParams=ports=12000-12999</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SlurmctldPidFile=/var/run/slurmctld.pid</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SlurmdPidFile=/var/run/slurmd.pid</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">ProctrackType=proctrack/linuxproc</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PluginDir=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">CacheGroups=0</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#FirstJobId=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">ReturnToService=1</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#MaxJobCount=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PlugStackConfig=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PropagatePrioProcess=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PropagateResourceLimits=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PropagateResourceLimitsExcept=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#Prolog=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#Epilog=/etc/slurm/slurm.epilog.clean</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#SrunProlog=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#SrunEpilog=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#TaskProlog=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#TaskEpilog=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">TaskPlugin=task/cgroup</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">TaskPluginParam=Cores,Verbose</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#TrackWCKey=no</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#TreeWidth=50</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">TmpFS=/tmp</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#UsePAM=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style=""># TIMERS</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SlurmctldTimeout=120</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SlurmdTimeout=300</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">InactiveLimit=600</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">MinJobAge=300</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">KillWait=30</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">Waittime=0</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style=""># SCHEDULING</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SchedulerType=sched/builtin</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#SchedulerAuth=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#SchedulerPort=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#SchedulerRootFilter=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SelectType=select/cons_res</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SelectTypeParameters=CR_CPU_MEMORY</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">FastSchedule=1</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PriorityType=priority/multifactor</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PriorityDecayHalfLife=14-0</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PriorityUsageResetPeriod=14-0</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PriorityWeightFairshare=100000</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PriorityWeightAge=1000</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PriorityWeightPartition=10000</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PriorityWeightJobSize=1000</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#PriorityMaxAge=1-0</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style=""># LOGGING</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SlurmctldDebug=5</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#SlurmctldLogFile=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">SlurmdDebug=5</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#SlurmdLogFile=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">JobCompType=jobcomp/SlurmDBD</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#JobCompLoc=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style=""># ACCOUNTING</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">JobAcctGatherType=jobacct_gather/linux</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">JobAcctGatherFrequency=30</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">AccountingStorageType=accounting_storage/slurmdbd</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">AccountingStorageHost=ma-vm-slurm01</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">AccountingStorageBackupHost=ma-vm-slurm02</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#AccountingStorageLoc=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#AccountingStoragePass=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#AccountingStorageUser=</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">AccountingStorageEnforce=associations,limits</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p2" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;min-height:14px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style=""></span><br></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style=""># COMPUTE NODES</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">#GresTypes=gpu</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">NodeName=ma-pm-hpc[01-10] RealMemory=128000 Sockets=2 CoresPerSocket=8 ThreadsPerCore=1 State=UNKNOWN</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">NodeName=ma-pm-hpc[11,12] RealMemory=128000 Sockets=2 CoresPerSocket=12 ThreadsPerCore=1 State=UNKNOWN</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">NodeName=ma-pm-hpc[13-23] RealMemory=192000 Sockets=2 CoresPerSocket=14 ThreadsPerCore=1 State=UNKNOWN</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p1" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style="">PartitionName=math-hpc Nodes=ma-pm-hpc[01-23] Default=YES MaxTime=10-00:00:00 State=UP Shared=FORCE DefMemPerCPU=7680</span></p>
<p class="m_-8632593582344224631m_6234544833668369399gmail-p2" style="margin:0px;color:rgb(242,242,242);line-height:normal;font-family:Monaco;font-size:10px;min-height:14px;font-stretch:normal;background-color:rgba(0,0,0,0.85)"><span class="m_-8632593582344224631m_6234544833668369399gmail-s1" style=""></span><br></p></div><div><br></div><div>Thanks,</div><div><div class="m_-8632593582344224631m_6234544833668369399gmail_signature" dir="ltr"><div dir="ltr"><div><div dir="ltr"><b>Kirk J. Main</b><div>Systems Administrator, Department of Mathematics</div><div>College of Sciences</div><div>P: 919.515.6315</div><div><a href="mailto:kjmain@ncsu.edu" target="_blank">kjmain@ncsu.edu</a></div><div><br></div><div><img><br></div></div></div></div></div></div></div></div>
</div></div>
</blockquote></div>