<html><head></head><body><div class="yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;"><div dir="ltr" data-setdir="false">Hello,</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">I have a small 2 compute node GPU cluster, where each node as 2 GPUs.</div><div><br></div><div><br></div><div dir="ltr" data-setdir="false"><div><p class="ydp48251deMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">$ sinfo -o "%20N %10c %10m %25f %30G "</span></p><p class="ydp48251deMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">NODELIST CPUS MEMORY AVAIL_FEATURES GRES </span></p><p class="ydp48251deMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">o186i[126-127] 128 64000 (null) gpu:nvidia_a40:2(S:0-1) </span></p></div><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">In my batch script, I request 4 GPUs and let Slurm decide how many nodes to automatically allocate. I also tell it I want 1 task per node.</div><div><br></div><div><br></div><div dir="ltr" data-setdir="false"><div><p class="ydp785e2a74MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">$ cat rig_batch.sh</span></p><p class="ydp785e2a74MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">#!/usr/bin/env bash</span></p><p class="ydp785e2a74MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;"> </span></p><p class="ydp785e2a74MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">#SBATCH --ntasks-per-node=1</span></p><p class="ydp785e2a74MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">#SBATCH --nodes=1-9</span></p><p class="ydp785e2a74MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">#SBATCH --gpus=4</span></p><p class="ydp785e2a74MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">#SBATCH --error=/home/corujor/slurm-error.log</span></p><p class="ydp785e2a74MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">#SBATCH --output=/home/corujor/slurm-output.log</span></p><p class="ydp785e2a74MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;"> </span></p><p class="ydp785e2a74MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco; font-size: 12pt;">bash -c 'echo $(hostname):SLURM_JOBID=${SLURM_JOBID}:SLURM_PROCID=${SLURM_PROCID}:CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES}'</span><br></p></div><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">I submit my batch script on slurm-22.05.2.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><span><p class="ydpe7fcef1aMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">$ sbatch rig_batch.sh</span></p></span><span><p class="ydpc874a224MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">Submitted batch job 7</span></p></span><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">I get the expected results. That is, since each compute node has 2 GPUs and I requested 4 GPUs, Slurm allocated 2 nodes, and 1 task per node.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><p class="ydpf5b7ef46MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">$ cat slurm-output.log</span></p><p class="ydpf5b7ef46MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">o186i126:SLURM_JOBID=7:SLURM_PROCID=0:CUDA_VISIBLE_DEVICES=0,1</span></p><p class="ydpf5b7ef46MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">o186i127:SLURM_JOBID=7:SLURM_PROCID=1:CUDA_VISIBLE_DEVICES=0,1</span></p></div><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">However, when I try to submit the same batch script on slurm-22.05.7, it fails.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><p class="ydpff220c76MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">$ sbatch rig_batch.sh</span></p><p class="ydpff220c76MsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">sbatch: error: Batch job submission failed: Requested node configuration is not available</span></p></div><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Here is my configuration.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">$ scontrol show config</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">Configuration data as of 2023-01-12T21:38:55</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AccountingStorageBackupHost = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AccountingStorageEnforce = none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AccountingStorageHost = localhost</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AccountingStorageExternalHost = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AccountingStorageParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AccountingStoragePort = 6819</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AccountingStorageTRES = cpu,mem,energy,node,billing,fs/disk,vmem,pages</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AccountingStorageType = accounting_storage/slurmdbd</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AccountingStorageUser = N/A</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AccountingStoreFlags = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AcctGatherEnergyType = acct_gather_energy/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AcctGatherFilesystemType = acct_gather_filesystem/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AcctGatherInterconnectType = acct_gather_interconnect/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AcctGatherNodeFreq = 0 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AcctGatherProfileType = acct_gather_profile/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AllowSpecResourcesUsage = No</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AuthAltTypes = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AuthAltParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AuthInfo = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">AuthType = auth/munge</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">BatchStartTimeout = 10 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">BcastExclude = /lib,/usr/lib,/lib64,/usr/lib64</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">BcastParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">BOOT_TIME = 2023-01-12T17:17:11</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">BurstBufferType = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">CliFilterPlugins = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ClusterName = grenoble_test</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">CommunicationParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">CompleteWait = 0 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">CoreSpecPlugin = core_spec/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">CpuFreqDef = Unknown</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">CpuFreqGovernors = OnDemand,Performance,UserSpace</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">CredType = cred/munge</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">DebugFlags = Gres</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">DefMemPerNode = UNLIMITED</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">DependencyParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">DisableRootJobs = Yes</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">EioTimeout = 60</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">EnforcePartLimits = ANY</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">Epilog = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">EpilogMsgTime = 2000 usec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">EpilogSlurmctld = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ExtSensorsType = ext_sensors/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ExtSensorsFreq = 0 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">FederationParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">FirstJobId = 1</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">GetEnvTimeout = 2 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">GresTypes = gpu</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">GpuFreqDef = high,memory=high</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">GroupUpdateForce = 1</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">GroupUpdateTime = 600 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">HASH_VAL = Match</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">HealthCheckInterval = 0 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">HealthCheckNodeState = ANY</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">HealthCheckProgram = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">InactiveLimit = 0 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">InteractiveStepOptions = --interactive --preserve-env --pty $SHELL</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobAcctGatherFrequency = 30</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobAcctGatherType = jobacct_gather/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobAcctGatherParams = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobCompHost = localhost</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobCompLoc = /var/log/slurm_jobcomp.log</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobCompPort = 0</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobCompType = jobcomp/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobCompUser = root</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobContainerType = job_container/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobCredentialPrivateKey = /apps/slurm/etc/.slurm.key</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobCredentialPublicCertificate = /apps/slurm/etc/slurm.cert</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobDefaults = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobFileAppend = 0</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobRequeue = 1</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">JobSubmitPlugins = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">KillOnBadExit = 0</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">KillWait = 30 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">LaunchParameters = use_interactive_step</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">LaunchType = launch/slurm</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">Licenses = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">LogTimeFormat = iso8601_ms</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MailDomain = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MailProg = /bin/mail</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MaxArraySize = 1001</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MaxDBDMsgs = 20008</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MaxJobCount = 10000</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MaxJobId = 67043328</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MaxMemPerNode = UNLIMITED</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MaxNodeCount = 2</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MaxStepCount = 40000</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MaxTasksPerNode = 512</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MCSPlugin = mcs/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MCSParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MessageTimeout = 10 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MinJobAge = 300 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MpiDefault = pmix</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MpiParams = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">NEXT_JOB_ID = 274</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">NodeFeaturesPlugins = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">OverTimeLimit = 0 min</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PluginDir = /apps/slurm-22-05-7-1/lib/slurm</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PlugStackConfig = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PowerParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PowerPlugin =</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PreemptMode = OFF</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PreemptType = preempt/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PreemptExemptTime = 00:00:00</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PrEpParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PrEpPlugins = prep/script</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PriorityParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PrioritySiteFactorParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PrioritySiteFactorPlugin = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PriorityType = priority/basic</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PrivateData = none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ProctrackType = proctrack/linuxproc</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">Prolog = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PrologEpilogTimeout = 65534</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PrologSlurmctld = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PrologFlags = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PropagatePrioProcess = 0</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PropagateResourceLimits = ALL</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PropagateResourceLimitsExcept = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">RebootProgram = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ReconfigFlags = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">RequeueExit = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">RequeueExitHold = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ResumeFailProgram = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ResumeProgram = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ResumeRate = 300 nodes/min</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ResumeTimeout = 60 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ResvEpilog = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ResvOverRun = 0 min</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ResvProlog = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ReturnToService = 1</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">RoutePlugin = route/default</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SchedulerParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SchedulerTimeSlice = 30 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SchedulerType = sched/backfill</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">ScronParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SelectType = select/cons_tres</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SelectTypeParameters = CR_CPU</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmUser = slurm(1182)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldAddr = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldDebug = debug</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldHost[0] = o186i208</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldLogFile = /var/log/slurmctld.log</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldPort = 6817</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldSyslogDebug = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldPrimaryOffProg = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldPrimaryOnProg = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldTimeout = 120 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmdDebug = info</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmdLogFile = /var/log/slurmd.log</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmdParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmdPidFile = /var/run/slurmd.pid</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmdPort = 6818</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmdSpoolDir = /var/spool/slurmd</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmdSyslogDebug = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmdTimeout = 300 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmdUser = root(0)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmSchedLogFile = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmSchedLogLevel = 0</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldPidFile = /var/slurm/run/slurmctld.pid</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SlurmctldPlugstack = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SLURM_CONF = /apps/slurm-22-05-7-1/etc/slurm.conf</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SLURM_VERSION = 22.05.7</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SrunEpilog = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SrunPortRange = 0-0</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SrunProlog = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">StateSaveLocation = /var/spool/slurmctld</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SuspendExcNodes = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SuspendExcParts = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SuspendProgram = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SuspendRate = 60 nodes/min</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SuspendTime = INFINITE</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SuspendTimeout = 30 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SwitchParameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">SwitchType = switch/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">TaskEpilog = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">TaskPlugin = task/affinity</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">TaskPluginParam = (null type)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">TaskProlog = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">TCPTimeout = 2 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">TmpFS = /tmp</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">TopologyParam = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">TopologyPlugin = topology/none</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">TrackWCKey = No</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">TreeWidth = 50</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">UsePam = No</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">UnkillableStepProgram = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">UnkillableStepTimeout = 60 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">VSizeFactor = 0 percent</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">WaitTime = 0 sec</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">X11Parameters = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;"> </span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">MPI Plugins Configuration:</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxCliTmpDirBase = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxCollFence = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxDebug = 0</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxDirectConn = yes</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxDirectConnEarly = no</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxDirectConnUCX = no</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxDirectSameArch = no</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxEnv = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxFenceBarrier = no</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxNetDevicesUCX = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxTimeout = 300</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">PMIxTlsUCX = (null)</span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;"> </span></p><p class="ydp69fd81fcMsoNormal" style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;">Slurmctld(primary) at o186i208 is UP</span></p><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;"> </span></div></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><span style="font-family: Monaco;"><br></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);"><br></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);" dir="ltr" data-setdir="false"><span><span style="color: rgb(0, 0, 0); font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;">The only difference when I run this with slurm-22.05.2, is that I have to make this change or Slurm will complain. Other than that, the same configuration is used for both slurm-22.05.2 and slurm.05.7. In both cases, I am running on the same cluster using the same compute nodes, just pointing to different versions of Slurm.</span></span><br></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);" dir="ltr" data-setdir="false"><span><span style="color: rgb(0, 0, 0); font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;"><br></span></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif; color: rgb(0, 0, 0);" dir="ltr" data-setdir="false"><div><pre class="ydp6c78cf7bc-mrkdwn__pre" data-stringify-type="pre" style="margin-top: 4px; margin-bottom: 4px; padding: 8px; font-size: 12px; line-height: 1.50001; white-space: pre-wrap; background-size: ; border-top-width: ; border-right-width: ; border-bottom-width: ; border-left-width: ; border-radius: 4px; color: rgb(29, 28, 29); font-family: Monaco, Menlo, Consolas, "Courier New", monospace !important;">#MpiDefault=pmix
MpiDefault=none</pre></div></div><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Seems like a regression.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Thoughts?</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Thank you,</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Rigoberto</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div></div></body></html>