<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
I have 2 nodes that offer a "gc" feature. Node t-gc-1202 is "normal", and node t-gc-1201 is dynamic. I can successfully remove t-gc-1201 and bring it back dynamically. Once I bring it back, that node appears JUST LIKE the "normal" node in the sinfo output,
as seen here:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
[rug262@testsch (RC) slurm] sinfo -o "%20N %10c %10m %25f %10G "
<div class="ContentPasted0">NODELIST CPUS MEMORY AVAIL_FEATURES GRES
</div>
<div class="ContentPasted0">t-sc-[1101-1104] 48 358400 nogpu,sc (null)
</div>
<div class="ContentPasted0">t-gc-1201 48 385420 gpu,gc,a100 gpu:2(S:0-
</div>
<div class="ContentPasted0">t-gc-1202 48 358400 gpu,gc,a100 gpu:2 </div>
t-ic-1051 36 500000 ic,a40 (null) <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
When I execute a job requiring 24 CPUs and the gc feature, then it runs on t-gc-1202 only. If I sbatch 3 of the same jobs at once, then 2 run on t-gc-1202 and the 3<span><sup>rd</sup> is pending for resources.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted1">[rug262@testsch (RC) slurm] squeue
<div class="ContentPasted1"> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)</div>
<div class="ContentPasted1"> 405 open-requ gpu_test rug262 PD 0:00 1 (Resources)</div>
<div class="ContentPasted1"> 404 open-requ gpu_test rug262 R 0:06 1 t-gc-1202</div>
<div class="ContentPasted1"> 403 open-requ gpu_test rug262 R 0:07 1 t-gc-1202</div>
<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted1">Both nodes show up in the partitions and show idle before starting the jobs:</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted1"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted1 ContentPasted2">[rug262@testsch (RC) slurm] sinfo
<div class="ContentPasted2">PARTITION AVAIL TIMELIMIT NODES STATE NODELIST</div>
<div class="ContentPasted2">open* up 2-00:00:00 4 idle t-sc-[1101-1104]</div>
<div class="ContentPasted2">open-requeue up 2-00:00:00 6 idle t-gc-[1201-1202],t-sc-[1101-1104]</div>
<div class="ContentPasted2">intr up 2-00:00:00 1 idle t-ic-1051</div>
<div class="ContentPasted2">sla-prio up infinite 6 idle t-gc-[1201-1202],t-sc-[1101-1104]</div>
<div class="ContentPasted2">burst up infinite 4 idle t-sc-[1101-1104]</div>
<div class="ContentPasted2">burst-requeue up infinite 6 idle t-gc-[1201-1202],t-sc-[1101-1104]</div>
<div class="ContentPasted2">debug up infinite 7 idle t-gc-[1201-1202],t-ic-1051,t-sc-[1101-1104]</div>
<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted1 ContentPasted2"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted1 ContentPasted2">So my 2 questions:</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<ol>
<li><span><span class="ContentPasted1 ContentPasted2"></span>How do I get my dynamic node to be utilized like the non-dynamic nodes?</span></li><li><span>I want to have a DIFFERENT feature on my dynamic node, that is not present in the "normal" nodes. When a job is submitted that requires the feature of the dynamic node, I need the job to suspend until the dynamic node becomes available. How do I
go about setting that up? </span></li></ol>
<div><span>Thanks.</span></div>
<div><span><br>
</span></div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted1 ContentPasted2"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted1 ContentPasted2"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<span class="ContentPasted1 ContentPasted2"><br>
</span></div>
</body>
</html>