<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hoping someone can tell me if I’m just thinking about this wrong, or if maybe this is somewhere with room for improvement.<div class=""><br class=""></div><div class="">I recently upgraded my cluster to 22.05.8 and am testing out gpu sharding on a subset of GPUs, specifically my T4’s.</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class=""><font face="Menlo" class="">-------------------------------------------------------------------------------</font></div><div class=""><font face="Menlo" class="">Cluster Utilization 2023-02-13T00:00:00 - 2023-02-13T23:59:59</font></div><div class=""><font face="Menlo" class="">Usage reported in Percentage of Total</font></div><div class=""><font face="Menlo" class="">-------------------------------------------------------------------------------</font></div><div class=""><font face="Menlo" class=""> TRES Name Allocate Down PLND Dow Idle Planned Reported</font></div><div class=""><font face="Menlo" class="">-------------- -------- ----------- -------- ------------ -------- ------------</font></div><div class=""><font face="Menlo" class=""> gres/gpu:t4 0.00% 0.00% 0.00% 100.00% 0.00% 100.00%</font></div><div class=""><font face="Menlo" class=""> gres/shard 37.06% 0.00% 0.00% 62.94% 0.00% 100.00%</font></div></blockquote><br class=""></div><div class="">What seems odd to me is that I have shards being consumed, which is implicitly consuming the gpu:t4(s).</div><div class="">However, sreport makes it appear as though the T4’s were completely idle, which is not true.</div><div class=""><br class=""></div><div class="">I know that shards and gpu’s are not a 1:1 allocation, if anything the gpu allocation would almost always be greater than shard allocation.</div><div class="">But in my head that seems like that should be the case, given that the gpu’s are not idle, and in fact allocated, if only “partially.”</div><div class=""><br class=""></div><div class="">I know shards are a new concept and likely will evolve over time, but wanted to see if anyone had run into or thought similarly about this concept.</div><div class=""><br class=""></div><div class="">Reed</div></body></html>