<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">My only thought here that is a little off-kilter would be to get a stupid do-nothing job assigned to the failing GPU for 100,000 hours… It might take a bit of work - and some to and fro- but “fake occupy” the failing GPU and every other job will maneuver around it.</div><div class=""><br class=""></div><div class="">Again - it’s not a great solution, but I think it would work.</div><br class=""><div class="">
<div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Take care,</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Fuzzy Rogers</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">(he, his)</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Research Computing Administrator</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Materials Research Laboratory</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Santa Barbara, CA 93106-5121</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class=""></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline">
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Jun 4, 2021, at 11:35 AM, Jason Simms <<a href="mailto:simmsj@lafayette.edu" class="">simmsj@lafayette.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">You don't need to chide me for making what is, to me, a reasonable solution. *You* may not be able to make hardware changes, but why the people who can would want failing GPUs remaining in a system is anathema to my approach to cluster management. In other words, I do not recommend you try to find a workaround to a solution that, in my opinion, is best solved by eliminating the faulty hardware. I understand the impulse, and if there is a simple solution to specifying a specific GPU, then fine, do that. But again it goes against treating such resources as generic - nodes and hardware should be thought of as cattle, not pets, and should be managed accordingly. Again, I believe you are trying to solve a problem that should not be yours to solve. Sorry if this irritates you.</div><div class=""><br class=""></div><div class="">JLS<br class=""> </div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 4, 2021 at 2:17 PM Ahmad Khalifa <<a href="mailto:underoath006@gmail.com" class="">underoath006@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">I can't make hardware changes, but I still want to make use of the cluster. Let's keep the discussion on how to get slurm to do it, if that's possible. </div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 4, 2021 at 11:13 AM Jason Simms <<a href="mailto:simmsj@lafayette.edu" target="_blank" class="">simmsj@lafayette.edu</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">Unpopular opinion: remove the failing GPU.</div><div class=""><br class=""></div><div class="">JLS<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 4, 2021 at 2:07 PM Ahmad Khalifa <<a href="mailto:underoath006@gmail.com" target="_blank" class="">underoath006@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Because there are failing GPUs that I'm trying to avoid. </div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 4, 2021 at 5:04 AM Stephan Roth <<a href="mailto:stephan.roth@ee.ethz.ch" target="_blank" class="">stephan.roth@ee.ethz.ch</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 03.06.21 07:11, Ahmad Khalifa wrote:<br class="">
> How to send a job to a particular gpu card using its ID (0,1,2...etc)?<br class="">
<br class="">
Why do you need to access a GPU based on its ID?<br class="">
<br class="">
If its to select a certain GPU type, there are other methods you can use.<br class="">
<br class="">
You could create partitions for the same GPU types or add features.<br class="">
Due to our heterogenous nodes with mixed GPU types we do the latter, we <br class="">
added a feature for the GPU architectures and one for the GPU types to <br class="">
each node.<br class="">
<br class="">
Cheers,<br class="">
Stephan<br class="">
<br class="">
</blockquote></div>
</blockquote></div><br clear="all" class=""><br class="">-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div style="font-family: Helvetica; font-size: 14px; margin: 0px;" class=""><span style="color:rgb(130,36,51)" class=""><font face="Century Gothic" class=""><b class="">Jason L. Simms, Ph.D., M.P.H.</b></font></span></div><div style="font-family: Helvetica; font-size: 14px; margin: 0px;" class=""><font face="Century Gothic" class=""><span class="">Manager of Research and High-Performance Computing</span></font></div><div style="font-family: Helvetica; font-size: 14px; margin: 0px;" class=""><font face="Century Gothic" class=""><span class="">XSEDE Campus Champion<br class=""></span><span style="color:gray" class="">Lafayette College<br class="">Information Technology Services<br class="">710 Sullivan Rd | Easton, PA 18042<br class="">Office: 112 Skillman Library<br class="">p: (610) 330-5632</span></font></div></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all" class=""><br class="">-- <br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div style="font-family: Helvetica; font-size: 14px; margin: 0px;" class=""><span style="color:rgb(130,36,51)" class=""><font face="Century Gothic" class=""><b class="">Jason L. Simms, Ph.D., M.P.H.</b></font></span></div><div style="font-family: Helvetica; font-size: 14px; margin: 0px;" class=""><font face="Century Gothic" class=""><span class="">Manager of Research and High-Performance Computing</span></font></div><div style="font-family: Helvetica; font-size: 14px; margin: 0px;" class=""><font face="Century Gothic" class=""><span class="">XSEDE Campus Champion<br class=""></span><span style="color:gray" class="">Lafayette College<br class="">Information Technology Services<br class="">710 Sullivan Rd | Easton, PA 18042<br class="">Office: 112 Skillman Library<br class="">p: (610) 330-5632</span></font></div></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></body></html>