<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">From the NVIDIA docs re: MPS:<div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><br class=""></div><div class="">On systems with a mix of Volta / pre-Volta GPUs, if the MPS server is set to enumerate any Volta GPU, it will discard all pre-Volta GPUs. In other words, the MPS server will either operate only on the Volta GPUs and expose Volta capabilities, or operate only on pre-Volta GPUs.</div></blockquote><div class=""><br class=""></div><div class=""><br class=""></div><div class="">I'd be curious what happens if you change the ordering (RTX then V100) in the gres.conf -- would the RTX work with MPS and the V100 would not?</div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class="">On Nov 13, 2020, at 07:23 , Holger Badorreck <<a href="mailto:h.badorreck@lzh.de" class="">h.badorreck@lzh.de</a>> wrote:<br class=""><br class="">Hello,<br class=""> <br class="">I have a heterogeneous GPU Node with one V100 and two RTX cards. When I request resources with --gres=mps:100, always the V100 is chosen, and jobs are waiting if the V100 is completely allocated, while RTX cards are free. If I use --gres=gpu:1, also the RTX cards are used. Is something wrong with the configuration or is it another problem?<br class=""> <br class="">The node configuration in slurm.conf:<br class="">NodeName=node1 CPUs=48 RealMemory=128530 Sockets=1 CoresPerSocket=24 ThreadsPerCore=2 Gres=gpu:v100:1,gpu:rtx:2,mps:600 State=UNKNOWN<br class=""> <br class="">gres.conf:<br class="">Name=gpu Type=v100 File=/dev/nvidia0<br class="">Name=gpu Type=rtx File=/dev/nvidia1<br class="">Name=gpu Type=rtx File=/dev/nvidia2<br class="">Name=mps Count=200 File=/dev/nvidia0<br class="">Name=mps Count=200 File=/dev/nvidia1<br class="">Name=mps Count=200 File=/dev/nvidia2<br class=""> <br class="">Best regards,<br class="">Holger<br class=""></blockquote><br class=""></div></body></html>