<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<font face="monospace">Hi Rob,<br>
<br>
thank you very much for that hint. I tried setting the MIG slices
manually in the gres.conf and it works now.<br>
<br>
Thank you very much.<br>
Best regards,<br>
Timon<br>
</font>
<pre class="moz-signature" cols="72">--
Timon Vogt
Arbeitsgruppe "Computing"
Nationales Hochleistungsrechnen (NHR)
Scientific Employee NHR
Tel.: +49 551 39-30146, E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:timon.vogt@gwdg.de">timon.vogt@gwdg.de</a>
-------------------------------------------------------------------------
Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG)
Burckhardtweg 4, 37077 Göttingen, URL: <a class="moz-txt-link-freetext" href="https://gwdg.de">https://gwdg.de</a>
Support: Tel.: +49 551 39-30000, URL: <a class="moz-txt-link-freetext" href="https://gwdg.de/support">https://gwdg.de/support</a>
Sekretariat: Tel.: +49 551 39-30001, E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:gwdg@gwdg.de">gwdg@gwdg.de</a>
Geschäftsführer: Prof. Dr. Ramin Yahyapour
Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger
Sitz der Gesellschaft: Göttingen
Registergericht: Göttingen, Handelsregister-Nr. B 598
Zertifiziert nach ISO 9001 und ISO 27001
-------------------------------------------------------------------------</pre>
<div class="moz-cite-prefix">Am 19.07.23 um 21:21 schrieb Groner,
Rob:<br>
</div>
<blockquote type="cite" cite="mid:BL0PR02MB44992DE18307194E32A451728039A@BL0PR02MB4499.namprd02.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
At some point when we were experimenting with MIG, I was being
entirely frustrated in getting it to work until I finally
removed the autodetect from gres.conf and explicitly listed the
stuff instead. THEN it worked. I think you can find the list
of files that are the device files using nvidia-smi.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Here is the entry we use in our gres.conf for one of the nodes:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
NodeName=p-gc-3037 Name=gpu Type=1g.5gb
File=/dev/nvidia-caps/nvidia-cap[66,75,84,102,111,120,129,201,210,219,228,237,246,255]<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
Something to TRY anyway. Odd that 3g.20gb works. You might try
reconfiguring the node for that instead and see if it works
then. We've used 3g.20gb and 1g.5gb on our nodes and it works
fine, never tried 2g.10gb.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
Rob</div>
<div>
<div><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
<hr tabindex="-1" style="display:inline-block; width:98%;">
<b>From:</b> slurm-users on behalf of Vogt, Timon<br>
<b>Sent:</b> Wednesday, July 19, 2023 3:08 PM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@schedmd.com">slurm-users@schedmd.com</a><br>
<b>Subject:</b> [slurm-users] MIG-Slice: Unavailable GRES
<div><br>
</div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Dear Slurm Mailing List,<br>
<br>
I am experiencing a problem which affects our cluster
and for which I am <br>
completely out of ideas by now, so I would like to ask
the community for <br>
hints or ideas.<br>
<br>
We run a partition on our cluster containing multiple
nodes with Nvidia <br>
A100 GPUs (40GB), which we have sliced up using Nvidia
Multi-Instance <br>
GPUs (MIG) into one 3g.20gb slice and two 2g.10gb slices
per GPU.<br>
<br>
Now, when submitting a job to it and requesting the
3g.20gb slice (like <br>
with "srun -p mig-partition -G 3g.20gb:1 hostname"), the
job runs fine, <br>
but when a job requests one of the 2g.10gb slices
instead (like with <br>
"srun -p mig-partition -G 2g.10gb:1 hostname"), the job
does not get <br>
scheduled and the controller repeatedly outputs the
error:<br>
<br>
slurmctld[28945]: error: _set_job_bits1: job 4780824
failed to find any <br>
available GRES on node 1471<br>
slurmctld[28945]: error:
gres_select_filter_select_and_set job 4780824 <br>
failed to satisfy gres-per-job counter<br>
<br>
Our cluster uses the AutoDetect=nvml feature for the
nodes in the <br>
gres.conf and both slice types are defined in
"AccountingStorageTRES" <br>
and in the GRES parameter of the node definition. The
slurmd on the node <br>
also finds both types of slices and reports the correct
amounts. They <br>
are also visible in the "Gres=" section of "scontrol
show node", again <br>
in correct amounts.<br>
<br>
I have also ensured that the nodes are not used
otherwise by creating a <br>
reservation on them accessible only to me, and I have
restarted all <br>
slurmd's and the slurmctld.<br>
<br>
By now, I am out of ideas. Does someone here have a
suggestion on what <br>
else I can try? Has someone already seen this error and
knows more about it?<br>
<br>
Thank you very much in advance and<br>
best regards,<br>
Timon<br>
<br>
-- <br>
Timon Vogt<br>
Arbeitsgruppe "Computing"<br>
Nationales Hochleistungsrechnen (NHR)<br>
Scientific Employee NHR<br>
Tel.: +49 551 39-30146, E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:timon.vogt@gwdg.de">timon.vogt@gwdg.de</a><br>
-------------------------------------------------------------------------<br>
Gesellschaft für wissenschaftliche Datenverarbeitung mbH
Göttingen (GWDG)<br>
Burckhardtweg 4, 37077 Göttingen, URL: <a href="https://gwdg.de" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" data-loopstyle="linkonly" moz-do-not-send="true" class="moz-txt-link-freetext">
https://gwdg.de</a><br>
<br>
Support: Tel.: +49 551 39-30000, URL: <a href="https://gwdg.de/support" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" moz-do-not-send="true" class="moz-txt-link-freetext">
https://gwdg.de/support</a><br>
Sekretariat: Tel.: +49 551 39-30001, E-Mail:
<a class="moz-txt-link-abbreviated" href="mailto:gwdg@gwdg.de">gwdg@gwdg.de</a><br>
<br>
Geschäftsführer: Prof. Dr. Ramin Yahyapour<br>
Aufsichtsratsvorsitzender: Prof. Dr. Christian
Griesinger<br>
Sitz der Gesellschaft: Göttingen<br>
Registergericht: Göttingen, Handelsregister-Nr. B 598<br>
<br>
Zertifiziert nach ISO 9001 und ISO 27001<br>
-------------------------------------------------------------------------<br>
<br>
</div>
</span></font></div>
</div>
</blockquote>
<br>
</body>
</html>