<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
At some point when we were experimenting with MIG, I was being entirely frustrated in getting it to work until I finally removed the autodetect from gres.conf and explicitly listed the stuff instead. THEN it worked. I think you can find the list of files
that are the device files using nvidia-smi.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Here is the entry we use in our gres.conf for one of the nodes:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
NodeName=p-gc-3037 Name=gpu Type=1g.5gb File=/dev/nvidia-caps/nvidia-cap[66,75,84,102,111,120,129,201,210,219,228,237,246,255]<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
Something to TRY anyway. Odd that 3g.20gb works. You might try reconfiguring the node for that instead and see if it works then. We've used 3g.20gb and 1g.5gb on our nodes and it works fine, never tried 2g.10gb.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
Rob</div>
<div>
<div><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
<hr tabindex="-1" style="display:inline-block; width:98%;">
<b>From:</b> slurm-users on behalf of Vogt, Timon<br>
<b>Sent:</b> Wednesday, July 19, 2023 3:08 PM<br>
<b>To:</b> slurm-users@schedmd.com<br>
<b>Subject:</b> [slurm-users] MIG-Slice: Unavailable GRES
<div><br>
</div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Dear Slurm Mailing List,<br>
<br>
I am experiencing a problem which affects our cluster and for which I am <br>
completely out of ideas by now, so I would like to ask the community for <br>
hints or ideas.<br>
<br>
We run a partition on our cluster containing multiple nodes with Nvidia <br>
A100 GPUs (40GB), which we have sliced up using Nvidia Multi-Instance <br>
GPUs (MIG) into one 3g.20gb slice and two 2g.10gb slices per GPU.<br>
<br>
Now, when submitting a job to it and requesting the 3g.20gb slice (like <br>
with "srun -p mig-partition -G 3g.20gb:1 hostname"), the job runs fine, <br>
but when a job requests one of the 2g.10gb slices instead (like with <br>
"srun -p mig-partition -G 2g.10gb:1 hostname"), the job does not get <br>
scheduled and the controller repeatedly outputs the error:<br>
<br>
slurmctld[28945]: error: _set_job_bits1: job 4780824 failed to find any <br>
available GRES on node 1471<br>
slurmctld[28945]: error: gres_select_filter_select_and_set job 4780824 <br>
failed to satisfy gres-per-job counter<br>
<br>
Our cluster uses the AutoDetect=nvml feature for the nodes in the <br>
gres.conf and both slice types are defined in "AccountingStorageTRES" <br>
and in the GRES parameter of the node definition. The slurmd on the node <br>
also finds both types of slices and reports the correct amounts. They <br>
are also visible in the "Gres=" section of "scontrol show node", again <br>
in correct amounts.<br>
<br>
I have also ensured that the nodes are not used otherwise by creating a <br>
reservation on them accessible only to me, and I have restarted all <br>
slurmd's and the slurmctld.<br>
<br>
By now, I am out of ideas. Does someone here have a suggestion on what <br>
else I can try? Has someone already seen this error and knows more about it?<br>
<br>
Thank you very much in advance and<br>
best regards,<br>
Timon<br>
<br>
-- <br>
Timon Vogt<br>
Arbeitsgruppe "Computing"<br>
Nationales Hochleistungsrechnen (NHR)<br>
Scientific Employee NHR<br>
Tel.: +49 551 39-30146, E-Mail: timon.vogt@gwdg.de<br>
-------------------------------------------------------------------------<br>
Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG)<br>
Burckhardtweg 4, 37077 Göttingen, URL: <a href="https://gwdg.de" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" data-loopstyle="linkonly">
https://gwdg.de</a><br>
<br>
Support: Tel.: +49 551 39-30000, URL: <a href="https://gwdg.de/support" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">
https://gwdg.de/support</a><br>
Sekretariat: Tel.: +49 551 39-30001, E-Mail: gwdg@gwdg.de<br>
<br>
Geschäftsführer: Prof. Dr. Ramin Yahyapour<br>
Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger<br>
Sitz der Gesellschaft: Göttingen<br>
Registergericht: Göttingen, Handelsregister-Nr. B 598<br>
<br>
Zertifiziert nach ISO 9001 und ISO 27001<br>
-------------------------------------------------------------------------<br>
<br>
</div>
</span></font></div>
</div>
</body>
</html>