<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Further testing and looking at the source code confirms what looks to me like a bug in Slurm. GPUs that are not configured in gres.conf are detected by slurmd in the system and discarded since they aren't found in gres.conf. That's fine except they should also
be hidden through cgroup control so that they aren't visible along with allocated GPUs when a job is run. Slurm assumes that the job can only see the GPUs that it allocates to the job and sets the $CUDA_VISIBLE_DEVICES accordingly. Unfortunately, the job actually
sees the allocated GPUs plus any unconfigured GPUs and $CUDA_VISIBLE_DEVICES may or may not happen to correspond to the GPU(s) allocated by Slurm.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I was hoping that I could write a Prolog script that would adjust $CUDA_VISIBLE_DEVICES to remove any unconfigured GPUs but any changes using "export CUDA_VISIBLE_DEVICES=..." don't seem to have an effect upon the actual environment of the job.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Steve<br>
</div>
<div id="appendonsend"></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size: 11pt; color: rgb(0, 0, 0);" face="Calibri, sans-serif"><b>From:</b> Wilson, Steven M <stevew@purdue.edu><br>
<b>Sent:</b> Friday, July 14, 2023 4:10 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Unconfigured GPUs being allocated</font>
<div> </div>
</div>
<div dir="ltr">
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
It's not so much whether a job may or may not access the GPU but rather which GPU(s) is(are) included in $CUDA_VISIBLE_DEVICES. That is what controls what our CUDA jobs can see and therefore use (within any cgroups constraints, of course). In my case, Slurm
is sometimes setting $CUDA_VISIBLE_DEVICES to a GPU that is not in the Slurm configuration because it is intended only for driving the display and not GPU computations.</div>
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for your thoughts!</div>
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Steve<br>
</div>
<div id="x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size: 11pt; color: rgb(0, 0, 0);" face="Calibri, sans-serif"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Christopher Samuel <chris@csamuel.org><br>
<b>Sent:</b> Friday, July 14, 2023 1:57 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Unconfigured GPUs being allocated</font>
<div> </div>
</div>
<div class="x_BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="x_PlainText">[You don't often get email from chris@csamuel.org. Learn why this is important at
<a href="https://aka.ms/LearnAboutSenderIdentification" data-auth="NotApplicable">
https://aka.ms/LearnAboutSenderIdentification</a> ]<br>
<br>
---- External Email: Use caution with attachments, links, or sharing data ----<br>
<br>
<br>
On 7/14/23 10:20 am, Wilson, Steven M wrote:<br>
<br>
> I upgraded Slurm to 23.02.3 but I'm still running into the same problem.<br>
> Unconfigured GPUs (those absent from gres.conf and slurm.conf) are still<br>
> being made available to jobs so we end up with compute jobs being run on<br>
> GPUs which should only be used<br>
<br>
I think this is expected - it's not that Slurm is making them available,<br>
it's that it's unaware of them and so doesn't control them in the way it<br>
does for the GPUs it does know about. So you get the default behaviour<br>
(any process can access them).<br>
<br>
If you want to stop them being accessed from Slurm you'd need to find a<br>
way to prevent that access via cgroups games or similar.<br>
<br>
All the best,<br>
Chris<br>
--<br>
Chris Samuel : <a href="http://www.csamuel.org/" data-auth="NotApplicable">https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&data=05%7C01%7Cstevew%40purdue.edu%7C6fba97485b73413521d208db8494160a%7C4130bd397c53419cb1e58758d6d63f21%7C0%7C0%7C638249543794377751%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VslW51ree1Ibt3xfYyy99Aj%2BREZh7BqpM6Ipg3jAM84%3D&reserved=0</a>
: Berkeley, CA, USA<br>
<br>
<br>
</div>
</span></font></div>
</div>
</body>
</html>