<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I just found an error in my attempt. I ran on saga-test02 while I'd made the change to saga-test01. Things are working better now.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Erik</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Erik Bryer <ebryer@isi.edu><br>
<b>Sent:</b> Wednesday, December 16, 2020 8:51 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] gres names</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Loris,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
That actually makes some sense. There is one thing that troubles me though. If, on a VM with no GPUs, I define...</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
NodeName=saga-test01 CPUS=2 SocketsPerBoard=1 CoresPerSocket=2 ThreadsPerCore=1 RealMemory=1800 State=UNKNOWN Gres=gpu:gtx1080ti:4<br>
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
...and try to run the following I get an error...</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
$ sbatch -w saga-test02 --gpus=gtx1080ti:1 --partition scavenge --wrap "ls -l" --qos scavengesbatch: error: Batch job submission failed: Requested node configuration is not available</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
This also fouls the whole cluster. Directly after issuing the sbatch, this occurs: </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Dec 16 07:39:03 saga-test03 slurmctld[1169]: error: Setting node saga-test01 state to DRAIN</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
During past tests I've been unable to get both nodes back online without removing the spurious gres from the node definition. All this still makes me wonder whether there is a direct link between the hardware and gres names. I think so. Someone mentioned the
gres names get spit out by NVML (but you can also make up your own (?)), but I can't find a record of ours. Any thoughts?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Erik</div>
<div id="x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Loris Bennett <loris.bennett@fu-berlin.de><br>
<b>Sent:</b> Wednesday, December 16, 2020 12:07 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] gres names</font>
<div> </div>
</div>
<div class="x_BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="x_PlainText">Hi Erik,<br>
<br>
Erik Bryer <ebryer@isi.edu> writes:<br>
<br>
> Thanks for your reply. I can't find NVML in the logs going back to<br>
> 11/22. dmesg goes back to the last boot, but has no mention of<br>
> NVML. Regarding make one up on my own, how does Slurm know string<br>
> "xyzzy" corresponds to a tesla gpu, e.g.?<br>
<br>
As I understand it, Slurm doesn't need to know the correspondence, since<br>
all it is doing is counting. If you define a GRES, say,<br>
<br>
magic:wand<br>
<br>
you can configure your nodes to have, say, 2 of these. Then if a job<br>
requests <br>
<br>
--gres=magic:wand:1<br>
<br>
and starts, a subsequent job which requests <br>
<br>
--gres=magic:wand:2<br>
<br>
will have to wait until the first magic wand become free again.<br>
However, Slurm doesn't need to know whether your nodes really do have<br>
magic wands, but your users do need to request them, if their jobs<br>
require them. To prevent them using a magic wand without requesting<br>
one, you have to check the job parameters on submission, which you can<br>
do via the job submit plugin.<br>
<br>
Regards<br>
<br>
Loris<br>
<br>
> Thanks,<br>
> Erik<br>
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
> From: slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Michael Di Domenico <mdidomenico4@gmail.com><br>
> Sent: Tuesday, December 15, 2020 1:24 PM<br>
> To: Slurm User Community List <slurm-users@lists.schedmd.com><br>
> Subject: Re: [slurm-users] gres names <br>
> <br>
> you can either make them up on your own or they get spit out by NVML<br>
> in the slurmd.log file<br>
><br>
> On Tue, Dec 15, 2020 at 12:55 PM Erik Bryer <ebryer@isi.edu> wrote:<br>
>><br>
>> Hi,<br>
>><br>
>> Where do I get the gres names, e.g. "rtx2080ti", to use for my gpus in my node definitions in slurm.conf?<br>
>><br>
>> Thanks,<br>
>> Erik<br>
><br>
-- <br>
Dr. Loris Bennett (Hr./Mr.)<br>
ZEDAT, Freie Universität Berlin Email loris.bennett@fu-berlin.de<br>
<br>
</div>
</span></font></div>
</div>
</body>
</html>