<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Worth a try, but the documentation says that by default the count is the same as the number of files specified...so, should automatically be 1.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
If you want to stop the node from going to INVAL, you can always set config_overrides in slurm.conf. That will tell the node what it has, instead of what it thinks it has. Useful for testing.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Rob</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Hermann Schwärzler <hermann.schwaerzler@uibk.ac.at><br>
<b>Sent:</b> Wednesday, July 19, 2023 9:04 AM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] GRES and GPUs</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hi Xaver,<br>
<br>
I think you are missing the "Count=..." part in gres.conf<br>
<br>
It should read<br>
<br>
NodeName=NName Name=gpu File=/dev/tty0 Count=1<br>
<br>
in your case.<br>
<br>
Regards,<br>
Hermann<br>
<br>
On 7/19/23 14:19, Xaver Stiensmeier wrote:<br>
> Okay,<br>
> <br>
> thanks to S. Zhang I was able to figure out why nothing changed. While I <br>
> did restart systemctld at the beginning of my tests, I didn't do so <br>
> later, because I felt like it was unnecessary, but it is right there in <br>
> the fourth line of the log that this is needed. Somehow I misread it and <br>
> thought it automatically restarted slurmctld.<br>
> <br>
> Given the setup:<br>
> <br>
> slurm.conf<br>
> ...<br>
> GresTypes=gpu<br>
> NodeName=NName SocketsPerBoard=8 CoresPerSocket=1 RealMemory=8000 <br>
> GRES=gpu:1 State=UNKNOWN<br>
> ...<br>
> <br>
> gres.conf<br>
> NodeName=NName Name=gpu File=/dev/tty0<br>
> <br>
> When restarting, I get the following error:<br>
> <br>
> error: Setting node NName state to INVAL with reason:gres/gpu count <br>
> reported lower than configured (0 < 1)<br>
> <br>
> So it is still not working, but at least I get a more helpful log <br>
> message. Because I know that this /dev/tty trick works, I am still <br>
> unsure where the current error lies, but I will try to investigate it <br>
> further. I am thankful for any ideas in that regard.<br>
> <br>
> Best regards,<br>
> Xaver<br>
> <br>
> On 19.07.23 10:23, Xaver Stiensmeier wrote:<br>
>><br>
>> Alright,<br>
>><br>
>> I tried a few more things, but I still wasn't able to get past: srun: <br>
>> error: Unable to allocate resources: Invalid generic resource (gres) <br>
>> specification.<br>
>><br>
>> I should mention that the node I am trying to test GPU with, doesn't <br>
>> really have a gpu, but Rob was so kind to find out that you do not <br>
>> need a gpu as long as you just link to a file in /dev/ in the <br>
>> gres.conf. As mentioned: This is just for testing purposes - in the <br>
>> end we will run this on a node with a gpu, but it is not available at <br>
>> the moment.<br>
>><br>
>> *The error isn't changing*<br>
>><br>
>> If I omitt "GresTypes=gpu" and "Gres=gpu:1", I still get the same error.<br>
>><br>
>> *Debug Info*<br>
>><br>
>> I added the gpu debug flag and logged the following:<br>
>><br>
>> [2023-07-18T14:59:45.026] restoring original state of nodes<br>
>> [2023-07-18T14:59:45.026] select/cons_tres: part_data_create_array: <br>
>> select/cons_tres: preparing for 2 partitions<br>
>> [2023-07-18T14:59:45.026] error: GresPlugins changed from (null) to <br>
>> gpu ignored<br>
>> [2023-07-18T14:59:45.026] error: Restart the slurmctld daemon to <br>
>> change GresPlugins<br>
>> [2023-07-18T14:59:45.026] read_slurm_conf: backup_controller not specified<br>
>> [2023-07-18T14:59:45.026] error: GresPlugins changed from (null) to <br>
>> gpu ignored<br>
>> [2023-07-18T14:59:45.026] error: Restart the slurmctld daemon to <br>
>> change GresPlugins<br>
>> [2023-07-18T14:59:45.026] select/cons_tres: select_p_reconfigure: <br>
>> select/cons_tres: reconfigure<br>
>> [2023-07-18T14:59:45.027] select/cons_tres: part_data_create_array: <br>
>> select/cons_tres: preparing for 2 partitions<br>
>> [2023-07-18T14:59:45.027] No parameter for mcs plugin, default values set<br>
>> [2023-07-18T14:59:45.027] mcs: MCSParameters = (null). ondemand set.<br>
>> [2023-07-18T14:59:45.028] _slurm_rpc_reconfigure_controller: completed <br>
>> usec=5898<br>
>> [2023-07-18T14:59:45.952] <br>
>> SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=2,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=2<br>
>><br>
>> I am a bit unsure what to do next to further investigate this issue.<br>
>><br>
>> Best regards,<br>
>> Xaver<br>
>><br>
>> On 17.07.23 15:57, Groner, Rob wrote:<br>
>>> That would certainly do it. If you look at the slurmctld log when it <br>
>>> comes up, it will say that it's marking that node as invalid because <br>
>>> it has less (0) gres resources then you say it should have. That's <br>
>>> because slurmd on that node will come up and say "What gres resources??"<br>
>>><br>
>>> For testing purposes, you can just create a dummy file on the node, <br>
>>> then in gres.conf, point to that file as the "graphics file" <br>
>>> interface. As long as you don't try to actually use it as a graphics <br>
>>> file, that should be enough for that node to think it has gres/gpu <br>
>>> resources. That's what I do in my vagrant slurm cluster.<br>
>>><br>
>>> Rob<br>
>>><br>
>>> ------------------------------------------------------------------------<br>
>>> *From:* slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf <br>
>>> of Xaver Stiensmeier <xaverstiensmeier@gmx.de><br>
>>> *Sent:* Monday, July 17, 2023 9:43 AM<br>
>>> *To:* slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
>>> *Subject:* Re: [slurm-users] GRES and GPUs<br>
>>> Hi Hermann,<br>
>>><br>
>>> Good idea, but we are already using `SelectType=select/cons_tres`. After<br>
>>> setting everything up again (in case I made an unnoticed mistake), I saw<br>
>>> that the node got marked STATE=inval.<br>
>>><br>
>>> To be honest, I thought I can just claim that a node has a gpu even if<br>
>>> it doesn't have one - just for testing purposes. Could this be the issue?<br>
>>><br>
>>> Best regards,<br>
>>> Xaver Stiensmeier<br>
>>><br>
>>> On 17.07.23 14:11, Hermann Schwärzler wrote:<br>
>>> > Hi Xaver,<br>
>>> ><br>
>>> > what kind of SelectType are you using in your slurm.conf?<br>
>>> ><br>
>>> > Per <br>
>>> <a href="https://slurm.schedmd.com/gres.html">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fgres.html&data=05%7C01%7Crug262%40psu.edu%7Ca605d51361ff4715490b08db8858d027%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638253687267659689%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fMujgwxjxUlumFDIXy60JKlBQz6Qy6kSxMNGDmnhOOo%3D&reserved=0</a>
<<a href="https://slurm.schedmd.com/gres.html">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fgres.html&data=05%7C01%7Crug262%40psu.edu%7Ca605d51361ff4715490b08db8858d027%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638253687267659689%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fMujgwxjxUlumFDIXy60JKlBQz6Qy6kSxMNGDmnhOOo%3D&reserved=0</a>>
you have to consider:<br>
>>> > "As for the --gpu* option, these options are only supported by Slurm's<br>
>>> > select/cons_tres plugin."<br>
>>> ><br>
>>> > So you can use "--gpus ..." only when you state<br>
>>> > SelectType = select/cons_tres<br>
>>> > in your slurm.conf.<br>
>>> ><br>
>>> > But "--gres=gpu:1" should work always.<br>
>>> ><br>
>>> > Regards<br>
>>> > Hermann<br>
>>> ><br>
>>> ><br>
>>> > On 7/17/23 13:43, Xaver Stiensmeier wrote:<br>
>>> >> Hey,<br>
>>> >><br>
>>> >> I am currently trying to understand how I can schedule a job that<br>
>>> >> needs a GPU.<br>
>>> >><br>
>>> >> I read about GRES <br>
>>> <a href="https://slurm.schedmd.com/gres.html">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fgres.html&data=05%7C01%7Crug262%40psu.edu%7Ca605d51361ff4715490b08db8858d027%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638253687267659689%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fMujgwxjxUlumFDIXy60JKlBQz6Qy6kSxMNGDmnhOOo%3D&reserved=0</a>
<<a href="https://slurm.schedmd.com/gres.html">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fgres.html&data=05%7C01%7Crug262%40psu.edu%7Ca605d51361ff4715490b08db8858d027%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638253687267659689%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fMujgwxjxUlumFDIXy60JKlBQz6Qy6kSxMNGDmnhOOo%3D&reserved=0</a>>
and tried to use:<br>
>>> >><br>
>>> >> GresTypes=gpu<br>
>>> >> NodeName=test Gres=gpu:1<br>
>>> >><br>
>>> >> But calling - after a 'sudo scontrol reconfigure':<br>
>>> >><br>
>>> >> srun --gpus 1 hostname<br>
>>> >><br>
>>> >> didn't work:<br>
>>> >><br>
>>> >> srun: error: Unable to allocate resources: Invalid generic resource<br>
>>> >> (gres) specification<br>
>>> >><br>
>>> >> so I read more <br>
>>> <a href="https://slurm.schedmd.com/gres.conf.html">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fgres.conf.html&data=05%7C01%7Crug262%40psu.edu%7Ca605d51361ff4715490b08db8858d027%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638253687267659689%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fpQKfO7iKJKZoUe15dPLKRl33XQwFC%2BFJ%2FqKlMtB2V0%3D&reserved=0</a>
<<a href="https://slurm.schedmd.com/gres.conf.html">https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fgres.conf.html&data=05%7C01%7Crug262%40psu.edu%7Ca605d51361ff4715490b08db8858d027%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638253687267659689%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fpQKfO7iKJKZoUe15dPLKRl33XQwFC%2BFJ%2FqKlMtB2V0%3D&reserved=0</a>>
but that<br>
>>> >> didn't really help me.<br>
>>> >><br>
>>> >><br>
>>> >> I am rather confused. GRES claims to be generic resources but then it<br>
>>> >> comes with three defined resources (GPU, MPS, MIG) and using one of<br>
>>> >> those didn't work in my case.<br>
>>> >><br>
>>> >> Obviously, I am misunderstanding something, but I am unsure where to<br>
>>> >> look.<br>
>>> >><br>
>>> >><br>
>>> >> Best regards,<br>
>>> >> Xaver Stiensmeier<br>
>>> >><br>
>>> ><br>
>>><br>
<br>
</div>
</span></font></div>
</body>
</html>