<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
I had this same issue today again.<br>
<br>
<blockquote type="cite">
<p class="MsoNormal"><span lang="EN-US">sbatch: error: CPU count
per node can not be satisfied</span></p>
<span lang="EN-US">sbatch: error: Batch job submission failed:
Requested node configuration is not available</span></blockquote>
<br>
After restarting slurmctld, the user could submit his job with the
very same jobscript.<br>
<br>
One of the oddities of SLURM we have learned to live with.<br>
<br>
Best<br>
Marcus<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 2/5/20 3:00 AM,
<a class="moz-txt-link-abbreviated" href="mailto:dean.w.schulze@gmail.com">dean.w.schulze@gmail.com</a> wrote:<br>
</div>
<blockquote type="cite"
cite="mid:104801d5dbc8$09a1abb0$1ce50310$@gmail.com">
<pre class="moz-quote-pre" wrap="">
This started working for me this morning. I have no idea why it started to work. Maybe it was multiple restarts of the various daemons that did it.
-----Original Message-----
From: slurm-users <a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> On Behalf Of Brian W. Johanson
Sent: Tuesday, February 4, 2020 1:35 PM
To: <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a>
Subject: Re: [slurm-users] sbatch script won't accept --gres that requires more than 1 gpu
Please include the output for:
scontrol show node=liqidos-dean-node1
scontrol show partition=Partition_you_are_attempting_to_submit_to
and
any other #SBATCH lines submitted with the failing job.
On 2/4/20 9:42 AM, <a class="moz-txt-link-abbreviated" href="mailto:dean.w.schulze@gmail.com">dean.w.schulze@gmail.com</a> wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">I've already restarted slurmctld and slurmd on all nodes. Still get the same problem.
-----Original Message-----
From: slurm-users <a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> On Behalf Of
Marcus Wagner
Sent: Tuesday, February 4, 2020 2:31 AM
To: <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a>
Subject: Re: [slurm-users] sbatch script won't accept --gres that
requires more than 1 gpu
Hi Dean,
could you please try to restart the slurmctld?
This usually helps on our site.
Never saw this with gres happening, but many other times.
This is, why we restart slurmctld once a day by a cron job.
Best
Marcus
On 2/4/20 12:59 AM, Dean Schulze wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">When I run an sbatch script with the line
#SBATCH --gres=gpu:gp100:1
it runs. When I change it to
#SBATCH --gres=gpu:gp100:3
it fails with "Requested node configuration is not available". But I
have a node with 4 gp100s available. Here's my slurm.conf:
NodeName=liqidos-dean-node1 CPUs=2 Boards=1 SocketsPerBoard=2
CoresPerSocket=1 ThreadsPerCore=1 RealMemory=3770 Gres=gpu:gp100:4
That node has a gres.conf with these lines:
Name=gpu Type=gp100 File=/dev/nvidia0 Name=gpu Type=gp100
File=/dev/nvidia1 Name=gpu Type=gp100 File=/dev/nvidia2 Name=gpu
Type=gp100 File=/dev/nvidia3
The character devices all exist in /dev.
What's the controller complaining about?
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de">www.itc.rwth-aachen.de</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de">www.itc.rwth-aachen.de</a>
</pre>
</body>
</html>