<div dir="ltr">+1 on checking the memory allocation. <div>Or add/check if you have any DefMemPerX set in your slurm.conf</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 19, 2024 at 12:33 AM mohammed shambakey <<a href="mailto:shambakey1@gmail.com">shambakey1@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi</div><div><br>I'm not an expert, but is it possible that the currently running jobs is consuming the whole node because it is allocated the whole memory of the node (so the other 2 jobs had to wait until it finishes)?<br>Maybe if you try to restrict the required memory for each job? </div><div><br></div><div>Regards</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 18, 2024 at 4:46 PM Ümit Seren <<a href="mailto:uemit.seren@gmail.com" target="_blank">uemit.seren@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div lang="en-AT">
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">This line also has tobe changed:
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"><br>
</span><span style="color:black;background:white">#SBATCH --gpus-per-node=4</span><span style="color:black;background:white">
</span><span lang="EN-US" style="font-family:Wingdings;color:black;background:white"></span><span lang="EN-US" style="color:black;background:white">
</span><span style="color:black;background:white">#SBATCH --gpus-per-node=</span><span lang="EN-US" style="color:black;background:white">1<br>
<br>
</span><span style="color:black;background:white">--gpus-per-node</span><span lang="EN-US" style="color:black;background:white"> seems to be the new parameter that is replacing the
</span><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"> </span><span lang="EN-US" style="font-size:11pt">--gres= one, so you can remove the –gres line completely.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">Best<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">Ümit<br>
<br>
</span><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif"><u></u> <u></u></span></p>
<div id="m_44866278241234610m_7475454801666884512mail-editor-reference-message-container">
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12pt"><b><span lang="FR" style="color:black">From:
</span></b><span lang="FR" style="color:black">slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Kherfani, Hafedh (Professional Services, TC) <<a href="mailto:hafedh.kherfani@hpe.com" target="_blank">hafedh.kherfani@hpe.com</a>><br>
<b>Date: </b>Thursday, 18. January 2024 at 15:40<br>
<b>To: </b>Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject: </b>Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres)<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="FR" style="font-size:11pt">Hi Noam and Matthias,</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="FR" style="font-size:11pt"> </span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">Thanks both for your answers.</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> </span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">I changed the “#SBATCH --gres=gpu:4“ directive (in the batch script) with “#SBATCH --gres=gpu:1“ as you suggested, but it didn’t make a difference, as running this batch script 3 times will result
in the first job to be in a running state, while the second and third jobs will still be in a pending state …
</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> </span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">[slurmtest@c-a100-master test-batch-scripts]$ cat gpu-job.sh</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">#!/bin/bash</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">#SBATCH --job-name=gpu-job</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">#SBATCH --partition=gpu</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">#SBATCH --nodes=1</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">#SBATCH --gpus-per-node=4</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">#SBATCH --gres=gpu:1 # <<<< Changed from ‘4’ to ‘1’
</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">#SBATCH --tasks-per-node=1</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">#SBATCH --output=gpu_job_output.%j
</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">#SBATCH --error=gpu_job_error.%j
</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> </span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">hostname</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">date</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">sleep 40</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">pwd</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> </span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">[slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">Submitted batch job
<b>217</b></span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">[slurmtest@c-a100-master test-batch-scripts]$ squeue</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> 217 gpu gpu-job slurmtes R 0:02 1 c-a100-cn01</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">[slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">Submitted batch job
<b>218</b></span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">[slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">Submitted batch job
<b>219</b></span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">[slurmtest@c-a100-master test-batch-scripts]$ squeue</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> 219 gpu gpu-job slurmtes
<b>PD</b> 0:00 1 (Priority)</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> 218 gpu gpu-job slurmtes
<b>PD</b> 0:00 1 (Resources)</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> 217 gpu gpu-job slurmtes
<b>R</b> 0:07 1 c-a100-cn01</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> </span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">Basically I’m seeking for some help/hints on how to tell Slurm, from the batch script for example: “I want only 1 or 2 GPUs to be used/consumed by the job”, and then I run the batch script/job
a couple of times with sbatch command, and confirm that we can indeed have multiple jobs using a GPU and running in parallel, at the same time.
</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> </span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt">Makes sense ? </span>
<span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> </span><span lang="FR"><u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt"> </span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Arial,sans-serif;color:black">Best regards,</span><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal" style="break-after:avoid"><b><span lang="EN-US" style="font-size:10pt;font-family:Arial,sans-serif;color:black"> </span></b><span lang="FR"><u></u><u></u></span></p>
<p class="MsoNormal" style="break-after:avoid"><b><span lang="EN-US" style="font-size:10pt;font-family:Arial,sans-serif;color:black">Hafedh
</span></b><span lang="FR"><u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="FR"> <u></u><u></u></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>>
<b>On Behalf Of </b>Bernstein, Noam CIV USN NRL (6393) Washington DC (USA)<br>
<b>Sent:</b> jeudi 18 janvier 2024 2:30 PM<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> Re: [slurm-users] Need help with running multiple instances/executions of a batch script in parallel (with NVIDIA HGX A100 GPU as a Gres)</span><span lang="FR"><u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="FR"> <u></u><u></u></span></p>
<div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal"><span lang="FR">On Jan 18, 2024, at 7:31 AM, Matthias Loose <<a href="mailto:m.loose@mindcode.de" target="_blank">m.loose@mindcode.de</a>> wrote:<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="FR"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="FR">Hi Hafedh,<br>
<br>
Im no expert in the GPU side of SLURM, but looking at you current configuration to me its working as intended at the moment. You have defined 4 GPUs and start multiple jobs each consuming 4 GPUs each. So the jobs wait for the ressource the be free again.<br>
<br>
I think what you need to look into is the MPS plugin, which seems to do what you are trying to achieve:<br>
<a href="https://slurm.schedmd.com/gres.html#MPS_Management" target="_blank">https://slurm.schedmd.com/gres.html#MPS_Management</a><u></u><u></u></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="FR"> <u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="FR">I agree with the first paragraph. How many GPUs are you expecting each job to use? I'd have assumed, based on the original text, that each job is supposed to use 1 GPU, and the 4 jobs were supposed to be running side-by-side
on the one node you have (with 4 GPUs). If so, you need to tell each job to request only 1 GPU, and currently each one is requesting 4.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="FR"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="FR">If your jobs are actually supposed to be using 4 GPUs each, I still don't see any advantage to MPS (at least in what is my usual GPU usage pattern): all the jobs will take longer to finish, because they are sharing the fixed
resource. If they take turns, at least the first ones finish as fast as they can, and the last one will finish no later than it would have if they were all time-sharing the GPUs. I guess NVIDIA had something in mind when they developed MPS, so I guess our
pattern may not be typical (or at least not universal), and in that case the MPS plugin may well be what you need.<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Mohammed<br></div>
</blockquote></div>