<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I would set “--ntasks"= number of cpus you want use for your job and remove “--cpus-per-task” which would be set to 1 by default.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Selch, Brigitte (FIDD)<br>
<b>Sent:</b> Friday, September 22, 2023 7:58 AM<br>
<b>To:</b> slurm-users@schedmd.com<br>
<b>Subject:</b> [EXT] [slurm-users] Submitting hybrid OpenMPI and OpenMP Jobs<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div id="APLWarningText">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td width="100%" style="width:100.0%;background:#E0E0E0;padding:0in 0in 0in 0in">
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<b><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:red">APL external email warning:
</span></b><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:black">Verify sender
<a href="mailto:slurm-users-bounces@lists.schedmd.com">slurm-users-bounces@lists.schedmd.com</a> before clicking links or attachments</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p><span lang="DE"> <o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="DE">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE"><o:p> </o:p></span></p>
<p class="MsoNormal">one of our applications need hybrid OpenMPI and OpenMP Job-Submit.<o:p></o:p></p>
<p class="MsoNormal">Only one task is allowed on one node, but this task should use all cores of the node.<o:p></o:p></p>
<p class="MsoNormal">So, for example I made:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i>#!/bin/bash<o:p></o:p></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i><o:p> </o:p></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i>#SBATCH --nodes=5<o:p></o:p></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i>#SBATCH --ntasks=5<o:p></o:p></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i>#SBATCH --cpus-per-task=44<o:p></o:p></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i>#SBATCH --export=ALL<o:p></o:p></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i><o:p> </o:p></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i>export OMP_NUM_THREADS=44<o:p></o:p></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i>mpiexec PreonNode test.prscene<o:p></o:p></i></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:black;mso-fareast-language:DE">But the job does not take more than one thread:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black;mso-fareast-language:DE">…<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i><span style="color:black;mso-fareast-language:DE">Thread binding will be disabled because the full machine is not available for the process.<o:p></o:p></span></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><b><i><span style="color:black;mso-fareast-language:DE">Detected 44 CPU threads</span></i></b><i><span style="color:black;mso-fareast-language:DE">, 2 l3 caches and 2 packages on the machine.<o:p></o:p></span></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><b><i><span style="color:black;mso-fareast-language:DE">Number of CPU processors reported by OpenMP: 1<o:p></o:p></span></i></b></p>
<p class="MsoNormal" style="margin-left:35.4pt"><b><i><span style="color:black;mso-fareast-language:DE">Maximum number of CPU threads reported by OpenMP: 44<o:p></o:p></span></i></b></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i><span style="color:black;mso-fareast-language:DE"><o:p> </o:p></span></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i><span style="color:black;mso-fareast-language:DE">Warning:
<b>OMP_NUM_THREADS was set to 44, which is higher than the number of available processors of
</b>1. Will use 1 threads now.<o:p></o:p></span></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i><span style="color:black;mso-fareast-language:DE">…<o:p></o:p></span></i></p>
<p class="MsoNormal"><span style="color:black;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black;mso-fareast-language:DE">What did I wrong?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black;mso-fareast-language:DE">Does anyone have any idea why OpenMP thinks it can only use one thread per node?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black;mso-fareast-language:DE">Thanks !<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black;mso-fareast-language:DE">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black;mso-fareast-language:DE">Brigitte Selch<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="color:black;mso-fareast-language:DE"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="color:black;mso-fareast-language:DE">MAN Truck & Bus SE<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="DE" style="color:black;mso-fareast-language:DE">IT Produktentwicklung Simulation (FIDD)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="color:black;mso-fareast-language:DE">Vogelweiher Str. 33<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="color:black;mso-fareast-language:DE">90441 Nürnberg<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span lang="DE" style="font-size:12.0pt;font-family:"Times New Roman",serif">
<hr size="2" width="100%" align="center">
</span></div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="DE" style="font-size:7.5pt;font-family:"Arial",sans-serif;color:gray"><br>
MAN Truck & Bus SE<br>
Sitz der Gesellschaft: München<br>
Registergericht: Amtsgericht München, HRB 247520<br>
Vorsitzender des Aufsichtsrats: Christian Levin, Vorstand: Alexander Vlaskamp (Vorsitzender), Murat Aksel, Friedrich-W. Baumann, Michael Kobriger, Inka Koljonen, Arne Puls, Dr. Frederik Zohm<br>
<br>
You can find information about how we process your personal data and your rights in our data protection notice:
<a href="http://www.man.eu/data-protection-notice">www.man.eu/data-protection-notice</a><br>
<br>
This e-mail (including any attachments) is confidential and may be privileged.<br>
If you have received it by mistake, please notify the sender by e-mail and delete this message from your system.<br>
Any unauthorised use or dissemination of this e-mail in whole or in part is strictly prohibited.<br>
Please note that e-mails are susceptible to change.<br>
MAN Truck & Bus SE (including its group companies) shall not be liable for the improper or incomplete transmission of the information contained in this communication nor for any delay in its receipt.<br>
MAN Truck & Bus SE (or its group companies) does not guarantee that the integrity of this communication has been maintained nor that this communication is free of viruses, interceptions or interference.</span><span lang="DE" style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p></o:p></span></p>
</div>
</body>
</html>