<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Texte de bulles Car";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.m-1635612127832957102msolistparagraph, li.m-1635612127832957102msolistparagraph, div.m-1635612127832957102msolistparagraph
{mso-style-name:m_-1635612127832957102msolistparagraph;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.TextedebullesCar
{mso-style-name:"Texte de bulles Car";
mso-style-priority:99;
mso-style-link:"Texte de bulles";
font-family:"Tahoma","sans-serif";
mso-fareast-language:FR;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks for your inputs, the automatic reporting is definitely a great idea and seems easy to implement in Slurm. At our site we have a web portal
developed internally where users can see in real time everything that is happening on the cluster, and every metric of their own job. There is especially a color code regarding the under/overestimation of memory allocation.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We have constraints, we cannot afford loosing time killing jobs, or performance if a 16G job is allocated to a node where there is only 4 left.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">In PBS taking into account the actual free memory as a resource for allocation is a great way to handle this. I find it too bad not to use Slurm’s
allocation algorithms and develop another, hacky one with “numerical features” per node.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’ll admit I’m not comfortable enough editing the cons_res plugin source code, but there doesn’t seem to be another way around for this need.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Alexandre<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De :</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> slurm-users [mailto:slurm-users-bounces@lists.schedmd.com]
<b>De la part de</b> John Hearns<br>
<b>Envoyé :</b> mardi 29 mai 2018 13:16<br>
<b>À :</b> Slurm User Community List<br>
<b>Objet :</b> Re: [slurm-users] Using free memory available when allocating a node to a job<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Alexandre, you have made a very good point here. <span lang="EN-US">
"Oftentimes users only input 1G as they really have no idea of the memory requirements,"</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">At my last job we introduced cgroups. (this was in PBSPro). We had to enforce a minumum request for memory.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Users then asked us how much memory their jobs used - so that they could request an amoutn of memory next time which would let the job run to completion.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">We were giving users information manually regarding how much memory their jobs used.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I realise tha tthe tools are there for users to get the information on memory usage after a job, but I really do not expec tusrs to have to figure this out.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">What do other sites do in this case?</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 29 May 2018 at 12:57, PULIDO, Alexandre <<a href="mailto:alexandre.pulido@ariane.group" target="_blank">alexandre.pulido@ariane.group</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Hello John, this behavior is needed because the memory usage of the codes executed on the nodes are particularly hard to guess. Usually, when exceeded the ratio
is between 1.1 and 1.3 more than expected. Sometimes much larger.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="m-1635612127832957102msolistparagraph"><span lang="EN-US">A)</span><span lang="EN-US" style="font-size:7.0pt">
</span><span lang="EN-US">Indeed there is a partition running only exclusive jobs, but a large amounts of nodes are also needed working on an nonexclusive allocation. That’s why the exact amount of available memory is required in this configuration. Tasks are
not killed if they take more than allocated.</span><o:p></o:p></p>
<p class="m-1635612127832957102msolistparagraph"><span lang="EN-US">B)</span><span lang="EN-US" style="font-size:7.0pt">
</span><span lang="EN-US">Yes currently cgroup is configured and working as expected (I believe), but as I said tasks need to grow larger.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Oftentimes users only input 1G as they really have no idea of the memory requirements, and with the high demand of HPC time a lower memory requirement is set
so the job will start.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">So a job cannot be started on a node where another job would be filling up the RAM, and would start on another node.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Would this behavior cause problems in the scheduling/allocation algorithms ? The way I see it the actual free memory would be just another consumable resource.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">But the only way I can see this working is by tweaking the plugin, correct ?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Thank you for your inputs.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De :</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> slurm-users [mailto:<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>]
<b>De la part de</b> John Hearns<br>
<b>Envoyé :</b> mardi 29 mai 2018 12:39<br>
<b>À :</b> Slurm User Community List<br>
<b>Objet :</b> Re: [slurm-users] Using free memory available when allocating a node to a job</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Also regarding memory, there are system tunings you can set for the behaviour of the OurOfMemory Killer and also the VM overcommit.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I have seen the VM overcommit parameters being discussed elsewhere, and generally for HPC people advise to disable overcommit<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="https://www.suse.com/support/kb/doc/?id=7002775" target="_blank">https://www.suse.com/support/kb/doc/?id=7002775</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This of course is very dependent on what your environment and applications are. Would you be able to say please what the problems you are having with memory?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On 29 May 2018 at 12:26, John Hearns <<a href="mailto:hearnsj@googlemail.com" target="_blank">hearnsj@googlemail.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Alexandre, it would be helpful if you could say why this behaviour is desirable.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">For instance, do you have codes which need a large amount of memory and your users are seeing that these codes are crashing because other codes running on the same nodes are using
memory.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I have two thoughts:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">A) enable job exclusive - ie run one job on one compute node. Then that job has all the memory.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This is a very good way to run HPC in my experience. Yes I know it is inefficient if there are lots of single core jobs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">SO this depends on what your mix of jobs is.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">B) Have you considered implementing cgroups? Then each job will be allocated memory and cpu cores.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jobs will not be able to grow larger than their allocated cgroup limits.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I would really ask you to consider cgroups.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On 29 May 2018 at 11:34, PULIDO, Alexandre <<a href="mailto:alexandre.pulido@ariane.group" target="_blank">alexandre.pulido@ariane.group</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Hi,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">in the cluster where I'm deploying Slurm the job allocation has to be based on the actual free memory available on the node, not just the allocated by Slurm.
This is nonnegotiable and I understand that it's not how Slurm is designed to work, but I'm trying anyway.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Among the solutions that I'm envisaging:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">1) Create and update periodically a numerical node feature, with a string and a special character separating the wanted value (memfree_2048). This definitely
seems like a mess to implement and too hacky, but is there an equivalent to PBS' numerical complexes and sensors in Slurm?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">2) Modifying the select cons_res pluging to compare against the actual free memory instead of the allocated memory. Is it as simple as editing the "_add_job_to_res"
(<a href="https://github.com/SchedMD/slurm/blob/master/src/plugins/select/cons_res/select_cons_res.c#L816" target="_blank">https://github.com/SchedMD/slurm/blob/master/src/plugins/select/cons_res/select_cons_res.c#L816</a>) function and using the real left
memory ? I don't want to break anything else so that's my main question here, if you can guide me towards the solution or other thoughts on its feasibility.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Thanks a lot in advance!</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="565" style="width:423.75pt">
<tbody>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Best regards,<o:p></o:p></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:0cm 0cm 0cm 0cm"></td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm"></td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="565" style="width:423.75pt">
<tbody>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="565" style="width:423.75pt">
<tbody>
<tr>
<td width="45" style="width:33.75pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"><img border="0" width="55" height="1" id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_13" src="cid:image001.png@01D3F764.41513E30" alt="px"></span><o:p></o:p></p>
</td>
<td valign="top" style="padding:0cm 0cm 0cm 0cm">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr style="height:4.5pt">
<td width="233" style="width:174.8pt;padding:0cm 0cm 0cm 0cm;height:4.5pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:4.5pt">
<span style="font-size:1.0pt;color:#1F497D"><img border="0" width="1" height="6" id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_12" src="cid:image002.png@01D3F764.41513E30" alt="px"></span><o:p></o:p></p>
</td>
</tr>
<tr>
<td width="233" style="width:174.8pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:12.0pt">
<b><span lang="EN-US" style="font-size:10.5pt;font-family:"Arial","sans-serif";color:#2D3A44">Alexandre PULIDO</span></b><o:p></o:p></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="233" style="width:174.8pt;padding:0cm 0cm 0cm 0cm;height:15.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:1.0pt;color:#1F497D"><img border="0" width="1" height="20" id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_11" src="cid:image003.png@01D3F764.41513E30" alt="px"></span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td valign="top" style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"><img border="0" width="170" height="29" id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_10" src="cid:image004.jpg@01D3F764.41513E30" alt="arianegroup"></span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="595" style="width:446.05pt">
<tbody>
<tr>
<td width="57" style="width:42.5pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"><img border="0" width="55" height="1" id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_9" src="cid:image001.png@01D3F764.41513E30" alt="px"></span><o:p></o:p></p>
</td>
<td width="538" valign="top" style="width:403.55pt;padding:0cm 0cm 0cm 0cm">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="540" style="width:405.35pt">
<tbody>
<tr style="height:9.0pt">
<td width="540" style="width:405.35pt;padding:0cm 0cm 0cm 0cm;height:9.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:9.0pt">
<span style="font-size:1.0pt;color:#1F497D"><img border="0" width="1" height="12" id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_8" src="cid:image005.png@01D3F764.41513E30" alt="px"></span><o:p></o:p></p>
</td>
</tr>
<tr style="height:4.0pt">
<td width="540" style="width:405.35pt;padding:0cm 0cm 0cm 0cm;height:4.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:4.0pt">
<span style="font-size:1.0pt;color:#1F497D"><img border="0" width="1" height="20" id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_1" src="cid:image003.png@01D3F764.41513E30" alt="px"></span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Ce courriel (incluant ses éventuelles pièces jointes) peut contenir des informations confidentielles et/ou protégées ou dont la diffusion est restreinte ou soumise aux règlementations
relatives au contrôle des exportations ou ayant un caractère privé. Si vous avez reçu ce courriel par erreur, vous ne devez ni le reproduire, ni l'utiliser, ni en divulguer le contenu à quiconque. Merci d'en avertir immédiatement l'expéditeur et de supprimer
de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés. Toute exportation ou réexportation non autorisée est interdite. ArianeGroup SAS décline toute responsabilité en cas de corruption par virus, d'altération ou de falsification
de ce courriel lors de sa transmission par voie électronique. This email (including any attachments) may contain confidential or proprietary and/or privileged information or information otherwise protected from disclosure or may be subject to export control
laws and regulations. If you are not the intended recipient, please notify the sender immediately, do not reproduce this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments
from your system. Unauthorized export or re-export is prohibited. ArianeGroup SAS disclaims any and all liability if this email transmission was virus corrupted, altered or falsified. ArianeGroup SAS (519 032 247 RCS PARIS) - Capital social : 265 904 408 EUR
- Siège social : Tour Cristal, <a href="https://maps.google.com/?q=7-11+Quai+Andr%C3%A9+Citro%C3%ABn,+75015+Paris&entry=gmail&source=g" target="_blank">
7-11 Quai André Citroën, 75015 Paris</a> - TVA FR 82 519 032 247 - APE/NAF 3030Z <o:p>
</o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Ce courriel (incluant ses éventuelles pièces jointes) peut contenir des informations confidentielles et/ou protégées ou dont la diffusion est restreinte ou soumise aux règlementations relatives au contrôle des exportations ou ayant un caractère
privé. Si vous avez reçu ce courriel par erreur, vous ne devez ni le reproduire, ni l'utiliser, ni en divulguer le contenu à quiconque. Merci d'en avertir immédiatement l'expéditeur et de supprimer de votre système informatique ce courriel ainsi que tous les
documents qui y sont attachés. Toute exportation ou réexportation non autorisée est interdite. ArianeGroup SAS décline toute responsabilité en cas de corruption par virus, d'altération ou de falsification de ce courriel lors de sa transmission par voie électronique.
This email (including any attachments) may contain confidential or proprietary and/or privileged information or information otherwise protected from disclosure or may be subject to export control laws and regulations. If you are not the intended recipient,
please notify the sender immediately, do not reproduce this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Unauthorized export or re-export is
prohibited. ArianeGroup SAS disclaims any and all liability if this email transmission was virus corrupted, altered or falsified. ArianeGroup SAS (519 032 247 RCS PARIS) - Capital social : 265 904 408 EUR - Siège social : Tour Cristal,
<a href="https://maps.google.com/?q=7-11+Quai+Andr%C3%A9+Citro%C3%ABn,+75015+Paris&entry=gmail&source=g">
7-11 Quai André Citroën, 75015 Paris</a> - TVA FR 82 519 032 247 - APE/NAF 3030Z <o:p>
</o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
Ce courriel (incluant ses éventuelles pièces jointes) peut contenir des informations confidentielles et/ou protégées ou dont la diffusion est restreinte ou soumise aux règlementations relatives au contrôle des exportations ou ayant un caractère privé. Si vous avez reçu ce courriel par erreur, vous ne devez ni le reproduire, ni l'utiliser, ni en divulguer le contenu à quiconque. Merci d'en avertir immédiatement l'expéditeur et de supprimer de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés. Toute exportation ou réexportation non autorisée est interdite. ArianeGroup SAS décline toute responsabilité en cas de corruption par virus, d'altération ou de falsification de ce courriel lors de sa transmission par voie électronique.
This email (including any attachments) may contain confidential or proprietary and/or privileged information or information otherwise protected from disclosure or may be subject to export control laws and regulations. If you are not the intended recipient, please notify the sender immediately, do not reproduce this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Unauthorized export or re-export is prohibited. ArianeGroup SAS disclaims any and all liability if this email transmission was virus corrupted, altered or falsified. ArianeGroup SAS (519 032 247 RCS PARIS) - Capital social : 265 904 408 EUR - Siège social : Tour Cristal, 7-11 Quai André Citroën, 75015 Paris - TVA FR 82 519 032 247 - APE/NAF 3030Z
</body>
</html>