<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>On thing that seems concerning to me is that you may start a job
on a node before a currently running job has 'expanded' as much as
it will.</p>
<p>If there is 128G on the node and current job is using 64G but
will eventually use 112G, your approach could start another
similar job and they would both start swapping.</p>
<p>We had always pushed the users to know what they need before they
submit a job. They can ask for too much and then go down from
there, but it is really their responsibility to know what their
program will do. You are giving them the keys to a Tesla and they
want to blame you if they put the pedal to the metal and crash.
Learn the tools before you use them.</p>
<p>Brian Andrus<br>
</p>
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 5/29/2018 6:56 AM, PULIDO, Alexandre
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:C68A3AAF4A64CB4DBB3005DED53A103A117CA963@dg011mur-mx006.dg011.fg01.fr.space.corp">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Texte de bulles Car";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.m-1635612127832957102msolistparagraph, li.m-1635612127832957102msolistparagraph, div.m-1635612127832957102msolistparagraph
{mso-style-name:m_-1635612127832957102msolistparagraph;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.TextedebullesCar
{mso-style-name:"Texte de bulles Car";
mso-style-priority:99;
mso-style-link:"Texte de bulles";
font-family:"Tahoma","sans-serif";
mso-fareast-language:FR;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"
lang="EN-US">Thanks for your inputs, the automatic reporting
is definitely a great idea and seems easy to implement in
Slurm. At our site we have a web portal developed internally
where users can see in real time everything that is
happening on the cluster, and every metric of their own job.
There is especially a color code regarding the
under/overestimation of memory allocation.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"
lang="EN-US">We have constraints, we cannot afford loosing
time killing jobs, or performance if a 16G job is allocated
to a node where there is only 4 left.
<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"
lang="EN-US">In PBS taking into account the actual free
memory as a resource for allocation is a great way to handle
this. I find it too bad not to use Slurm’s allocation
algorithms and develop another, hacky one with “numerical
features” per node.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"
lang="EN-US">I’ll admit I’m not comfortable enough editing
the cons_res plugin source code, but there doesn’t seem to
be another way around for this need.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Alexandre<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De :</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
slurm-users [<a class="moz-txt-link-freetext" href="mailto:slurm-users-bounces@lists.schedmd.com">mailto:slurm-users-bounces@lists.schedmd.com</a>]
<b>De la part de</b> John Hearns<br>
<b>Envoyé :</b> mardi 29 mai 2018 13:16<br>
<b>À :</b> Slurm User Community List<br>
<b>Objet :</b> Re: [slurm-users] Using free memory available
when allocating a node to a job<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Alexandre, you have made a very good
point here. <span lang="EN-US">
"Oftentimes users only input 1G as they really have no
idea of the memory requirements,"</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">At my last job we
introduced cgroups. (this was in PBSPro). We had to
enforce a minumum request for memory.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Users then asked us
how much memory their jobs used - so that they could
request an amoutn of memory next time which would let
the job run to completion.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">We were giving users
information manually regarding how much memory their
jobs used.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I realise tha tthe
tools are there for users to get the information on
memory usage after a job, but I really do not expec
tusrs to have to figure this out.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">What do other sites
do in this case?</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 29 May 2018 at 12:57, PULIDO,
Alexandre <<a
href="mailto:alexandre.pulido@ariane.group"
target="_blank" moz-do-not-send="true">alexandre.pulido@ariane.group</a>>
wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">Hello John, this behavior is needed
because the memory usage of the codes executed on
the nodes are particularly hard to guess. Usually,
when exceeded the ratio is between 1.1 and 1.3 more
than expected. Sometimes much larger.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US"> </span><o:p></o:p></p>
<p class="m-1635612127832957102msolistparagraph"><span
lang="EN-US">A)</span><span style="font-size:7.0pt"
lang="EN-US">
</span><span lang="EN-US">Indeed there is a partition
running only exclusive jobs, but a large amounts of
nodes are also needed working on an nonexclusive
allocation. That’s why the exact amount of available
memory is required in this configuration. Tasks are
not killed if they take more than allocated.</span><o:p></o:p></p>
<p class="m-1635612127832957102msolistparagraph"><span
lang="EN-US">B)</span><span style="font-size:7.0pt"
lang="EN-US">
</span><span lang="EN-US">Yes currently cgroup is
configured and working as expected (I believe), but
as I said tasks need to grow larger.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">Oftentimes users only input 1G as they
really have no idea of the memory requirements, and
with the high demand of HPC time a lower memory
requirement is set so the job will start.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">So a job cannot be started on a node
where another job would be filling up the RAM, and
would start on another node.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">Would this behavior cause problems in
the scheduling/allocation algorithms ? The way I see
it the actual free memory would be just another
consumable resource.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">But the only way I can see this working
is by tweaking the plugin, correct ?</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">Thank you for your inputs.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"
lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"
lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De :</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
slurm-users [mailto:<a
href="mailto:slurm-users-bounces@lists.schedmd.com"
target="_blank" moz-do-not-send="true">slurm-users-bounces@lists.schedmd.com</a>]
<b>De la part de</b> John Hearns<br>
<b>Envoyé :</b> mardi 29 mai 2018 12:39<br>
<b>À :</b> Slurm User Community List<br>
<b>Objet :</b> Re: [slurm-users] Using free memory
available when allocating a node to a job</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Also
regarding memory, there are system tunings you
can set for the behaviour of the OurOfMemory
Killer and also the VM overcommit.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I
have seen the VM overcommit parameters being
discussed elsewhere, and generally for HPC
people advise to disable overcommit<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a
href="https://www.suse.com/support/kb/doc/?id=7002775" target="_blank"
moz-do-not-send="true">https://www.suse.com/support/kb/doc/?id=7002775</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This
of course is very dependent on what your
environment and applications are. Would you be
able to say please what the problems you are
having with memory?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On
29 May 2018 at 12:26, John Hearns <<a
href="mailto:hearnsj@googlemail.com"
target="_blank" moz-do-not-send="true">hearnsj@googlemail.com</a>>
wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Alexandre,
it would be helpful if you could say why
this behaviour is desirable.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">For
instance, do you have codes which need a
large amount of memory and your users are
seeing that these codes are crashing
because other codes running on the same
nodes are using memory.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I
have two thoughts:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">A)
enable job exclusive - ie run one job on
one compute node. Then that job has all
the memory.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This
is a very good way to run HPC in my
experience. Yes I know it is inefficient
if there are lots of single core jobs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">SO
this depends on what your mix of jobs is.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">B)
Have you considered implementing cgroups?
Then each job will be allocated memory and
cpu cores.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jobs
will not be able to grow larger than their
allocated cgroup limits.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I
would really ask you to consider cgroups.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On
29 May 2018 at 11:34, PULIDO,
Alexandre <<a
href="mailto:alexandre.pulido@ariane.group"
target="_blank"
moz-do-not-send="true">alexandre.pulido@ariane.group</a>>
wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">Hi,</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">in the cluster
where I'm deploying Slurm the
job allocation has to be based
on the actual free memory
available on the node, not just
the allocated by Slurm. This is
nonnegotiable and I understand
that it's not how Slurm is
designed to work, but I'm trying
anyway.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">Among the solutions
that I'm envisaging:</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">1) Create and
update periodically a numerical
node feature, with a string and
a special character separating
the wanted value (memfree_2048).
This definitely seems like a
mess to implement and too hacky,
but is there an equivalent to
PBS' numerical complexes and
sensors in Slurm?</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">2) Modifying the
select cons_res pluging to
compare against the actual free
memory instead of the allocated
memory. Is it as simple as
editing the "_add_job_to_res" (<a
href="https://github.com/SchedMD/slurm/blob/master/src/plugins/select/cons_res/select_cons_res.c#L816"
target="_blank"
moz-do-not-send="true">https://github.com/SchedMD/slurm/blob/master/src/plugins/select/cons_res/select_cons_res.c#L816</a>)
function and using the real left
memory ? I don't want to break
anything else so that's my main
question here, if you can guide
me towards the solution or other
thoughts on its feasibility.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US">Thanks a lot in
advance!</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
lang="EN-US"> </span><o:p></o:p></p>
<table class="MsoNormalTable"
style="width:423.75pt"
cellspacing="0" cellpadding="0"
width="565" border="0">
<tbody>
<tr>
<td style="padding:0cm 0cm 0cm
0cm">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Best
regards,<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm
0cm" valign="top"><br>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm
0cm"><br>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<table class="MsoNormalTable"
style="width:423.75pt"
cellspacing="0" cellpadding="0"
width="565" border="0">
<tbody>
<tr>
<td style="padding:0cm 0cm 0cm
0cm">
<table
class="MsoNormalTable"
style="width:423.75pt"
cellspacing="0"
cellpadding="0"
width="565" border="0">
<tbody>
<tr>
<td
style="width:33.75pt;padding:0cm
0cm 0cm 0cm"
width="45">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:#1F497D"><img
id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_13"
src="cid:part7.485FF269.59EC6B14@gmail.com" alt="px" class="" height="1"
width="55"
border="0"></span><o:p></o:p></p>
</td>
<td style="padding:0cm
0cm 0cm 0cm"
valign="top">
<table
class="MsoNormalTable"
cellspacing="0"
cellpadding="0"
border="0">
<tbody>
<tr
style="height:4.5pt">
<td
style="width:174.8pt;padding:0cm
0cm 0cm
0cm;height:4.5pt"
width="233">
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:4.5pt">
<span
style="font-size:1.0pt;color:#1F497D"><img
id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_12"
src="cid:part8.B2D50E49.6D4C60BF@gmail.com" alt="px" class="" height="6"
width="1"
border="0"></span><o:p></o:p></p>
</td>
</tr>
<tr>
<td
style="width:174.8pt;padding:0cm
0cm 0cm 0cm"
width="233">
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:12.0pt">
<b><span
style="font-size:10.5pt;font-family:"Arial","sans-serif";color:#2D3A44"
lang="EN-US">Alexandre
PULIDO</span></b><o:p></o:p></p>
</td>
</tr>
<tr
style="height:15.0pt">
<td
style="width:174.8pt;padding:0cm
0cm 0cm
0cm;height:15.0pt"
width="233">
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:1.0pt;color:#1F497D"><img
id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_11"
src="cid:part9.B33D7C3E.CA7BB8D3@gmail.com" alt="px" class=""
height="20"
width="1"
border="0"></span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm
0cm" valign="top">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:#1F497D"><img
id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_10"
src="cid:part10.9CCEC769.DC2EBAEF@gmail.com" alt="arianegroup" class=""
height="29"
width="170" border="0"></span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm
0cm">
<table
class="MsoNormalTable"
style="width:446.05pt"
cellspacing="0"
cellpadding="0"
width="595" border="0">
<tbody>
<tr>
<td
style="width:42.5pt;padding:0cm
0cm 0cm 0cm"
width="57">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:#1F497D"><img
id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_9"
src="cid:part7.485FF269.59EC6B14@gmail.com" alt="px" class="" height="1"
width="55"
border="0"></span><o:p></o:p></p>
</td>
<td
style="width:403.55pt;padding:0cm
0cm 0cm 0cm"
valign="top"
width="538">
<table
class="MsoNormalTable"
style="width:405.35pt" cellspacing="0" cellpadding="0" width="540"
border="0">
<tbody>
<tr
style="height:9.0pt">
<td
style="width:405.35pt;padding:0cm
0cm 0cm
0cm;height:9.0pt"
width="540">
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:9.0pt">
<span
style="font-size:1.0pt;color:#1F497D"><img
id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_8"
src="cid:part12.AA9229DA.16F2D5B0@gmail.com" alt="px" class=""
height="12"
width="1"
border="0"></span><o:p></o:p></p>
</td>
</tr>
<tr
style="height:4.0pt">
<td
style="width:405.35pt;padding:0cm
0cm 0cm
0cm;height:4.0pt"
width="540">
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:4.0pt">
<span
style="font-size:1.0pt;color:#1F497D"><img
id="m_-1635612127832957102m_-299635788708544835m_6621646543627534914Image_x0020_1"
src="cid:part9.B33D7C3E.CA7BB8D3@gmail.com" alt="px" class=""
height="20"
width="1"
border="0"></span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Ce
courriel (incluant ses éventuelles
pièces jointes) peut contenir des
informations confidentielles et/ou
protégées ou dont la diffusion est
restreinte ou soumise aux
règlementations relatives au
contrôle des exportations ou ayant
un caractère privé. Si vous avez
reçu ce courriel par erreur, vous ne
devez ni le reproduire, ni
l'utiliser, ni en divulguer le
contenu à quiconque. Merci d'en
avertir immédiatement l'expéditeur
et de supprimer de votre système
informatique ce courriel ainsi que
tous les documents qui y sont
attachés. Toute exportation ou
réexportation non autorisée est
interdite. ArianeGroup SAS décline
toute responsabilité en cas de
corruption par virus, d'altération
ou de falsification de ce courriel
lors de sa transmission par voie
électronique. This email (including
any attachments) may contain
confidential or proprietary and/or
privileged information or
information otherwise protected from
disclosure or may be subject to
export control laws and regulations.
If you are not the intended
recipient, please notify the sender
immediately, do not reproduce this
message or any attachments and do
not use it for any purpose or
disclose its content to any person,
but delete this message and any
attachments from your system.
Unauthorized export or re-export is
prohibited. ArianeGroup SAS
disclaims any and all liability if
this email transmission was virus
corrupted, altered or falsified.
ArianeGroup SAS (519 032 247 RCS
PARIS) - Capital social : 265 904
408 EUR - Siège social : Tour
Cristal, <a
href="https://maps.google.com/?q=7-11+Quai+Andr%C3%A9+Citro%C3%ABn,+75015+Paris&entry=gmail&source=g"
target="_blank"
moz-do-not-send="true">
7-11 Quai André Citroën, 75015
Paris</a> - TVA FR 82 519 032 247
- APE/NAF 3030Z <o:p>
</o:p></p>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Ce courriel (incluant ses
éventuelles pièces jointes) peut contenir des
informations confidentielles et/ou protégées ou dont
la diffusion est restreinte ou soumise aux
règlementations relatives au contrôle des
exportations ou ayant un caractère privé. Si vous
avez reçu ce courriel par erreur, vous ne devez ni
le reproduire, ni l'utiliser, ni en divulguer le
contenu à quiconque. Merci d'en avertir
immédiatement l'expéditeur et de supprimer de votre
système informatique ce courriel ainsi que tous les
documents qui y sont attachés. Toute exportation ou
réexportation non autorisée est interdite.
ArianeGroup SAS décline toute responsabilité en cas
de corruption par virus, d'altération ou de
falsification de ce courriel lors de sa transmission
par voie électronique. This email (including any
attachments) may contain confidential or proprietary
and/or privileged information or information
otherwise protected from disclosure or may be
subject to export control laws and regulations. If
you are not the intended recipient, please notify
the sender immediately, do not reproduce this
message or any attachments and do not use it for any
purpose or disclose its content to any person, but
delete this message and any attachments from your
system. Unauthorized export or re-export is
prohibited. ArianeGroup SAS disclaims any and all
liability if this email transmission was virus
corrupted, altered or falsified. ArianeGroup SAS
(519 032 247 RCS PARIS) - Capital social : 265 904
408 EUR - Siège social : Tour Cristal,
<a
href="https://maps.google.com/?q=7-11+Quai+Andr%C3%A9+Citro%C3%ABn,+75015+Paris&entry=gmail&source=g"
moz-do-not-send="true">
7-11 Quai André Citroën, 75015 Paris</a> - TVA FR
82 519 032 247 - APE/NAF 3030Z <o:p>
</o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
Ce courriel (incluant ses éventuelles pièces jointes) peut
contenir des informations confidentielles et/ou protégées ou dont
la diffusion est restreinte ou soumise aux règlementations
relatives au contrôle des exportations ou ayant un caractère
privé. Si vous avez reçu ce courriel par erreur, vous ne devez ni
le reproduire, ni l'utiliser, ni en divulguer le contenu à
quiconque. Merci d'en avertir immédiatement l'expéditeur et de
supprimer de votre système informatique ce courriel ainsi que tous
les documents qui y sont attachés. Toute exportation ou
réexportation non autorisée est interdite. ArianeGroup SAS décline
toute responsabilité en cas de corruption par virus, d'altération
ou de falsification de ce courriel lors de sa transmission par
voie électronique.
This email (including any attachments) may contain confidential or
proprietary and/or privileged information or information otherwise
protected from disclosure or may be subject to export control laws
and regulations. If you are not the intended recipient, please
notify the sender immediately, do not reproduce this message or
any attachments and do not use it for any purpose or disclose its
content to any person, but delete this message and any attachments
from your system. Unauthorized export or re-export is prohibited.
ArianeGroup SAS disclaims any and all liability if this email
transmission was virus corrupted, altered or falsified.
ArianeGroup SAS (519 032 247 RCS PARIS) - Capital social : 265 904
408 EUR - Siège social : Tour Cristal, 7-11 Quai André Citroën,
75015 Paris - TVA FR 82 519 032 247 - APE/NAF 3030Z
</blockquote>
<br>
</body>
</html>