<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
@font-face
{font-family:"Open Sans";
panose-1:2 11 6 6 3 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.searchhighlight
{mso-style-name:searchhighlight;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:12.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">HI,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">So I’m testing the use of Open MPI 5.0.0 pre-release with the Slurm/PMIx setup currently on NERSC Perlmutter system.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">First off, if I use the PRRte launch system, I don’t see the issue I’m raising here.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">But, many NERSC users prefer to use the srun “native” launch method with applications compiled against Open MPI, hence this emal.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The SLURM version on Perlmutter is currently 2023.02.2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The PMIx version that the admins used to build slurm against is pmix-4.2.3. I’ve attached the output of pmix_info.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’ve tested with Open MPI 5.0.0rc11 (or HEAD of 5.0.x) with both the PMIx embedded in the Open MPI and using the external PMIx 4.2.3 install.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I get the same results below when my app is linked either against the system PMIx or the embedded one.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">My test application “works” but if I use srun, I get these types of messages:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">srun -n 2 -N 2 --mpi=pmix ./ring_c<o:p></o:p></span></p>
<p class="p1" style="margin:0in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">[cn316:2770176]</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">PMIX</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">ERROR</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">:
OUT-OF-RESOURCE in file base/bfrop_base_unpack.c at line 750</span></span><span style="font-size:8.5pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">[cn315:1037721]</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">PMIX</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">ERROR</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">:
OUT-OF-RESOURCE in file base/bfrop_base_unpack.c at line 750</span></span><span style="font-size:8.5pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">[cn315:1037721]</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">PMIX</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">ERROR</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">:
OUT-OF-RESOURCE in file base/bfrop_base_unpack.c at line 750</span></span><span style="font-size:8.5pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">[cn315:1037721]</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">PMIX</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">ERROR</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">:
UNPACK-INADEQUATE-SPACE in file base/gds_base_fns.c at line 268</span></span><span style="font-size:8.5pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">[cn315:1037721]</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">PMIX</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">ERROR</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">:
UNPACK-INADEQUATE-SPACE in file dstore_base.c at line 2624</span></span><span style="font-size:8.5pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">[cn315:1037721]</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">PMIX</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">ERROR</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">:
UNPACK-INADEQUATE-SPACE in file server/</span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">pmix</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">_server.c
at line 3417</span></span><span style="font-size:8.5pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">[cn316:2770176]</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">PMIX</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">ERROR</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">:
OUT-OF-RESOURCE in file base/bfrop_base_unpack.c at line 750</span></span><span style="font-size:8.5pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">[cn316:2770176]</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">PMIX</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">ERROR</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">:
UNPACK-INADEQUATE-SPACE in file base/gds_base_fns.c at line 268</span></span><span style="font-size:8.5pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">[cn316:2770176]</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">PMIX</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">ERROR</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">:
UNPACK-INADEQUATE-SPACE in file dstore_base.c at line 2624</span></span><span style="font-size:8.5pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0in;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">[cn316:2770176]</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">PMIX</span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo;color:black"> </span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">ERROR</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">:
UNPACK-INADEQUATE-SPACE in file server/</span></span><span class="searchhighlight"><span style="font-size:8.5pt;font-family:Menlo;color:#070706;background:#FFEE94">pmix</span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo;color:black">_server.c
at line 3417</span></span><span style="font-size:8.5pt;font-family:Menlo;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">After a lot of stracing and adding debug statements to the PMIx I have control over – the one in the embedded Open MPI tarball, I realized that these<br>
messages are not coming from the app, but some transient process between the srun/slurmd processes and the application processes.<br>
The pids in these error messages are the parents of the MPI processes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’ve tried various things like turning off the PMIX GDS shmem but that doesn’t help. Also I’ve toggled the various SLURM_PMIX env. variables but to no effect.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This problem does not appear to be related to a recent slurm/pmix patch -
</span><a href="https://bugs.schedmd.com/show_bug.cgi?id=16306#a0" target="_blank"><span style="font-size:10.0pt;font-family:"Open Sans",sans-serif;color:#386FE5">https://bugs.schedmd.com/show_bug.cgi?id=16306#a0</span></a> and anyway it looks like that patch
should be in 2023.02.2.<span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Another bit of info:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">scontrol show config | grep -i pmix<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxCliTmpDirBase = (null)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxCollFence = (null)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxDebug = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxDirectConn = yes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxDirectConnEarly = no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxDirectConnUCX = no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxDirectSameArch = no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxEnv = (null)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxFenceBarrier = no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxNetDevicesUCX = (null)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxTimeout = 300<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">PMIxTlsUCX = (null)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Now I myself don’t care too much about these messages.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">But for users it might be disconcerting and also may cause automated regression testing frameworks to report lots of errors.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Should I ask NERSC to file a ticket with SchedMD or does someone know how to turn these messages off if in fact they are not important, or better yet know why a slurm process may be emitting these errors and
how to fix it?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for any ideas,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Howard<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#0B1A8D;mso-ligatures:none"><br>
—</span><span style="font-size:11.0pt;mso-ligatures:none"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#0B1A8D;mso-ligatures:none"><o:p> </o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="375" style="width:281.25pt;border-collapse:collapse">
<tbody>
<tr style="height:104.45pt">
<td width="78" valign="top" style="width:58.4pt;padding:0in 5.4pt 0in 5.4pt;height:104.45pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;mso-ligatures:none"><img border="0" width="55" height="55" style="width:.5729in;height:.5729in" id="Picture_x0020_5" src="cid:image001.png@01D98A53.1A78BFE0" alt="signature_61897647"><o:p></o:p></span></p>
</td>
<td width="297" valign="top" style="width:223.1pt;padding:0in 5.4pt 0in 5.4pt;height:104.45pt">
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#0B1A8C;mso-ligatures:none">Howard Pritchard<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#545961;mso-ligatures:none">Research Scientist<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#545961;mso-ligatures:none">HPC-ENV<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;mso-ligatures:none"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;mso-ligatures:none">Los Alamos National Laboratory<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;mso-ligatures:none">howardp@lanl.gov<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;mso-ligatures:none"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><a href="https://www.instagram.com/losalamosnatlab/"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;text-decoration:none"><img border="0" width="27" height="27" style="width:.2812in;height:.2812in" id="Picture_x0020_4" src="cid:image002.png@01D98A53.1A78BFE0" alt="signature_2560999014"></span></b></a><a href="https://twitter.com/LosAlamosNatLab"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;text-decoration:none"><img border="0" width="27" height="27" style="width:.2812in;height:.2812in" id="Picture_x0020_3" src="cid:image003.png@01D98A53.1A78BFE0" alt="signature_3849187500"></span></b></a><a href="https://www.linkedin.com/company/los-alamos-national-laboratory/"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;text-decoration:none"><img border="0" width="27" height="27" style="width:.2812in;height:.2812in" id="Picture_x0020_2" src="cid:image004.png@01D98A53.1A78BFE0" alt="signature_1777390047"></span></b></a><a href="https://www.facebook.com/LosAlamosNationalLab/"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;text-decoration:none"><img border="0" width="27" height="27" style="width:.2812in;height:.2812in" id="Picture_x0020_1" src="cid:image005.png@01D98A53.1A78BFE0" alt="signature_210780453"></span></b></a><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;mso-ligatures:none"><o:p></o:p></span></b></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;mso-ligatures:none"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>