<html style="direction: ltr;">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css">body p { margin-bottom: 0cm; margin-top: 0pt; } </style>
<style type="text/css">body p { margin-bottom: 0cm; margin-top: 0pt; } </style>
</head>
<body bidimailui-charset-is-forced="true" style="direction: ltr;"
text="#000000" bgcolor="#FFFFFF">
<p>Thank you Artem,</p>
<p><br>
</p>
<p>I've made a mistake while typing the mail, in all cases it was
'OMPI_MCA_pml=ucx' and not as written. When I went over the mail
before sending, I must have erroneously 'fixed' it for some
reason. </p>
<p><br>
</p>
<p>----</p>
<p>Best regards,</p>
<p>--Dani_L.<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 7/9/19 9:06 PM, Artem Polyakov
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DB6PR0501MB2791254A7057631C06FA094DBAF10@DB6PR0501MB2791.eurprd05.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css">body p { margin-bottom: 0cm; margin-top: 0pt; } </style>
<style id="ms-outlook-ios-style" type="text/css">html {
background-color: transparent;
}
body {
color: #333;
line-height: 150%;
font-family: "-apple-system", "HelveticaNeue";
margin: 0;
}
.ms-outlook-ios-reference-expand {
display: block;
color: #999;
padding: 20px 0px;
text-decoration: none;
}
.ms-outlook-ios-availability-container {
max-width: 500px;
margin: auto;
padding: 12px 15px 15px 15px;
border: 1px solid #C7E0F4;
border-radius: 4px;
}
.ms-outlook-ios-availability-container > .ms-outlook-ios-availability-delete-button {
width: 25px;
height: 25px;
right: -12px;
top: -12px;
background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEsAAABLCAYAAAA4TnrqAAAAAXNSR0IArs4c6QAACxpJREFUeAHlnFuMXlUVx/fcOzNtp0ynF0U7hWKrEmKLosZEjUZ9MgZIQBNC0uAtJr745oOJIT74xgskJkQbAlQNJmBMfNDEG0YjEC7GIBQZ6IAI005L79O5+/+dfut0f5dzzt7nu8w37UrWt893zt5rr/U/e6+zL+ucHrcGtLq62q9qd4gnxTeKb6kcc267eEI8Kz4mnhFPi58Rv1g5nunp6VnS8ZVHAqdHPCk+KP6zuBWEHORNinvWNWoYIN4q/o74mLidhHzqob71AxzKij8g/p14LYh6qb97QUM58T7x38TdQOiBPt0FmhQaEf9M3I2EXiNr7tOkBK3pVvGCuJsJ/dCzqVbWWxZxVTygso+InxBz3M2Efuj5SEXvUrqWQloV7lRtT4vfX6rWtS30pqr/uMZp78SqEQ2WgPqQKvmnuKWtaWnFuaWVVbciXl51rk+a9fb2uP6EY80qzL+oHB8RYC8V5vQyRIEloD6tsk965UsdAsyZ+WV35uKyOz+/4uZ0YgmEMqhfyA3397rRoV63eUOf2zzUJxAzMsed/owA+2tokWCwmgVKDcadvLDsZs8vutMXV5zkhepYl08GurENvW5idMCNj/Q5Nb5mKBiwoGoqXe/fZTQCkmNnl9x/Ty+4xZzW0yeLB9WC6Hp0QbLSJRd0sAzSGTSgzO8bG3TbN/W7IGMay/lwSJcslC+gcOZviKN9FC3pjVML7uKi+l0NDakf0Sq2DPe5kYFeh9FZBMgXJOPU3HLSOufpxzW0QTJ2bRlMZNZcCvmLD9slwHKdfraGKi2gAGhKHPXUm1tcdVMn5t05+SWfMGjbaL+7RiABUFkCuHd1I46fX6q7ERvlz/ZsHXLDA7mmNaqap+QeAQZwDSlTooDiGuOouxqWzDjJ3X91dj55slkWWs216io7musqJi5N6Zwz6uJv1XRxnqA3TAwlrTbNHHZwWNnuFmAN+30eWLeqIAO5YHr7zKK63WLqvPFDOzcNuPeODSR+KFhQZEb82/9OL7p3zi6m/k0Gq1sOuPdsjvYet6nsrxup0BAstSrmUqfEQTVxG147seCOn7vcguly+7ZtKNMdGukZdI7uf+T4xaquuW3jgLt+62CM88eILQLsQm2ldY6j0v3uV8YgoBBYC9SYxkI37RzuKFDogZ+iXuo34gaiXwRh9/0VHKqK1bUsZdqnHC9X5cr5Q9ebfveyMnS73eODOSU6c+noyYWkW1ptk9cMxnbJD6p1HbHypFUtq4LmIT9D3jHOHB9l1C1AoQ83DH2M0BN9I+hQbeuqAkuCbhB/KkQg/oGnngQm2Wn63dCifN3Rx7okeqIvegcSOIBHSilYFRQfSK8UHDCOYuIL4cz3ypl3I6EX+kHoi94R9IDfulKwJGBc/KUQQYzMbcDJ8ICnXp8vKURIh/Kg1yX9Lrln9Eb/QAIPcEnIN/FOO5mX0paYwhjhF0qMlq14R1L0q/ZfCy64MzqX4pKAVWlq94ZozqTY5nqMzBlwrgdCT5t/oj92BNK91hWtZe1SwW1FhXFRrB4YMYXJmf9atiRl7vvz52fd4/86GXNXq2TYH1oFch59blZ+yM7mp+iJvkbYkbOYYdlIwQV8HNvo0OcuJfm/9HVbZsFpMtcLpV++MOvuPvyfJPs9n9jufnrnnphRdVoNQH3jsSl36Cl29l0i466b2e0vJvRlSkTLwg7smRi9PIDNkQA+D1nL+nZOxvQSC3dGrB7oZgXTcOWJRAEMxeAIv5HUUwsUJ325SaacH/RFbyPfHjuXkR7kfK/6I03sk/zJI5o7K5xGLLPE0O03jTtalFEsYI2AQt5tkhtDvt7YE9iNPyuckpXsj4VUxnq5CiRZWbiLXY/irtL1ygCWBVSZroze6A9hD3YF0g5KMRcsJDYYjFjhLENlAGslUKazr79vl13PSCeDwWIXxoil4LIUA1g7gEJvX3/frgKbbgSsvQWZkstsVxnFdkErZ2kIYO0CCh18/X27TL+M9BbA2ppxMT0NTravx/TGBndphhIHeYCx8ukPDxDfzHCjVj30xw4Iu7x2UJvV/z/Jc3STf6bRsU2YucZ2VavIAEOejZtIn5w6qxWCubSaVgJlQrFjrjIqxT7W7QsocfCFYPn7dnZHCgQHXzbA/Kdku4FCOd8O374cxXfSDYdzMiSX/GlB8Q0oklZ/HcAevGOPdmSqVeE/5wvveb3IwjO+Hb59OQXHAatuYb62QAnBtSJy/+PMv/WrqaquRwFaGOe53mrCLxoFepZZwDpnhbLSEk02S1TdeXSudeZ+C4sd6ddVkHGC0AAjQgYC6BhgnS3K6Ds/Yg9aRY2Awne9/P39pUb6MXr5dvj25ciYAawTORmSS8wOCPuBcIa28pCcKPmTBRRTGKoqOzUKUQf9zaljV2X2U1R0GrBeKcrFdeKjjIg1aIbygLIOQdouwHz9fbsKbHoGBKr2xrIKEEhmFLmlZMWSNAQoK9AuwHz9fbus3oz0xWCwiLYziljwtyJJGgOUFWwHYL7+RBIGUtINnw3JjFCCLSDio/ymHFK+DFAmt5WAobfFd2GP3wisvox0plcFpnXxtYwM6WlcFqGJRsRHxdATWjO3KQ3lYqcwWYAhN4Z8vbHHc8V5Yv4inJbM+j/l5bRrxHAaEUhGawmlOe+hEAuU1dEIMF+u5ctK0Re9jXx77FxG+hDnqZ8Vw68p+QXHecQ47vm3LqRDh93jQ9qPu7ymnVeWmT2bFqyZs8ScVJxXIOcaRtOiAOqr+ydCW4c2K5bc0ZOXdqRZeThw7Uho8O5ueqCBtVH1E085mqNjcolIu9e9CverwsoQrKjoml5nLP2Cd6Ov040O3J06LsV3CKzVpBvqgClPUJQfUcEWO8Dgjoi79UDoaYNp9MeOQPohQJHXfBbHD/NTRDRFooKN2IeLiEyxYh1N0e9t6WmE/hFu4DEr54P1B50MGs2z4E9UMMS0gdDE5eYG9YmsdvygF/rZxBm9/Q2Lgjp/r+vp4zYFS00Nc39cUDi9TPi0TUDZ4X1FCnUjoZfFZqAvekfQd60LUiYFqyLgUaXTlePchMgUwqclLMl3WvtvhCZ2E6EPekHoib4RET9/V7FXk8KVnyqwJJBByI/8DHnHbCkRPm2E/+oWwGpjStHT3wIznXPSe/xWRb4qsCoFDyl9qnJcmBBnTvi0EYC9NLN2PgwfRf3oYYR+kfHwYFDnvxs+FDRIPaDMfHQiaJbJc7U2vJvH85UWB98QLNnOqP4+Jd/jOJTW+g0Lhgf21MNHdeQNC8ARWAymcHIf5X8osVZ01b27AzgC7Holz4nH+B9KDAKvqrfCDBgB9hUdPy4O8l9WjpRFtqvmfUMzXIB9U8cP2v+YFOcf8yYr227sTLHCwexgXb3JasAIsB/oOHgMZuUsxXha2hX/jrQZ3CxgJoe1LSLuCCSLfvteczuWuANXOK3KrDT4ZXIEZA4dsqRXuuRPdD3ah2XJ5DwAEs1C16MV0hXpksznWgSMXz0j1vZ+18FqE2A4/YfFUU9JK7/G6Zuqv9QXQxpNdwpt0YDvN8p0szhoZ6hQYOcyHFZVvDSe+5Z9W9RRCxsU3ydeEnczteQrRy0BUSgdEP+jS9Hqju9n+UgLKL6l9XXx0S4BrTu/zFYDWr/AOig+skagdf83/3zAOBZQvOryRTEf+Donbid15GuS0eOsWlBC/gsl9iW/LP6C+PPi68TN0usS8EcxH6z4be2qZrPCG5XvCFi1FQu8SZ1j6YdXYeC9YuLxiZyGicQltpuoRPiEmJVLwqPgZwXOtNKO0v8BzRAPSFNM7HEAAAAASUVORK5CYII=");
background-size: 25px 25px;
background-position: center;
}
#ms-outlook-ios-main-container {
margin: 0 0 0 0;
margin-top: 120;
padding: 8;
}
#ms-outlook-ios-content-container {
padding: 0;
padding-top: 12;
padding-bottom: 20;
}
.ms-outlook-ios-mention {
color: #333;
background-color: #f1f1f1;
border-radius: 4px;
padding: 0 2px 0 2px;
pointer-events: none;
text-decoration: none;
}
.ms-outlook-ios-mention-external {
color: #ba8f0d;
background-color: #fdf7e7;
}
.ms-outlook-ios-mention-external-clear-design {
color: #ba8f0d;
background-color: #f1f1f1;
}</style>
<meta name="viewport" content="width=device-width,
user-scalable=no, initial-scale=1.0, minimum-scale=1.0,
maximum-scale=1.0">
<div style="direction: ltr;">
<div>
<div style="direction: ltr;">Hello, Daniel </div>
<div><br>
</div>
<div style="direction: ltr;">Let me try to reproduce locally
and get back to you.</div>
</div>
<div><br>
</div>
<div class="ms-outlook-ios-signature">
<div style="direction: ltr;">----</div>
<div style="direction: ltr;">Best regards, </div>
<div style="direction: ltr;">Artem Y. Polyakov, PhD</div>
<div style="direction: ltr;">Senior Architect, SW</div>
<div style="direction: ltr;">Mellanox Technologies</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>От:</b> <a
class="moz-txt-link-abbreviated"
href="mailto:pmix@googlegroups.com" moz-do-not-send="true">pmix@googlegroups.com</a>
<a class="moz-txt-link-rfc2396E"
href="mailto:pmix@googlegroups.com" moz-do-not-send="true"><pmix@googlegroups.com></a>
от имени Daniel Letai <a class="moz-txt-link-rfc2396E"
href="mailto:dani@letai.org.il" moz-do-not-send="true"><dani@letai.org.il></a><br>
<b>Отправлено:</b> Tuesday, July 9, 2019 3:25:22 AM<br>
<b>Кому:</b> Slurm User Community List; <a
class="moz-txt-link-abbreviated"
href="mailto:pmix@googlegroups.com" moz-do-not-send="true">pmix@googlegroups.com</a>;
<a class="moz-txt-link-abbreviated"
href="mailto:ucx-group@elist.ornl.gov"
moz-do-not-send="true">ucx-group@elist.ornl.gov</a><br>
<b>Тема:</b> [pmix] [Cross post - Slurm, PMIx, UCX] Using srun
with SLURM_PMIX_DIRECT_CONN_UCX=true fails with input/output
error</font>
<div> </div>
</div>
<div>
<p>Cross posting to Slurm, PMIx and UCX lists.<br>
</p>
<p><br>
</p>
<p>Trying to execute a simple openmpi (4.0.1) mpi-hello-world
via Slurm (19.05.0) compiled with both PMIx (3.1.2) and UCX
(1.5.0) results in:</p>
<p><br>
</p>
<p>[root@n1 ~]# SLURM_PMIX_DIRECT_CONN_UCX=true
SLURM_PMIX_DIRECT_CONN=true OMPI_MCA_pml=true
OMPI_MCA_btl='^vader,tcp,openib' UCX_NET_DEVICES='mlx4_0:1'
SLURM_PMIX_DIRECT_CONN_EARLY=false UCX_TLS=rc,shm srun
--export
SLURM_PMIX_DIRECT_CONN_UCX,SLURM_PMIX_DIRECT_CONN,OMPI_MCA_pml,OMPI_MCA_btl,
UCX_NET_DEVICES,SLURM_PMIX_DIRECT_CONN_EARLY,UCX_TLS
--mpi=pmix -N 2 -n 2 /data/mpihello/mpihello<br>
</p>
<p><br>
</p>
<p>slurmstepd: error: n1 [0] pmixp_dconn_ucx.c:668
[_ucx_connect] mpi/pmix: ERROR: ucp_ep_create failed:
Input/output error<br>
slurmstepd: error: n1 [0] pmixp_dconn.h:243
[pmixp_dconn_connect] mpi/pmix: ERROR: Cannot establish direct
connection to n2 (1)<br>
slurmstepd: error: n1 [0] pmixp_server.c:731
[_process_extended_hdr] mpi/pmix: ERROR: Unable to connect to
1<br>
srun: Job step aborted: Waiting up to 32 seconds for job step
to finish.<br>
slurmstepd: error: n2 [1] pmixp_dconn_ucx.c:668 [_ucx_connect]
mpi/pmix: ERROR: ucp_ep_create failed: Input/output error<br>
slurmstepd: error: n2 [1] pmixp_dconn.h:243
[pmixp_dconn_connect] mpi/pmix: ERROR: Cannot establish direct
connection to n1 (0)<br>
slurmstepd: error: *** STEP 7202.0 ON n1 CANCELLED AT
2019-07-01T13:20:36 ***<br>
slurmstepd: error: n2 [1] pmixp_server.c:731
[_process_extended_hdr] mpi/pmix: ERROR: Unable to connect to
0<br>
srun: error: n2: task 1: Killed<br>
srun: error: n1: task 0: Killed</p>
<p><br>
</p>
<p>However, the following works:</p>
<p><br>
</p>
<p>[root@n1 ~]# SLURM_PMIX_DIRECT_CONN_UCX=false
SLURM_PMIX_DIRECT_CONN=true OMPI_MCA_pml=true
OMPI_MCA_btl='^vader,tcp,openib' UCX_NET_DEVICES='mlx4_0:1'
SLURM_PMIX_DIRECT_CONN_EARLY=false UCX_TLS=rc,shm srun
--export
SLURM_PMIX_DIRECT_CONN_UCX,SLURM_PMIX_DIRECT_CONN,OMPI_MCA_pml,OMPI_MCA_btl,
UCX_NET_DEVICES,SLURM_PMIX_DIRECT_CONN_EARLY,UCX_TLS
--mpi=pmix -N 2 -n 2 /data/mpihello/mpihello</p>
<p><br>
n2: Process 1 out of 2<br>
n1: Process 0 out of 2</p>
<p><br>
</p>
<p>[root@n1 ~]# SLURM_PMIX_DIRECT_CONN_UCX=false
SLURM_PMIX_DIRECT_CONN=true OMPI_MCA_pml=true
OMPI_MCA_btl='^vader,tcp,openib' UCX_NET_DEVICES='mlx4_0:1'
SLURM_PMIX_DIRECT_CONN_EARLY=true UCX_TLS=rc,shm srun --export
SLURM_PMIX_DIRECT_CONN_UCX,SLURM_PMIX_DIRECT_CONN,OMPI_MCA_pml,OMPI_MCA_btl,
UCX_NET_DEVICES,SLURM_PMIX_DIRECT_CONN_EARLY,UCX_TLS
--mpi=pmix -N 2 -n 2 /data/mpihello/mpihello</p>
<p><br>
</p>
<p>n2: Process 1 out of 2<br>
n1: Process 0 out of 2</p>
<p><br>
</p>
<p>Executing mpirun directly (same env vars, without the slurm
vars) works, so UCX appears to function correctly.<br>
</p>
<p><br>
</p>
<p>If both SLURM_PMIX_DIRECT_CONN_EARLY=true and
SLURM_PMIX_DIRECT_CONN_UCX=true then I get collective timeout
errors from mellanox/hcoll and glibc detected
/data/mpihello/mpihello: malloc(): memory corruption (fast) <br>
</p>
<p><br>
</p>
<p>Can anyone help using PMIx direct connection with UCX in
Slurm?<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p>Some info about my setup:</p>
<p><br>
</p>
<p>UCX version<br>
</p>
<p>[root@n1 ~]# ucx_info -v</p>
<p># UCT version=1.5.0 revision 02078b9<br>
# configured with: --build=x86_64-redhat-linux-gnu
--host=x86_64-redhat-linux-gnu
--target=x86_64-redhat-linux-gnu --program-prefix=
--prefix=/usr --exec-prefix=/usr --bindir=/usr/bin
--sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share
--includedir=/usr/include --libdir=/usr/lib64
--libexecdir=/usr/libexec --localstatedir=/var
--sharedstatedir=/var/lib --mandir=/usr/share/man
--infodir=/usr/share/info --disable-optimizations
--disable-logging --disable-debug --disable-assertions
--enable-mt --disable-params-check</p>
<p><br>
</p>
<p>Mellanox OFED version:<br>
</p>
<p>[root@n1 ~]# ofed_info -s<br>
OFED-internal-4.5-1.0.1:</p>
<p><br>
</p>
<p>Slurm:<br>
</p>
<p>slurm was built with:<br>
rpmbuild -ta slurm-19.05.0.tar.bz2 --without debug --with ucx
--define '_with_pmix --with-pmix=/usr'</p>
<p><br>
</p>
<p>PMIx:<br>
</p>
<p>[root@n1 ~]# pmix_info -c --parsable<br>
config:user:root<br>
config:timestamp:"Mon Mar 25 09:51:04 IST 2019"<br>
config:host:slurm-test<br>
config:cli: '--host=x86_64-redhat-linux-gnu'
'--build=x86_64-redhat-linux-gnu' '--program-prefix='
'--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin'
'--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--datadir=/usr/share' '--includedir=/usr/include'
'--libdir=/usr/lib64' '--libexecdir=/usr/libexec'
'--localstatedir=/var' '--sharedstatedir=/var/lib'
'--mandir=/usr/share/man' '--infodir=/usr/share/info'</p>
<p><br>
</p>
<p>Thanks,</p>
<p>--Dani_L.<br>
</p>
-- <br>
You received this message because you are subscribed to the
Google Groups "pmix" group.<br>
To unsubscribe from this group and stop receiving emails from
it, send an email to <a
href="mailto:pmix+unsubscribe@googlegroups.com"
moz-do-not-send="true">pmix+unsubscribe@googlegroups.com</a>.<br>
To post to this group, send email to <a
href="mailto:pmix@googlegroups.com" moz-do-not-send="true">pmix@googlegroups.com</a>.<br>
Visit this group at <a
href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fgroup%2Fpmix&data=02%7C01%7Cartemp%40mellanox.com%7C5557609f38814da446a008d70457c245%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636982647290691590&sdata=IamJX3FfwfLzukF675l0A8IY%2Fb4eDX4iXZItVoy18fs%3D&reserved=0"
originalsrc="https://groups.google.com/group/pmix"
shash="Hfx1WZu52NGgTkO40eWZYMz95ZUvEdyXv6jen96M1eMwWdZh/g7LuGi6CNam3VMUveqyMX7bKX3cv3VG7cCPlWOF5Bj4QMDwCxNJvJ+xZj/nuXaEa+V3+Gm1lCAgymkLLI2iinrig/gZ7OCRzgwfUovVjRYx0zGm6GZa62Kz9eE="
moz-do-not-send="true"> https://groups.google.com/group/pmix</a>.<br>
To view this discussion on the web visit <a
href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fpmix%2Fce4a81a4-b3f7-48ce-4b9c-a5ebb098862c%2540letai.org.il%3Futm_medium%3Demail%26utm_source%3Dfooter&data=02%7C01%7Cartemp%40mellanox.com%7C5557609f38814da446a008d70457c245%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636982647290701603&sdata=BMGs7VW4TcdZkk21E1tKOFpICusHyYQD1JDWb0bU9qg%3D&reserved=0"
originalsrc="https://groups.google.com/d/msgid/pmix/ce4a81a4-b3f7-48ce-4b9c-a5ebb098862c%40letai.org.il?utm_medium=email&utm_source=footer"
shash="MiyUOVPww7dCwfSHGOLXv8mP4oWrWhWFMdTfbkBRAa1YjDmKuUZk4IZFgu+eb0ph+Um3EVgihwnV7/barg+1+MWWyUsqHMvoIX/L46M3F7JK989DRIXqUhkd9pSbabqioelvlM9AttubDRp9k8oUgjpAHVbjBzwAGKh4iPgMpYY="
moz-do-not-send="true">
https://groups.google.com/d/msgid/pmix/ce4a81a4-b3f7-48ce-4b9c-a5ebb098862c%40letai.org.il</a>.<br>
For more options, visit <a
href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Foptout&data=02%7C01%7Cartemp%40mellanox.com%7C5557609f38814da446a008d70457c245%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636982647290711607&sdata=TVCaY9YtmtVbZWE350bB4wWAorzKTYF1jm5sT%2F%2FhRBQ%3D&reserved=0"
originalsrc="https://groups.google.com/d/optout"
shash="wa3BrsGJXBK3ME+DM8FXWZ7HHPh6I7RmBYezN1Q96ZzA4zPWfiqct2hfr2hgCMsUtkSsc+k4LiZ4D+eE3s171BHsUVnOeLZgHB5IsQaGv4R2JfZvRba1a1/JoMy8HPKSnFWBWzdkUS7/3/6t7ryN+WZ5zcxXH/1eTGtRDfbprB0="
moz-do-not-send="true"> https://groups.google.com/d/optout</a>.<br>
</div>
-- <br>
You received this message because you are subscribed to the Google
Groups "pmix" group.<br>
To unsubscribe from this group and stop receiving emails from it,
send an email to <a
href="mailto:pmix+unsubscribe@googlegroups.com"
moz-do-not-send="true">pmix+unsubscribe@googlegroups.com</a>.<br>
To post to this group, send email to <a
href="mailto:pmix@googlegroups.com" moz-do-not-send="true">pmix@googlegroups.com</a>.<br>
Visit this group at <a
href="https://groups.google.com/group/pmix"
moz-do-not-send="true">https://groups.google.com/group/pmix</a>.<br>
To view this discussion on the web visit <a
href="https://groups.google.com/d/msgid/pmix/DB6PR0501MB2791254A7057631C06FA094DBAF10%40DB6PR0501MB2791.eurprd05.prod.outlook.com?utm_medium=email&utm_source=footer"
moz-do-not-send="true">https://groups.google.com/d/msgid/pmix/DB6PR0501MB2791254A7057631C06FA094DBAF10%40DB6PR0501MB2791.eurprd05.prod.outlook.com</a>.<br>
For more options, visit <a
href="https://groups.google.com/d/optout" moz-do-not-send="true">https://groups.google.com/d/optout</a>.<br>
</blockquote>
</body>
</html>