<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I have a 7 node jetson nano cluster running at home.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Send me what you want me to take a look at . If it's not </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
a big deal, then I can let you know.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Ubuntu 18 / slurm <some version from rpm></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Chris</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Christopher J. Cawley</i></span></div>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; orphans:2; widows:2; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Systems Engineer/Linux Engineer, Information Technology Services</i></span></p>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; orphans:2; widows:2; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">223 Aquia Building, Ffx,</i></span><i><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"> </span><acronym title="Mail Stop Number"><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">MSN</span></acronym></i><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">:
1B5</i></span></p>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; orphans:2; widows:2; background-color:rgb(255,255,255)">
<i style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt; font-variant-caps:inherit; line-height:normal">George Mason University</i></p>
<div><font color="#1f497d" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px"><i><br>
</i></span></font>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; orphans:2; widows:2; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Phone:</i></span><i><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"> </span><span style="margin-right:10px; padding-right:10px; border-right-width:1px; border-right-style:solid; border-right-color:rgb(204,204,204); font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">(703)
993-6397</span></i></p>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; orphans:2; widows:2; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Email:</i></span><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal"> </i></span><i><a style="color:rgb(0,102,51)"><span style="background-color:rgb(255,247,217); padding:3px 0px; font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">ccawley2</span><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">@gmu.edu</span></a></i></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:calibri,helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0); background-color:rgba(0,0,0,0)"></span><br>
</div>
</div>
</div>
</div>
</div>
<div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Avery Grieve <agrieve@umich.edu><br>
<b>Sent:</b> Thursday, December 10, 2020 10:51 AM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [slurm-users] slurm-wlm package OpenMPI PMIx implementation</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Hi Forum,</div>
<div><br>
</div>
<div>I've been putting together an ARM cluster for fun/learning and I've been a bit lost about how to get OpenMPI and slurm to behave together.
<br>
</div>
<div><br>
</div>
<div>I have installed the <a href="https://secure-web.cisco.com/14EwNb3UZYABzVqRN7IxszUw4L04o_2Bv7wm3a5vivtuqZhDuY3UrhulGE47J31qdoC16rhtMefWeyLXhK10TMim7oOCehTuBJR_47pTBDKcO_xYDX3yqOG1yzamsO31hXo3HS9tSUpOssM40vTLwy4Mxfggu2Qu_yXjJqtLE43mV2CrECvinY7hMt_cRMzi4b8xrKZXqngR31DMmyA9DzimeyLsN7nwxh6kJRMhcg2MjHlCOhu356VVZrErEM9ZafOD66sDUMluigARg1icclZaJOLhXE-7PlFRtAdk2dhXLEvRqSL3SUKrVeBy01MCmSi7sH8bkIijrujncTBU-DfWxY_JOwqhhsJAyXl0XJgjoOiGWHKcLPRRvrCbn_SGHGSw2Ogq3aC4sJLY1tBLwpgvXcOxFoURgb6y6WfJJg04H9ewyQ-Azr7kA_en7DIk_4KOux310uOWzo7XrHTxnLg/https%3A%2F%2Fpackages.debian.org%2Fbuster%2Fslurm-wlm">
slurm-wlm package </a>from the Debian apt search and compiled OpenMPI from source on my compute nodes. OpenMPI has been compiled with the option --with-slurm and the configure time log indicates openmpi has pmix v3 built in. I thought that would be enough for
slurm and calling a job with "srun -n 4 -N1 executable" (with slurm.conf having MpiDefault=pmix_v3) would be enough.
<br>
</div>
<div><br>
</div>
<div>Not the case, unfortunately as slurm doesn't have any idea what pmix_v3 means without being compiled against it I guess. I have also attempted to compile openmpi from source with the --with-pmi option but the slurm-wlm package doesn't install any of the
libraries/headers (pmi.h pmi2.h pmix.h etc). Neither does any of the slurm-llnl develop packages, so I'm at a loss of what to do here.
<br>
</div>
<div><br>
</div>
<div>A few notes: OpenMPI is working across my compute nodes. I'm able to ssh to my compute node and start a job manually with mpirun that executes successfully across the nodes. My slurmctld and slurmd daemons work for single thread resource allocation (and
presumably OpenMP multithreading, though I haven't tested this).</div>
<div><br>
</div>
<div>Beyond compiling slurm from source (assuming this installs the pmi headers that I can use to build openmpi), which I have tried with no luck on my devices, is there a way to get slurm and openmpi to behave together using the precompiled package slurm-wlm?</div>
<div><br>
</div>
<div>Thank you,<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">~Avery Grieve</div>
<div>They/Them/Theirs please!<br>
</div>
<div dir="ltr">
<div>University of Michigan</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>