<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="color: rgb(0, 0, 0);"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Hi Avery -</span></div>
<div style="color: rgb(0, 0, 0);"><font face="Calibri, Arial, Helvetica, sans-serif" size="3"><br>
</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Calibri, Arial, Helvetica, sans-serif" size="3">No worries,</font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I took all of the defaults for software, etc.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Jetson's are locked into ubuntu 18.04.5,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
slurm 19.05.7, etc.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Do you have code to test? I'm more of a python person.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Chris</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Christopher J. Cawley</i></span></div>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; orphans:2; widows:2; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Systems Engineer/Linux Engineer, Information Technology Services</i></span></p>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; orphans:2; widows:2; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">223 Aquia Building, Ffx,</i></span><i><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"> </span><acronym title="Mail Stop Number"><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">MSN</span></acronym></i><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">:
1B5</i></span></p>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; orphans:2; widows:2; background-color:rgb(255,255,255)">
<i style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt; font-variant-caps:inherit; line-height:normal">George Mason University</i></p>
<div><font color="#1f497d" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px"><i><br>
</i></span></font>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; orphans:2; widows:2; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Phone:</i></span><i><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"> </span><span style="margin-right:10px; padding-right:10px; border-right-width:1px; border-right-style:solid; border-right-color:rgb(204,204,204); font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">(703)
993-6397</span></i></p>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; orphans:2; widows:2; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Email:</i></span><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal"> </i></span><i><a style="color:rgb(0,102,51)"><span style="background-color:rgb(255,247,217); padding:3px 0px; font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">ccawley2</span><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">@gmu.edu</span></a></i></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:calibri,helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0); background-color:rgba(0,0,0,0)"></span><br>
</div>
</div>
</div>
</div>
</div>
<div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Avery Grieve <agrieve@umich.edu><br>
<b>Sent:</b> Thursday, December 10, 2020 11:38 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] slurm-wlm package OpenMPI PMIx implementation</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Oop, sorry I meant to also include the following:</div>
<div><br>
</div>
<div><font face="monospace"># srun --mpi=list</font></div>
<div><font face="monospace">srun: MPI types are...</font></div>
<div><font face="monospace">srun: none</font></div>
<div><font face="monospace">srun: pmi2</font></div>
<div><font face="monospace">srun: openmpi</font><br>
</div>
<div><br>
</div>
<div>running srun with --mpi=openmpi gives the same errors as with MpiDefault=none.<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">~Avery Grieve</div>
<div>They/Them/Theirs please!<br>
</div>
<div dir="ltr">
<div>University of Michigan</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Thu, Dec 10, 2020 at 11:34 AM Avery Grieve <<a href="mailto:agrieve@umich.edu">agrieve@umich.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div>Hi Chris,</div>
<div><br>
</div>
<div>Thank you for the offer. Here's some quick information on my system:</div>
<div><br>
</div>
<div>All nodes on Debian 10 (armbian buster converted to DietPi v6.33.3).</div>
<div>sinfo --version: slurm-wlm 18.08.5-2</div>
<div><br>
</div>
<div>With MpiDefault=pmix I get the following srun errors:</div>
<div><span style="font-family:monospace">srun: error: Couldn't find the specified plugin name for mpi/pmix looking at all files<br>
srun: error: cannot find mpi plugin for mpi/pmix<br>
srun: error: cannot create mpi context for mpi/pmix<br>
srun: error: invalid MPI type 'pmix', --mpi=list for acceptable types</span></div>
<div><br>
<span style="font-family:monospace"></span></div>
<div>With MpiDefault=none<br>
</div>
<div>I get OpenMPI yelling at me and giving me two options, only one relevant to the version of Slurm I'm running:</div>
<div><span style="font-family:monospace">version 16.05 or later: you can use SLURM's PMIx support. This<br>
requires that you configure and build SLURM --with-pmix.<br>
<br>
</span></div>
<div>However, as I stated, I'm using the slurm-wlm package which seems to not include the pmix functionality by default.
<br>
</div>
<div><br>
</div>
<div>The other option provided:</div>
<div><span style="font-family:monospace">Versions earlier than 16.05: you must use either SLURM's PMI-1 or<br>
PMI-2 support. SLURM builds PMI-1 by default, or you can manually<br>
install PMI-2. You must then build Open MPI using --with-pmi pointing<br>
to the SLURM PMI library location.</span><br>
</div>
<div><br>
</div>
<div>Similar issue, not building slurm from source doesn't include the PMI library. I've installed some develop level packages, including the<a href="https://secure-web.cisco.com/10RRF9yf9i-7-zx463otbTEYwHQ8PfAu0MWZDt-sp7LOW0FINJnwC3j6h5vCPjyEOcW_d4xkUjeu3yTU6e1aZhW7bYufUDkpZVQ_-IyrDmd5Zl9_4fUvMFvlohRkxN3_18FcN6caXv3CyLnSbizh35atzHBeuC4aREJBiZNIfqjJfrO5LI5pK6coWiyFVqweLpTWOQUWbdjzpnAG7whBZiDXOyp20-eAgY5v3ChNlgtjLNr_9d-OV97ux4pQyrfVyOZ2yTcVNF-THuyFtghkAO6wbbfRVQ9bRhzVsADarrzAyye3nTdVUQeehuf2hANuF7JLIXPuBmV-cAwMpZfuoC6PYr_FI0PoBsPUDaBCrFbagrJzfPr3ufkZsVxor3fyBaY8Vcphc-CfnpwfraZSpFHI8BC4Xq0l475bH5OSBCoQAFV8aE_hF-rycPo-gDRkmSqqHnKTUMuyHb5G46NQ7NQ/https%3A%2F%2Fpackages.debian.org%2Fbuster%2Flibpmi2-0" target="_blank">
libpmi2-0 package</a> which didn't seem to actually install anything useful as far as I can tell using the "find" command.
<br>
</div>
<div><br>
</div>
<div>It's sort of looking like I should be looking at building slurm from source again, I guess.</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">~Avery Grieve</div>
<div>They/Them/Theirs please!<br>
</div>
<div dir="ltr">
<div>University of Michigan</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Thu, Dec 10, 2020 at 11:16 AM Christopher J Cawley <<a href="mailto:ccawley2@gmu.edu" target="_blank">ccawley2@gmu.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I have a 7 node jetson nano cluster running at home.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Send me what you want me to take a look at . If it's not </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
a big deal, then I can let you know.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Ubuntu 18 / slurm <some version from rpm></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Chris</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_gmail-m_4536644827806804856gmail-m_3493327182582563164Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Christopher J. Cawley</i></span></div>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Systems Engineer/Linux Engineer, Information Technology Services</i></span></p>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">223 Aquia Building, Ffx,</i></span><i><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"> </span><acronym title="Mail Stop Number"><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">MSN</span></acronym></i><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">:
1B5</i></span></p>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; background-color:rgb(255,255,255)">
<i style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt; font-variant-caps:inherit; line-height:normal">George Mason University</i></p>
<div><font face="Calibri, sans-serif" color="#1f497d"><span style="font-size:14.6667px"><i><br>
</i></span></font>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Phone:</i></span><i><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"> </span><span style="margin-right:10px; padding-right:10px; border-right:1px solid rgb(204,204,204); font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">(703)
993-6397</span></i></p>
<p style="text-align:start; margin:0px 0px 5px; color:rgb(51,51,51); font-family:Helvetica,Arial,sans-serif; font-size:12.8px; background-color:rgb(255,255,255)">
<span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal">Email:</i></span><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)"><i style="line-height:normal"> </i></span><i><a style="color:rgb(0,102,51)"><span style="background-color:rgb(255,247,217); padding:3px 0px; font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">ccawley2</span><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0)">@gmu.edu</span></a></i></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:calibri,helvetica,sans-serif; font-size:12pt; line-height:normal; color:rgb(0,0,0); background-color:rgba(0,0,0,0)"></span><br>
</div>
</div>
</div>
</div>
</div>
<div>
<div id="x_gmail-m_4536644827806804856gmail-m_3493327182582563164appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_4536644827806804856gmail-m_3493327182582563164divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Avery Grieve <<a href="mailto:agrieve@umich.edu" target="_blank">agrieve@umich.edu</a>><br>
<b>Sent:</b> Thursday, December 10, 2020 10:51 AM<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a> <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> [slurm-users] slurm-wlm package OpenMPI PMIx implementation</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Hi Forum,</div>
<div><br>
</div>
<div>I've been putting together an ARM cluster for fun/learning and I've been a bit lost about how to get OpenMPI and slurm to behave together.
<br>
</div>
<div><br>
</div>
<div>I have installed the <a href="https://secure-web.cisco.com/14EwNb3UZYABzVqRN7IxszUw4L04o_2Bv7wm3a5vivtuqZhDuY3UrhulGE47J31qdoC16rhtMefWeyLXhK10TMim7oOCehTuBJR_47pTBDKcO_xYDX3yqOG1yzamsO31hXo3HS9tSUpOssM40vTLwy4Mxfggu2Qu_yXjJqtLE43mV2CrECvinY7hMt_cRMzi4b8xrKZXqngR31DMmyA9DzimeyLsN7nwxh6kJRMhcg2MjHlCOhu356VVZrErEM9ZafOD66sDUMluigARg1icclZaJOLhXE-7PlFRtAdk2dhXLEvRqSL3SUKrVeBy01MCmSi7sH8bkIijrujncTBU-DfWxY_JOwqhhsJAyXl0XJgjoOiGWHKcLPRRvrCbn_SGHGSw2Ogq3aC4sJLY1tBLwpgvXcOxFoURgb6y6WfJJg04H9ewyQ-Azr7kA_en7DIk_4KOux310uOWzo7XrHTxnLg/https%3A%2F%2Fpackages.debian.org%2Fbuster%2Fslurm-wlm" target="_blank">
slurm-wlm package </a>from the Debian apt search and compiled OpenMPI from source on my compute nodes. OpenMPI has been compiled with the option --with-slurm and the configure time log indicates openmpi has pmix v3 built in. I thought that would be enough for
slurm and calling a job with "srun -n 4 -N1 executable" (with slurm.conf having MpiDefault=pmix_v3) would be enough.
<br>
</div>
<div><br>
</div>
<div>Not the case, unfortunately as slurm doesn't have any idea what pmix_v3 means without being compiled against it I guess. I have also attempted to compile openmpi from source with the --with-pmi option but the slurm-wlm package doesn't install any of the
libraries/headers (pmi.h pmi2.h pmix.h etc). Neither does any of the slurm-llnl develop packages, so I'm at a loss of what to do here.
<br>
</div>
<div><br>
</div>
<div>A few notes: OpenMPI is working across my compute nodes. I'm able to ssh to my compute node and start a job manually with mpirun that executes successfully across the nodes. My slurmctld and slurmd daemons work for single thread resource allocation (and
presumably OpenMP multithreading, though I haven't tested this).</div>
<div><br>
</div>
<div>Beyond compiling slurm from source (assuming this installs the pmi headers that I can use to build openmpi), which I have tried with no luck on my devices, is there a way to get slurm and openmpi to behave together using the precompiled package slurm-wlm?</div>
<div><br>
</div>
<div>Thank you,<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">~Avery Grieve</div>
<div>They/Them/Theirs please!<br>
</div>
<div dir="ltr">
<div>University of Michigan</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>