<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Fixed with:<br></div><div dir="ltr"><a href="https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-reference-linux/top/environment-variable-reference/hydra-environment-variables.html">Hydra Environment Variables (intel.com)</a><br></div><div dir="ltr">I_MPI_HYDRA_BOOTSTRAP=ssh<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 16, 2022 at 11:09 AM Joe Teumer <<a href="mailto:joe.teumer@gmail.com">joe.teumer@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"><div>Hello!</div><div><br></div><div>Is there a way to turn off slurm MPI hooks?</div><div>A job submitted via sbatch executes Intel MPI and the thread affinity settings are incorrect.</div><div>However, running MPI manually over SSH works and all bindings are correct.</div><div><br></div><div>We are looking to run our MPI jobs via slurm sbatch and have the same behavior as running the job manually over SSH.</div><div dir="ltr"><br></div><div dir="ltr">slurmd -V</div><div dir="ltr">slurm 22.05.3</div></span></div><div dir="ltr"><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"><br></span></div><div dir="ltr"><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)">RUNNING OMP_NUM_THREADS=, cmd=numactl -C 0-63,128-191 -m 0 mpirun -verbose -genv I_MPI_DEBUG=4 -genv KMP_AFFINITY=verbose,granularity=fine,compact -np 64 -ppn 64 ./mpiprogram -in in.program -log program -pk intel 0 omp 2 -sf intel -screen none -v d 1</span><br></div><div dir="ltr"><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"><br></span></div><div><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)">which mpirun</span></div><div><span style="background-color:rgb(244,245,247)"><font color="#333333" face="-apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen, Ubuntu, Fira Sans, Droid Sans, Helvetica Neue, sans-serif"><span style="font-size:14px">/opt/intel/psxe_runtime_2019.6.324/linux/mpi/intel64/bin/mpirun</span></font><br></span></div><div dir="ltr"><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"><br></span></div><div><font color="#333333" face="-apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen, Ubuntu, Fira Sans, Droid Sans, Helvetica Neue, sans-serif"><span style="font-size:14px;background-color:rgb(244,245,247)">slurm sbatch:</span></font></div><div dir="ltr"><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"><br></span></div><div dir="ltr"><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)">[mpiexec@node]</span><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"> </span><b style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)">Launch arguments: /usr/local/bin/srun -N 1 -n 1 --ntasks-per-node 1 --nodelist node --input none /opt/intel/psxe_runtime_2019.6.324/linux/mpi/intel64/bin//hydra_bstrap_proxy</b><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"> --upstream-host node --upstream-port 45427 --pgid 0 --launcher slurm --launcher-number 1 --base-path /opt/intel/psxe_runtime_2019.6.324/linux/mpi/intel64/bin/ --tree-width 16 --tree-level 1 --time-left -1 --collective-launch 1 --debug /opt/intel/psxe_runtime_2019.6.324/linux/mpi/intel64/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9</span><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"><br></span></div><div dir="ltr"><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"><br></span></div><div><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)">SSH manual run:</span></div><div dir="ltr"><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"><br></span></div><div dir="ltr"><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)">[mpiexec@node]</span><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"> Launch arguments: </span><b style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)">/opt/intel/psxe_runtime_2019.6.324/linux/mpi/intel64/bin//hydra_bstrap_proxy</b><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"> --upstream-host node --upstream-port 35747 --pgid 0 --launcher ssh --launcher-number 0 --base-path /opt/intel/psxe_runtime_2019.6.324/linux/mpi/intel64/bin/ --tree-width 16 --tree-level 1 --time-left -1 --collective-launch 1 --debug --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 7 /opt/intel/psxe_runtime_2019.6.324/linux/mpi/intel64/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9</span><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"><br></span></div></div></div></div></div></div>
</blockquote></div></div></div></div>