<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1024793224;
mso-list-type:hybrid;
mso-list-template-ids:2117789984 -1935888404 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hello –<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m trying to get gpu container jobs working on virtual nodes. The jobs fail with “Test CUDA failure common.cu:893 'invalid device ordinal'” in the output file and “slurmstepd: error: mpi/pmix_v3: _errhandler: n4 [0]: pmixp_client_v2.c:211:
Error handler invoked: status = -25, source = [slurm.pmix.126.0:1]” in the error file. Google points me to issues where others are selecting the wrong GPU or too many GPUs but I’m just trying to get one GPU (per node) working.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Some infos:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">slurm-22.05<o:p></o:p></li><ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">slurm-22.05.5-1.el9.x86_64<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">slurm-contribs-22.05.5-1.el9.x86_64<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">slurm-devel-22.05.5-1.el9.x86_64<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">slurm-libpmi-22.05.5-1.el9.x86_64<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">slurm-pam_slurm-22.05.5-1.el9.x86_64<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">slurm-perlapi-22.05.5-1.el9.x86_64<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">slurm-slurmctld-22.05.5-1.el9.x86_64<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">slurm-example-configs-22.05.5-1.el9.x86_64<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">nvslurm-plugin-pyxis-0.14.0-1.el9.x86_64<o:p></o:p></li></ul>
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Rocky Linux release 9.0 (Blue Onyx)<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">KVM virtualization<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">6-node cluster n0 - n5. n4 and n5 have one Tesla V100-SXM2-16GB each.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Driver Version: 530.30.02<o:p></o:p></li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My attempt at setting this up:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Configure GresTypes=gpu in slurm.conf<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Separate n4 and n5 in slurm.conf to use the GresType<o:p></o:p></li><ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">NodeName=n[4-5] GRES=gpu:1 CPUs=3 State=UNKNOWN<o:p></o:p></li></ul>
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Create /etc/slurm/gres.conf on each gpu node<o:p></o:p></li><ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">Name=gpu File=/dev/nvidia0
<o:p></o:p></li></ul>
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Sync slurm.conf across the cluster and restart slurmd on n[1-5]<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Restart slurmctld on n0<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Resume n4 and n5<o:p></o:p></li><ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">scontrol update nodename=n[4-5] state=resume<o:p></o:p></li></ul>
</ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">References: https://slurm.schedmd.com/gres.html, https://slurm.schedmd.com/gres.conf.html<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This little test script works and gives me gpu info:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#!/bin/sh<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -J gpu_test<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -N 1<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -n 3<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -w n5<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -o %j.o<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -e %j.e<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">nvidia-smi<o:p></o:p></p>
<p class="MsoNormal">nvidia-debugdump -l<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This script fails with the errors I mentioned above:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#!/bin/sh<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -J tfmpi<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -N 2<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -n 6<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -w n[4-5]<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -o %j.o<o:p></o:p></p>
<p class="MsoNormal">#SBATCH -e %j.e<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#SBATCH --gres=gpu:1<o:p></o:p></p>
<p class="MsoNormal">#SBATCH --gpus=1<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">srun --mpi=pmix --container-image=nvcr.io#nvidia/tensorflow:23.02-tf2-py3 all_reduce_perf_mpi -b 1G -e 1G -c 1<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What am I missing to get the second script to run?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you.<o:p></o:p></p>
<p class="MsoNormal">Cornelius Henderson<o:p></o:p></p>
<p class="MsoNormal">Senior Systems Administrator<o:p></o:p></p>
<p class="MsoNormal">NASA Center for Climate Simulation (NCCS)<o:p></o:p></p>
<p class="MsoNormal">ASRC Federal InuTeq, LLC<o:p></o:p></p>
<p class="MsoNormal">Goddard Space Flight Center<o:p></o:p></p>
<p class="MsoNormal">Greenbelt, MD 20771<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>