<div dir="ltr">Dear, <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Miguel</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">>Do you have --gres=gpu:0 on your job script? Is gres.conf properly configured?</span></span></div><div><span style="font-size:12.8px"><br></span></div><div><div>No. <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">gres.conf worked weel on the Slurm <span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgba(27,31,35,0.05);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">17.02.2.</span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgba(27,31,35,0.05);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">It is my job file.</span></span></div><div><br></div><div><div><span style="font-size:12.8px">#!/bin/bash</span></div><div><span style="font-size:12.8px">#</span></div><div><span style="font-size:12.8px">#SBATCH --job-name=GPU</span></div><div><span style="font-size:12.8px">#SBATCH --partition=gpu</span></div><div><span style="font-size:12.8px">#</span></div><div><span style="font-size:12.8px">#SBATCH --ntasks=1</span></div><div><span style="font-size:12.8px">#SBATCH --mem=100</span></div><div><span style="font-size:12.8px">#SBATCH --time=10</span></div><div><span style="font-size:12.8px">#SBATCH --gres=gpu:K40:1</span></div><div><span style="font-size:12.8px">srun echo "Job id: $SLURM_JOB_ID"</span><br></div><div><span style="font-size:12.8px">srun echo "GPU id: $SLURM_JOB_GPUS"</span></div><div><span style="font-size:12.8px">srun echo "VisDev: $CUDA_VISIBLE_DEVICES"</span></div><div><span style="font-size:12.8px">srun sleep 120</span></div><div style="font-size:12.8px"><br></div></div><div><div><span style="font-size:12.8px">My gres.conf:</span><br></div><div><div><div># Configure support for our two GPUs (n[01-10])</div><div>Name=gpu Type=K40 File=/dev/nvidia0 CPUs=0</div><div>Name=gpu Type=K40 File=/dev/nvidia1 CPUs=10</div><div>Name=gpu Type=cpu      CPUs=2-9,12-19 Count=16</div><div>Name=gpu Type=debugcpu CPUs=1,11      Count=2</div><div>#CPUs 1 and 11 are reserved for system or for debug.<br></div></div><div><br></div><div><br></div><div>Thank you, for your help.</div><div>Vova.<br></div><div><br></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2018-05-21 17:18 GMT+10:00 Miguel Gila <span dir="ltr"><<a href="mailto:miguel.gila@cscs.ch" target="_blank">miguel.gila@cscs.ch</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">Vova, <div><br></div><div>Do you have --gres=gpu:0 on your job script? Is gres.conf properly configured?</div><div><br></div><div>I think this is what sets the variable: <span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;background-color:rgb(255,255,255)"><a href="https://github.com/SchedMD/slurm/blob/bcdd09d3386f4b4038ae9263b0e69d4d742988b2/src/plugins/gres/gpu/gres_gpu.c#L96" target="_blank">https://github.com/<wbr>SchedMD/slurm/blob/<wbr>bcdd09d3386f4b4038ae9263b0e69d<wbr>4d742988b2/src/plugins/gres/<wbr>gpu/gres_gpu.c#L96</a></span><div><br></div><div>Cheers,</div><div>Miguel</div><div><div class="h5"><div><br></div><div><br><blockquote type="cite"><div>On 21 May 2018, at 08:28, Vladimir Goy <<a href="mailto:vovagoy@gmail.com" target="_blank">vovagoy@gmail.com</a>> wrote:</div><br class="m_2086950861350177787Apple-interchange-newline"><div><div dir="ltr"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Hello,</span><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Please, help me, </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">I would like to ask you about the next bug. Why slurm does not set <span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgba(27,31,35,0.05);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">CUDA_VISIBLE_DEVICES variabels before runs user aplication? This bug I observe after update slurm 17.02.2 -> 17.11.6 . Who/How can fix this problem?</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgba(27,31,35,0.05);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Version 17.02.2 works well, but now I can not downgrate, because slurmdbd update my database.</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Best regards,</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:11.9px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgba(27,31,35,0.05);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Vova.</span></div><br></div>
</div></blockquote></div><br></div></div></div></div></blockquote></div><br></div>