<div dir="ltr"><font color="#000000"><span style="font-size:12.8px">Dear, Danny Auble, Developers and Users,</span></font><div><font color="#000000"><span style="font-size:12.8px"><br></span></font></div><div><font color="#000000"><span style="font-size:12.8px">After update SLURM from version 17.02.2 to the 17.11.6 The behavior of the plugin gres has changed.</span></font></div><div><font color="#000000"><span style="font-size:12.8px"><br></span></font></div><div><font color="#000000"><span style="font-size:12.8px">On the version <span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">17.02.2, gres.conf can be:</span></span></font></div><div><span style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><div style="color:rgb(0,0,0)"><span style="font-size:12.8px">Name=gpu Type=K40 File=/dev/nvidia0   COREs=0</span></div><div style="color:rgb(0,0,0)"><span style="font-size:12.8px">Name=gpu Type=K40 File=/dev/nvidia1   COREs=10</span></div><div style="color:rgb(0,0,0)"><span style="font-size:12.8px">Name=gpu Type=cpu                     COREs=2-9,12-19 Count=16</span></div><div style="color:rgb(0,0,0)"><span style="font-size:12.8px">Name=gpu Type=debugcpu                COREs=1,11      Count=2</span></div><div style="color:rgb(0,0,0)"><span style="font-size:12.8px"><br></span></div><div style="color:rgb(0,0,0)"><span style="font-size:12.8px">All GPU jobs starts succesfully on <span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">the slurm-</span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);float:none;display:inline">17.02.2</span>.</span></div><div style="color:rgb(0,0,0)">But now slurm-<span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">17.11.6 does not set variables <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">CUDA_VISIBLE_DEVICES, and all jobs on the same node use only one GPU.  This is due to the generation of errors in the function common_gres_set_env(...) in file s</span></span><span style="color:rgb(34,34,34)">rc/plugins/gres/common/gres_common.c:196</span></div><div style="color:rgb(0,0,0)"><span style="color:rgb(34,34,34)">for this </span></div><div>len = bit_size(bit_alloc); //Equal to 20<br></div><div>list_count(gres_devices) equl to 2<br></div><div><br></div><div>Why is gres.conf not working now? I use this gres.conf to be sure that COREs=0,10 used only with GPU and never for tasks without GPU.</div><div><br></div><div>In general bug can be in</div></span></div><div><div><font color="#000000"><span style="font-size:12.8px">src/plugins/gres/common/gres_common.c</span><br></font></div><div><font color="#000000"><span style="font-size:12.8px">src/common/gres.c</span><br></font></div></div><div><font color="#000000"><span style="font-size:12.8px"><br></span></font></div><div><font color="#000000"><span style="font-size:12.8px">I think I can not fix this problem by myself.</span></font></div><div><font color="#000000"><span style="font-size:12.8px">Who know solution for this problem?</span></font></div><div><font color="#000000"><span style="font-size:12.8px"><br></span></font></div><div><font color="#000000"><span style="font-size:12.8px">Best regards, Vova.</span></font></div></div>