<div dir="ltr"><div>Yup, I think if you get stuck so badly, the first thing is to make sure the node does not get the number 10 from the controller, and the second just reimage the VM fresh. It maybe not the quickest way, but at least predictable in the sense of time spent.</div><div><br></div><div>Good luck!</div><div><br></div><div> -kkm<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 11, 2020 at 7:28 AM mike tie <<a href="mailto:mtie@carleton.edu">mtie@carleton.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div>Yep, slurmd -C is obviously getting the data from somewhere, either a local file or from the master node. hence my email to the group; I was hoping that someone would just say: "yeah, modify file xxxx". But oh well. I'll start playing with strace and gdb later this week; looking through the source might also be helpful. <div><br></div><div>I'm not cloning existing virtual machines with slurm. I have access to a vmware system that from time to time isn't running at full capacity; usage is stable for blocks of a month or two at a time, so my thought/plan was to spin up a slurm compute node on it, and resize it appropriately every few months (why not put it to work). I started with 10 cores, and it looks like I can up it to 16 cores for a while, and that's when I ran into the problem.<br clear="all"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br></div><div dir="ltr">-mike<br><br><table style="font-size:12.8px;border-collapse:collapse" cellspacing="0" cellpadding="0" border="0"><tbody><tr style="height:78.55pt"><td style="width:81.9pt;border-style:none solid none none;border-right:1pt solid windowtext;padding:0in 5.4pt;height:78.55pt" width="82" valign="top"><p><img src="https://apps.carleton.edu/reason_package/reason_4.0/www/images/1301326.png" width="96" height="96"><br></p></td><td style="width:269.85pt;border:medium none;padding:0in 5.4pt;height:78.55pt" width="270" valign="top"><p style="text-align:center" align="center"><b style="font-size:12.8px"><span style="font-size:10pt;font-family:Times">Michael Tie <br></span></b><span style="font-family:Times;font-size:10pt">Technical Director <br></span><span style="font-family:Times;font-size:10pt">Mathematics, Statistics, and Computer Science </span></p><p><span style="font-size:10pt;font-family:Times"> One North College Street phn: 507-222-4067<br></span><span style="font-family:Times;font-size:10pt"> Northfield, MN 55057 cel: 952-212-8933<br></span> <a href="mailto:mtie@carleton.edu" style="font-family:Times;font-size:10pt" target="_blank">mtie@carleton.edu</a><span style="font-family:Times;font-size:10pt"> fax: 507-222-4312</span></p></td></tr></tbody></table></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 11, 2020 at 1:15 AM Kirill 'kkm' Katsnelson <<a href="mailto:kkm@pobox.com" target="_blank">kkm@pobox.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">On Tue, Mar 10, 2020 at 1:41 PM mike tie <<a href="mailto:mtie@carleton.edu" target="_blank">mtie@carleton.edu</a>> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Here is the output of lstopo</div></blockquote><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(68,37,213)"><b>$</b></span><span style="font-variant-ligatures:no-common-ligatures"> lstopo -p</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">Machine (63GB)</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>Package P#0 + L3 (16MB)</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#0</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#1</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#2</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#3</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>Package P#1 + L3 (16MB)</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#4</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#5</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#6</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#7</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>Package P#2 + L3 (16MB)</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#8</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#9</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#10</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#11</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>Package P#3 + L3 (16MB)</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#12</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#13</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#14</span></span></p><span style="background-color:rgb(255,255,255)">
</span><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#15</span></span></p></div></div></blockquote><div><br></div><div>
<div>There is no sane way to derive the number 10 from this topology.
obviously: it has a prime factor of 5, but everything in the lstopo output is sized in powers of 2 (4 packages, a.k.a. sockets, 4 single-threaded CPU cores per). <br></div><div><br></div><div>I responded yesterday but somehow managed to plop my signature into the middle of it, so maybe you have missed inline replies?</div><div><br></div><div>It's
very, very likely
that the number is stored *somewhere*. First to eliminate is the hypothesis that the number is acquired from the control daemon. That's the simplest step and the largest landgrab in the divide-and-conquer analysis plan. Then just look where it comes from on the VM. strace(1) will reveal all files slurmd reads. <br></div><div><br></div><div>You are not rolling out the VMs from an image, ain't you? I'm wondering why do you need to tweak an existing VM that is already in a weird state. Is simply setting its snapshot aside and creating a new one from an image hard/impossible? I did not touch VMWare for more than 10 years, so I may be a bit naive; in the platform I'm working now (GCE), create-use-drop pattern of VM use is much more common and simpler than create and maintain it to either *ad infinitum* or *ad nauseam*, whichever will have been reached the earliest. But I don't know anything about VMWare; maybe it's not possible or feasible with it.</div><div><br></div><div> -kkm</div><br></div></div></div>
</blockquote></div>
</blockquote></div>