<div dir="ltr"><div class="gmail_default" style=""><font color="#000000" face="arial, sans-serif" style="">@</font><span style="font-size:0.875rem;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;white-space:nowrap;letter-spacing:0.2px;color:rgb(32,33,36)">Rodrigo Santibáñez</span><span class="gmail_default" style="font-size:small;white-space:nowrap;letter-spacing:0.2px;font-weight:bold;font-family:arial,sans-serif;color:rgb(0,0,0)"> </span><span style="color:rgb(0,0,0);font-family:arial,sans-serif">Please see the updated setup related to my question.</span><span style="color:rgb(0,0,0);font-family:arial,sans-serif"></span></div><br>I have compiled in the latest tag `slurm-21-08-4-1` using `./configure --enable-debug --enable-front-end` in order to test in (`non-emulator mode`):<br><br>```<br>git checkout slurm-21-08-4-1<br>./configure --enable-debug --enable-multiple-slurmd<br>make<br>sudo make install<br>```<br><br>I have following lines in my `/usr/local/etc/slurm.conf` file:<br><br>For:<br> <br>```<br>NodeName=home NodeAddr=127.0.0.1 CPUs=4<br>PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP<br>```<br><br>I get the following error: `slurmd: fatal: Frontend not configured correctly in slurm.conf. See FrontEndName in slurm.conf man page.`<br><br><br>When I try:<br>```<br>FrontEndName=home<br>```<br><br>I get following error message:<br><br>```<br>$ sudo slurmd -Dvvv<br>slurmd: debug: Log file re-opened<br>slurmd: error: _find_node_record: lookup failure for node "home"<br>slurmd: error: _find_node_record: lookup failure for node "home", alias "home"<br>slurmd: error: slurmd initialization failed<br>```<br><br>Than when I have tried following:<div><br>```<br>FrontEndName=127.0.0.1<br>NodeName=home NodeHostName=localhost NodeAddr=127.0.0.1 CPUs=4<br>PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP<br>```<br><br>which keeps all the submitted jobs in a pending state with following error message:<br><br>`slurmctld: error: _slurm_rpc_node_registration node=home: Invalid node name specified`<div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Nov 14, 2021 at 3:09 PM Alper Alimoglu <<a href="mailto:alper.alimoglu@gmail.com">alper.alimoglu@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,sans-serif;font-size:small;color:rgb(0,0,0)">@<span style="color:rgb(32,33,36);font-size:0.875rem;letter-spacing:0.2px;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;white-space:nowrap">Rodrigo Santibáñez</span><span class="gmail_default" style="letter-spacing:0.2px;white-space:nowrap"> I think I was not able to clarify my question. </span></div><div class="gmail_default" style="font-family:arial,sans-serif;font-size:small;color:rgb(0,0,0)"><span class="gmail_default" style="letter-spacing:0.2px;white-space:nowrap"><br></span></div><div class="gmail_default" style="font-family:arial,sans-serif;font-size:small;color:rgb(0,0,0)"><span class="gmail_default" style="letter-spacing:0.2px;white-space:nowrap">I am able to successfully run `slurm` that has versions lower than 20, such as `</span>19-05-8-1<span style="letter-spacing:0.2px;white-space:nowrap">`. But with the same configuration slurm that has version 20 or higher does not properly work. </span></div><div class="gmail_default" style="font-family:arial,sans-serif;font-size:small;color:rgb(0,0,0)"><span style="letter-spacing:0.2px;white-space:nowrap">So I get lost to figure out correct configuration structure to work on the latest stable slurm version.</span></div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><font color="#000000" face="arial, sans-serif"><span class="gmail_default" style="font-family:arial,sans-serif;font-size:small;color:rgb(0,0,0)"></span></font><br><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Nov 14, 2021 at 2:11 AM Rodrigo Santibáñez <<a href="mailto:rsantibanez.uchile@gmail.com" target="_blank">rsantibanez.uchile@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Alper,</div><div><br></div><div>Maybe this is relevant to you:</div><div><br></div><div><a id="gmail-m_-8986003679350799010gmail-m_848223721954404611extra_procs"><b>Can Slurm emulate nodes with more
resources than physically exist on the node?</b></a><br>
Yes. In the slurm.conf file, configure <i>SlurmdParameters=config_overrides</i>
and specify
any desired node resource specifications (<i>CPUs</i>, <i>Sockets</i>,
<i>CoresPerSocket</i>, <i>ThreadsPerCore</i>, and/or <i>TmpDisk</i>).
Slurm will use the resource specification for each node that is
given in <i>slurm.conf</i> and will not check these specifications
against those actually found on the node. The system would best be configured
with <i>TaskPlugin=task/none</i>, so that launched tasks can run on any
available CPU under operating system control. <br></div><div><br></div><div>Best<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Nov 13, 2021 at 4:10 AM Alper Alimoglu <<a href="mailto:alper.alimoglu@gmail.com" target="_blank">alper.alimoglu@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,sans-serif;font-size:small;color:rgb(0,0,0)">My goal is to set up a single server `slurm` cluster (only using a single computer) that can run multiple jobs in parallel. <br><br>In my node `nproc` returns 4 so I believe I can run 4 jobs in parallel if they use a single core. In order to do it I run the controller and the worker daemon on the same node.<br>When I submit four jobs at the same time, only one of them is able to run and the other three are not able to run due to the following error: `queued and waiting for resources`. <br><br>I am using `Ubuntu 20.04.3 LTS"`. I have observe that this approach was working on tag version `<=19`: <br><br>```<br>$ git clone <a href="https://github.com/SchedMD/slurm" target="_blank">https://github.com/SchedMD/slurm</a> ~/slurm && cd ~/slurm<br>$ git checkout e2e21cb571ce88a6dd52989ec6fe30da8c4ef15f #slurm-19-05-8-1` <br>$ ./configure --enable-debug --enable-front-end --enable-multiple-slurmd<br>$ sudo make && sudo make install<br>```<br><br>but does not work on higher versions like `slurm 20.02.1` or its `master` branch. <br><br>------<br><br>```<br>❯ sinfo<br>Sat Nov 06 14:17:04 2021<br>NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON<br>home1 1 debug* idle 1 1:1:1 1 0 1 (null) none<br>home2 1 debug* idle 1 1:1:1 1 0 1 (null) none<br>home3 1 debug* idle 1 1:1:1 1 0 1 (null) none<br>home4 1 debug* idle 1 1:1:1 1 0 1 (null) none<br>$ srun -N1 sleep 10 # runs<br>$ srun -N1 sleep 10 # queued and waiting for resources<br>$ srun -N1 sleep 10 # queued and waiting for resources<br>$ srun -N1 sleep 10 # queued and waiting for resources<br>```<br><br>Here, I get lost where since its [emulate-mode][1] they should be able to run in parallel. <br><br><br>They way I build from the source-code:<br><br>```bash<br>git clone <a href="https://github.com/SchedMD/slurm" target="_blank">https://github.com/SchedMD/slurm</a> ~/slurm && cd ~/slurm<br>./configure --enable-debug --enable-multiple-slurmd<br>make<br>sudo make install<br>```<br><br>--------<br><br>```<br>$ hostname -s<br>home<br>$ nproc<br>4<br>```<br><br>##### Compute_node setup:<br><br>```<br>NodeName=home[1-4] NodeHostName=home NodeAddr=127.0.0.1 CPUs=1 ThreadsPerCore=1 Port=17001<br>PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP Shared=FORCE:1<br>```<br><br>I have also tried: `NodeHostName=localhost`<br><br>`slurm.conf` file:<br><br>```bash<br>ControlMachine=home # $(hostname -s)<br>ControlAddr=127.0.0.1<br>ClusterName=cluster<br>SlurmUser=alper<br>MailProg=/home/user/slurm_mail_prog.sh<br>MinJobAge=172800 # 48 h<br>SlurmdSpoolDir=/var/spool/slurmd<br>SlurmdLogFile=/var/log/slurm/slurmd.%n.log<br>SlurmdPidFile=/var/run/slurmd.%n.pid<br>AuthType=auth/munge<br>CryptoType=crypto/munge<br>MpiDefault=none<br>ProctrackType=proctrack/pgid<br>ReturnToService=1<br>SlurmctldPidFile=/var/run/slurmctld.pid<br>SlurmdPort=6820<br>SlurmctldPort=6821<br>StateSaveLocation=/tmp/slurmstate<br>SwitchType=switch/none<br>TaskPlugin=task/none<br>InactiveLimit=0<br>Waittime=0<br>SchedulerType=sched/backfill<br>SelectType=select/linear<br>PriorityDecayHalfLife=0<br>PriorityUsageResetPeriod=NONE<br>AccountingStorageEnforce=limits<br>AccountingStorageType=accounting_storage/slurmdbd<br>AccountingStoreFlags=YES<br>JobCompType=jobcomp/none<br>JobAcctGatherFrequency=30<br>JobAcctGatherType=jobacct_gather/none<br>NodeName=home[1-2] NodeHostName=home NodeAddr=127.0.0.1 CPUs=2 ThreadsPerCore=1 Port=17001<br>PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP Shared=FORCE:1<br>```<br><br>`slurmdbd.conf`:<br><br>```bash<br>AuthType=auth/munge<br>AuthInfo=/var/run/munge/munge.socket.2<br>DbdAddr=localhost<br>DbdHost=localhost<br>SlurmUser=alper<br>DebugLevel=4<br>LogFile=/var/log/slurm/slurmdbd.log<br>PidFile=/var/run/slurmdbd.pid<br>StorageType=accounting_storage/mysql<br>StorageUser=alper<br>StoragePass=12345<br>```<br><br>The way I run slurm:<br><br>```<br>sudo /usr/local/sbin/slurmd <br>sudo /usr/local/sbin/slurmdbd &<br>sudo /usr/local/sbin/slurmctld -cDvvvvvv<br>```<br>---------<br><br>Related:<br>- minimum number of computers for a slurm cluster (<a href="https://stackoverflow.com/a/27788311/2402577" target="_blank">https://stackoverflow.com/a/27788311/2402577</a>)<br>- [Running multiple worker daemons SLURM](<a href="https://stackoverflow.com/a/40707189/2402577" target="_blank">https://stackoverflow.com/a/40707189/2402577</a>)<br>- <a href="https://stackoverflow.com/a/47009930/2402577" target="_blank">https://stackoverflow.com/a/47009930/2402577</a><br><br><br> [1]: <a href="https://slurm.schedmd.com/faq.html#multi_slurmd" target="_blank">https://slurm.schedmd.com/faq.html#multi_slurmd</a><br></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>