[slurm-users] [EXT] slurmctld error

Thu Apr 8 07:53:31 UTC 2021

Total memory in each node is 3940 and free from 3353 to 3378, which value should I give to RealMemory

For each node  I have to create a different entry in slurm.conf ?

How can I check that each node can contact the slurmd port on every other node?

jb

From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Sean Crosby
Sent: Thursday, April 8, 2021 10:18 AM
To: Slurm User Community List <slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] [EXT] slurmctld error

The reason why your nodes are drained is "Low RealMemory"

This reason is because you have told Slurm about the RAM on the node, but it is less than the RAM on the node.

You have told Slurm that the amount of RAM on wn001 is 3934MB

What does

free -m

show on wn001?

The DBD looks good now!

Can you also double check that you can resolve the worker node names from the Slurm controller and between each node

e.g.

on ctl - ping wn001

on wn001:

  ping wn002

  ping se01

on wn002:

  ping wn001

  ping se01

Also double check that each node can contact the slurmd port on every other node

Sean

--
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead
Research Computing Services | Business Services
The University of Melbourne, Victoria 3010 Australia

On Thu, 8 Apr 2021 at 16:38, Ioannis Botsis <ibotsis at isc.tuc.gr <mailto:ibotsis at isc.tuc.gr> > wrote:

UoM notice: External email. Be cautious of links, attachments, or impersonation attempts

  _____  

Hi Sean

I made all the changes you recommended but the problem remains.

Attached you will find dbd & ctld log files an slurmd log file from one node wn001. Also slum configuration.

scontrol show node wn001

NodeName=wn001 Arch=x86_64 CoresPerSocket=2

   CPUAlloc=0 CPUTot=2 CPULoad=0.01

   AvailableFeatures=(null)

   ActiveFeatures=(null)

   Gres=(null)

   NodeAddr=wn001 NodeHostName=wn001 Version=19.05.5

   OS=Linux 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021

   RealMemory=3934 AllocMem=0 FreeMem=3101 Sockets=2 Boards=1

   State=DOWN+DRAIN ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A

   Partitions=aTUC

   BootTime=2021-04-01T13:26:24 SlurmdStartTime=2021-04-07T10:53:20

   CfgTRES=cpu=2,mem=3934M,billing=2

   AllocTRES=

   CapWatts=n/a

   CurrentWatts=0 AveWatts=0

   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

   Reason=Low RealMemory [root at 2021-04-

From: slurm-users <slurm-users-bounces at lists.schedmd.com <mailto:slurm-users-bounces at lists.schedmd.com> > On Behalf Of Sean Crosby
Sent: Tuesday, April 6, 2021 2:11 PM
To: Slurm User Community List <slurm-users at lists.schedmd.com <mailto:slurm-users at lists.schedmd.com> >
Subject: Re: [slurm-users] [EXT] slurmctld error

I just checked my cluster and my spool dir is

SlurmdSpoolDir=/var/spool/slurm

(i.e. without the d at the end)

It doesn't really matter, as long as the directory exists and has the correct permissions on all nodes

--
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead
Research Computing Services | Business Services
The University of Melbourne, Victoria 3010 Australia

On Tue, 6 Apr 2021 at 20:52, Sean Crosby <scrosby at unimelb.edu.au <mailto:scrosby at unimelb.edu.au> > wrote:

I think I've worked out a problem

I see in your slurm.conf you have this

SlurmdSpoolDir=/var/spool/slurm/d

It should be

SlurmdSpoolDir=/var/spool/slurmd

You'll need to restart slurmd on all the nodes after you make that change

I would also double check the permissions on that directory on all your nodes. It needs to be owned by user slurm

ls -lad /var/spool/slurmd

Sean

--
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead
Research Computing Services | Business Services
The University of Melbourne, Victoria 3010 Australia

On Tue, 6 Apr 2021 at 20:37, Sean Crosby <scrosby at unimelb.edu.au <mailto:scrosby at unimelb.edu.au> > wrote:

It looks like your ctl isn't contacting the slurmdbd properly. The control host, control port etc are all blank.

The first thing I would do is change the ClusterName in your slurm.conf from upper case TUC to lower case tuc. You'll then need to restart your ctld. Then recheck sacctmgr show cluster

If that doesn't work, try changing AccountingStorageHost in slurm.conf to localhost as well

For your worker nodes, your nodes are all in drain state.

Show the output of

scontrol show node wn001

It will give you the reason for why the node is drained.

Sean

--
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead
Research Computing Services | Business Services
The University of Melbourne, Victoria 3010 Australia

On Tue, 6 Apr 2021 at 20:19, <ibotsis at isc.tuc.gr <mailto:ibotsis at isc.tuc.gr> > wrote:

UoM notice: External email. Be cautious of links, attachments, or impersonation attempts

  _____  

sinfo -N -o "%N %T %C %m %P %a"

NODELIST STATE CPUS(A/I/O/T) MEMORY PARTITION AVAIL

wn001 drained 0/0/2/2 3934 TUC* up

wn002 drained 0/0/2/2 3934 TUC* up

wn003 drained 0/0/2/2 3934 TUC* up

wn004 drained 0/0/2/2 3934 TUC* up

wn005 drained 0/0/2/2 3934 TUC* up

wn006 drained 0/0/2/2 3934 TUC* up

wn007 drained 0/0/2/2 3934 TUC* up

wn008 drained 0/0/2/2 3934 TUC* up

wn009 drained 0/0/2/2 3934 TUC* up

wn010 drained 0/0/2/2 3934 TUC* up

wn011 drained 0/0/2/2 3934 TUC* up

wn012 drained 0/0/2/2 3934 TUC* up

wn013 drained 0/0/2/2 3934 TUC* up

wn014 drained 0/0/2/2 3934 TUC* up

wn015 drained 0/0/2/2 3934 TUC* up

wn016 drained 0/0/2/2 3934 TUC* up

wn017 drained 0/0/2/2 3934 TUC* up

wn018 drained 0/0/2/2 3934 TUC* up

wn019 drained 0/0/2/2 3934 TUC* up

wn020 drained 0/0/2/2 3934 TUC* up

wn021 drained 0/0/2/2 3934 TUC* up

wn022 drained 0/0/2/2 3934 TUC* up

wn023 drained 0/0/2/2 3934 TUC* up

wn024 drained 0/0/2/2 3934 TUC* up

wn025 drained 0/0/2/2 3934 TUC* up

wn026 drained 0/0/2/2 3934 TUC* up

wn027 drained 0/0/2/2 3934 TUC* up

wn028 drained 0/0/2/2 3934 TUC* up

wn029 drained 0/0/2/2 3934 TUC* up

wn030 drained 0/0/2/2 3934 TUC* up

wn031 drained 0/0/2/2 3934 TUC* up

wn032 drained 0/0/2/2 3934 TUC* up

wn033 drained 0/0/2/2 3934 TUC* up

wn034 drained 0/0/2/2 3934 TUC* up

wn035 drained 0/0/2/2 3934 TUC* up

wn036 drained 0/0/2/2 3934 TUC* up

wn037 drained 0/0/2/2 3934 TUC* up

wn038 drained 0/0/2/2 3934 TUC* up

wn039 drained 0/0/2/2 3934 TUC* up

wn040 drained 0/0/2/2 3934 TUC* up

wn041 drained 0/0/2/2 3934 TUC* up

wn042 drained 0/0/2/2 3934 TUC* up

wn043 drained 0/0/2/2 3934 TUC* up

wn044 drained 0/0/2/2 3934 TUC* up

From: slurm-users <slurm-users-bounces at lists.schedmd.com <mailto:slurm-users-bounces at lists.schedmd.com> > On Behalf Of Sean Crosby
Sent: Tuesday, April 6, 2021 12:47 PM
To: Slurm User Community List <slurm-users at lists.schedmd.com <mailto:slurm-users at lists.schedmd.com> >
Subject: Re: [slurm-users] [EXT] slurmctld error

It looks like your attachment of sinfo -R didn't come through

It also looks like your dbd isn't set up correctly

Can you also show the output of

sacctmgr list cluster

and

scontrol show config | grep ClusterName

Sean

--
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead
Research Computing Services | Business Services
The University of Melbourne, Victoria 3010 Australia

On Tue, 6 Apr 2021 at 19:18, Ioannis Botsis <ibotsis at isc.tuc.gr <mailto:ibotsis at isc.tuc.gr> > wrote:

UoM notice: External email. Be cautious of links, attachments, or impersonation attempts

  _____  

Hi Sean,

I am trying to submit a simple job but freeze

srun -n44 -l /bin/hostname

srun: Required node not available (down, drained or reserved)

srun: job 15 queued and waiting for resources

^Csrun: Job allocation 15 has been revoked

srun: Force Terminated job 15

daemons are active and running on server and all nodes 

nodes definition in slurm.conf is …

DefMemPerNode=3934

NodeName=wn0[01-44] CPUs=2 RealMemory=3934 Sockets=2 CoresPerSocket=2 State=UNKNOWN

PartitionName=TUC Nodes=ALL Default=YES MaxTime=INFINITE State=UP

tail -10 /var/log/slurmdbd.log

[2021-04-06T12:09:16.481] error: We should have gotten a new id: Table 'slurm_acct_db.tuc_job_table' doesn't exist

[2021-04-06T12:09:16.481] error: _add_registered_cluster: trying to register a cluster (tuc) with no remote port

[2021-04-06T12:09:16.482] error: We should have gotten a new id: Table 'slurm_acct_db.tuc_job_table' doesn't exist

[2021-04-06T12:09:16.482] error: It looks like the storage has gone away trying to reconnect

[2021-04-06T12:09:16.483] error: We should have gotten a new id: Table 'slurm_acct_db.tuc_job_table' doesn't exist

[2021-04-06T12:09:16.483] error: _add_registered_cluster: trying to register a cluster (tuc) with no remote port

[2021-04-06T12:09:16.484] error: We should have gotten a new id: Table 'slurm_acct_db.tuc_job_table' doesn't exist

[2021-04-06T12:09:16.484] error: It looks like the storage has gone away trying to reconnect

[2021-04-06T12:09:16.484] error: We should have gotten a new id: Table 'slurm_acct_db.tuc_job_table' doesn't exist

[2021-04-06T12:09:16.485] error: _add_registered_cluster: trying to register a cluster (tuc) with no remote port

tail -10 /var/log/slurmctld.log

[2021-04-06T12:09:35.701] debug:  backfill: no jobs to backfill

[2021-04-06T12:09:42.001] debug:  slurmdbd: PERSIST_RC is -1 from DBD_FLUSH_JOBS(1408): (null)

[2021-04-06T12:10:00.042] debug:  slurmdbd: PERSIST_RC is -1 from DBD_FLUSH_JOBS(1408): (null)

[2021-04-06T12:10:05.701] debug:  backfill: beginning

[2021-04-06T12:10:05.701] debug:  backfill: no jobs to backfill

[2021-04-06T12:10:05.989] debug:  sched: Running job scheduler

[2021-04-06T12:10:19.001] debug:  slurmdbd: PERSIST_RC is -1 from DBD_FLUSH_JOBS(1408): (null)

[2021-04-06T12:10:35.702] debug:  backfill: beginning

[2021-04-06T12:10:35.702] debug:  backfill: no jobs to backfill

[2021-04-06T12:10:37.001] debug:  slurmdbd: PERSIST_RC is -1 from DBD_FLUSH_JOBS(1408): (null)

Attached sinfo -R  

Any hint?

jb

From: slurm-users <slurm-users-bounces at lists.schedmd.com <mailto:slurm-users-bounces at lists.schedmd.com> > On Behalf Of Sean Crosby
Sent: Tuesday, April 6, 2021 7:54 AM
To: Slurm User Community List <slurm-users at lists.schedmd.com <mailto:slurm-users at lists.schedmd.com> >
Subject: Re: [slurm-users] [EXT] slurmctld error

The other thing I notice for my slurmdbd.conf is that I have

DbdAddr=localhost
DbdHost=localhost

You can try changing your slurmdbd.conf to set those 2 values as well to see if that gets slurmdbd to listen on port 6819

Sean

--
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead
Research Computing Services | Business Services
The University of Melbourne, Victoria 3010 Australia

On Tue, 6 Apr 2021 at 14:31, Sean Crosby < <mailto:scrosby at unimelb.edu.au> scrosby at unimelb.edu.au> wrote:

Interesting. It looks like slurmdbd is not opening the 6819 port

What does

ss -lntp | grep 6819

show? Is something else using that port?

You can also stop the slurmdbd service and run it in debug mode using

slurmdbd -D -vvv

Sean

--
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead
Research Computing Services | Business Services
The University of Melbourne, Victoria 3010 Australia

On Tue, 6 Apr 2021 at 14:02, < <mailto:ibotsis at isc.tuc.gr> ibotsis at isc.tuc.gr> wrote:

UoM notice: External email. Be cautious of links, attachments, or impersonation attempts

  _____  

Hi Sean

ss -lntp | grep $(pidof slurmdbd)     return nothing……

systemctl status slurmdbd.service

● slurmdbd.service - Slurm DBD accounting daemon

     Loaded: loaded (/lib/systemd/system/slurmdbd.service; enabled; vendor preset: enabled)

     Active: active (running) since Mon 2021-04-05 13:52:35 EEST; 16h ago

       Docs: man:slurmdbd(8)

    Process: 1453365 ExecStart=/usr/sbin/slurmdbd $SLURMDBD_OPTIONS (code=exited, status=0/SUCCESS)

   Main PID: 1453375 (slurmdbd)

      Tasks: 1

     Memory: 5.0M

     CGroup: /system.slice/slurmdbd.service

             └─1453375 /usr/sbin/slurmdbd

Apr 05 13:52:35  <http://se01.grid.tuc.gr> se01.grid.tuc.gr systemd[1]: Starting Slurm DBD accounting daemon...

Apr 05 13:52:35  <http://se01.grid.tuc.gr> se01.grid.tuc.gr systemd[1]: slurmdbd.service: Can't open PID file /run/slurmdbd.pid (yet?) after start: Operation not permitted

Apr 05 13:52:35  <http://se01.grid.tuc.gr> se01.grid.tuc.gr systemd[1]: Started Slurm DBD accounting daemon.

File /run/slurmdbd.pid exist and has  pidof slurmdbd   value….

From: slurm-users < <mailto:slurm-users-bounces at lists.schedmd.com> slurm-users-bounces at lists.schedmd.com> On Behalf Of Sean Crosby
Sent: Tuesday, April 6, 2021 12:49 AM
To: Slurm User Community List < <mailto:slurm-users at lists.schedmd.com> slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] [EXT] slurmctld error

What's the output of

ss -lntp | grep $(pidof slurmdbd)

on your dbd host?

Sean

--
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead
Research Computing Services | Business Services
The University of Melbourne, Victoria 3010 Australia

On Tue, 6 Apr 2021 at 05:00, < <mailto:ibotsis at isc.tuc.gr> ibotsis at isc.tuc.gr> wrote:

UoM notice: External email. Be cautious of links, attachments, or impersonation attempts

  _____  

Hi Sean,

10.0.0.100 is the dbd and ctld host with name se01. Firewall is inactive……

nc -nz 10.0.0.100 6819 || echo Connection not working

give me back …..  Connection not working

jb

From: slurm-users < <mailto:slurm-users-bounces at lists.schedmd.com> slurm-users-bounces at lists.schedmd.com> On Behalf Of Sean Crosby
Sent: Monday, April 5, 2021 2:52 PM
To: Slurm User Community List < <mailto:slurm-users at lists.schedmd.com> slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] [EXT] slurmctld error

The error shows

slurmctld: debug2: Error connecting slurm stream socket at  <http://10.0.0.100:6819> 10.0.0.100:6819: Connection refused

slurmctld: error: slurm_persist_conn_open_without_init: failed to open persistent connection to se01:6819: Connection refused

Is 10.0.0.100 the IP address of the host running slurmdbd?

If so, check the iptables firewall running on that host, and make sure the ctld server can access port 6819 on the dbd host.

You can check this by running the following from the ctld host (requires the package nmap-ncat installed)

nc -nz 10.0.0.100 6819 || echo Connection not working

This will try connecting to port 6819 on the host 10.0.0.100, and output nothing if the connection works, and would output Connection not working otherwise

I would also test this on the DBD server itself

--
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead
Research Computing Services | Business Services
The University of Melbourne, Victoria 3010 Australia

On Mon, 5 Apr 2021 at 21:00, Ioannis Botsis < <mailto:ibotsis at isc.tuc.gr> ibotsis at isc.tuc.gr> wrote:

UoM notice: External email. Be cautious of links, attachments, or impersonation attempts

  _____  

Hi Sean,

Thank you for your prompt response,  I made the changes you suggested, slurmctld refuse running……. find attached new slurmctld -Dvvvv

jb

From: slurm-users < <mailto:slurm-users-bounces at lists.schedmd.com> slurm-users-bounces at lists.schedmd.com> On Behalf Of Sean Crosby
Sent: Monday, April 5, 2021 11:46 AM
To: Slurm User Community List < <mailto:slurm-users at lists.schedmd.com> slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] [EXT] slurmctld error

Hi Jb,

You have set AccountingStoragePort to 3306 in slurm.conf, which is the MySQL port running on the DBD host.

AccountingStoragePort is the port for the Slurmdbd service, and not for MySQL.

Change AccountingStoragePort to 6819 and it should fix your issues.

I also think you should comment out the lines 

AccountingStorageUser=slurm
AccountingStoragePass=/run/munge/munge.socket.2

You shouldn't need those lines

Sean

--
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead
Research Computing Services | Business Services
The University of Melbourne, Victoria 3010 Australia

On Mon, 5 Apr 2021 at 18:03, Ioannis Botsis < <mailto:ibotsis at isc.tuc.gr> ibotsis at isc.tuc.gr> wrote:

UoM notice: External email. Be cautious of links, attachments, or impersonation attempts

  _____  

Hello everyone,

I installed the slurm 19.05.5 from Ubuntu repo,  for the first time in a cluster with 44  identical nodes but I have problem with slurmctld.service

When I try to activate slurmctd I get the following message…

fatal: You are running with a database but for some reason we have no TRES from it.  This should only happen if the database is down and you don't have any state files

*	Ubuntu 20.04.2 runs on the server and nodes in the exact same version.
*	munge 0.5.13 installed from Ubuntu repo running on server and nodes.
*	mysql  Ver 8.0.23-0ubuntu0.20.04.1 for Linux on x86_64 ((Ubuntu))  installed from ubuntu repo running on server.

slurm.conf is the same on all nodes and on server.

slurmd.service is active and running on all nodes without problem.

mysql.service is active and running on server.

slurmdbd.service is active and running on server (slurm_acct_db created).

Find attached slurm.conf  <http://slurmdbd.com> slurmdbd.com  and detailed output of slurmctld -Dvvvv  command.

Any hint?

Thanks in advance

jb

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210408/d59f3baf/attachment-0001.htm>