slurm not running on a warewulf node

List overview All Threads
Download

newer

older

Non-Standard Mail Notification in...

slurmd on a warwwulf node - not...

Steven Jones

4 Dec 2024 4 Dec '24

12:39 a.m.

Hi,

I have set a log creation/location in slurm.conf as,

SlurmdLogFile=/var/log/slurm/slurmd.log

But it is 0 length.

Slurm will not run, what else do I need to do to log why its failing pls?

regards

Steven

Attachments:

attachment.html (text/html — 1.2 KB)

Show replies by date

Jeffrey R. Lang

4 Dec 4 Dec

1:17 a.m.

New subject: Slurm not running on a warewulf node

Steve

Trying running the failing process from the command line and use the -D option.

Per man page: Run slurmd in the foreground. Error and debug messages will be copied to stderr.

Jeffrey R. Lang Advanced Research Computing Center University of Wyoming, Information Technology Center 1000 E. University Ave Laramie, WY 82071

Email: jrlang@uwyo.edu Work: 307.766.3381

From: Steven Jones via slurm-users slurm-users@lists.schedmd.com Sent: Tuesday, December 3, 2024 5:39 PM To: slurm-users@schedmd.com Subject: [slurm-users] slurm not running on a warewulf node

◆ This message was sent from a non-UWYO address. Please exercise caution when clicking links or opening attachments from external sources.

Hi,

I have set a log creation/location in slurm.conf as,

SlurmdLogFile=/var/log/slurm/slurmd.log

But it is 0 length.

Slurm will not run, what else do I need to do to log why its failing pls?

regards

Steven

Steven Jones

1:28 a.m.

New subject: Slurm not running on a warewulf node

Well that is a start, TY.

[root@node1 slurm]# /usr/sbin/slurmd -D slurmd: fatal: Unable to determine this slurmd's NodeName

Where is this set?

regards

Steven

________________________________ From: Jeffrey R. Lang JRLang@uwyo.edu Sent: Wednesday, 4 December 2024 1:17 pm To: Steven Jones steven.jones@vuw.ac.nz; slurm-users@schedmd.com slurm-users@schedmd.com Subject: RE: Slurm not running on a warewulf node

You don't often get email from jrlang@uwyo.edu. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

Steve

Trying running the failing process from the command line and use the -D option.

Per man page: Run slurmd in the foreground. Error and debug messages will be copied to stderr.

Jeffrey R. Lang

Advanced Research Computing Center

University of Wyoming, Information Technology Center

1000 E. University Ave

Laramie, WY 82071

Email: jrlang@uwyo.edu

Work: 307.766.3381

From: Steven Jones via slurm-users slurm-users@lists.schedmd.com Sent: Tuesday, December 3, 2024 5:39 PM To: slurm-users@schedmd.com Subject: [slurm-users] slurm not running on a warewulf node

◆ This message was sent from a non-UWYO address. Please exercise caution when clicking links or opening attachments from external sources.

Hi,

I have set a log creation/location in slurm.conf as,

SlurmdLogFile=/var/log/slurm/slurmd.log

But it is 0 length.

Slurm will not run, what else do I need to do to log why its failing pls?

regards

Steven

Steven Jones

1:47 a.m.

New subject: Slurm not running on a warewulf node

I guess I have the syntax wrong,

root@node1 slurm]# /usr/sbin/slurmd -D slurmd: fatal: Unable to create NodeAddr list from node[1-7].ods.vuw.ac.nz [root@node1 slurm]# tail /etc/slurm/slurm.conf #ResumeRate= #SuspendExcNodes= #SuspendExcParts= #SuspendRate= #SuspendTime= # # # COMPUTE NODES NodeName=node[1-7].ods.vuw.ac.nz CPUs=20 RealMemory=48 State=UNKNOWN PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP [root@node1 slurm]#

regards

Steven

________________________________ From: Steven Jones via slurm-users slurm-users@lists.schedmd.com Sent: Wednesday, 4 December 2024 1:28 pm To: slurm-users@schedmd.com slurm-users@schedmd.com Subject: [slurm-users] Re: Slurm not running on a warewulf node

Well that is a start, TY.

[root@node1 slurm]# /usr/sbin/slurmd -D slurmd: fatal: Unable to determine this slurmd's NodeName

Where is this set?

regards

Steven

You don't often get email from jrlang@uwyo.edu. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

Steve

Trying running the failing process from the command line and use the -D option.

Per man page: Run slurmd in the foreground. Error and debug messages will be copied to stderr.

Jeffrey R. Lang

Advanced Research Computing Center

University of Wyoming, Information Technology Center

1000 E. University Ave

Laramie, WY 82071

Email: jrlang@uwyo.edu

Work: 307.766.3381

From: Steven Jones via slurm-users slurm-users@lists.schedmd.com Sent: Tuesday, December 3, 2024 5:39 PM To: slurm-users@schedmd.com Subject: [slurm-users] slurm not running on a warewulf node

◆ This message was sent from a non-UWYO address. Please exercise caution when clicking links or opening attachments from external sources.

Hi,

I have set a log creation/location in slurm.conf as,

SlurmdLogFile=/var/log/slurm/slurmd.log

But it is 0 length.

Slurm will not run, what else do I need to do to log why its failing pls?

regards

Steven

Hermann Schwärzler

11:30 a.m.

New subject: Slurm not running on a warewulf node

Hi Steven,

yes, you have the syntax a bit wrong. If you consult the documentation (or the man-page) of slurm.conf you find this in the "NODE CONFIGURATION" section (in the paragraph about "NodeName"):

Note that if the short form of the hostname is not used, it may prevent use of hostlist expressions (the numeric portion in brackets must be at the end of the string)

So the respective part in your slurm.conf should be

NodeName=node[1-7] ...

and you have to configure your name resolution (default domain?) such that these short names are resolvable to IP-addresses.

If that's not feasible you might have to use e.g. something like this

NodeName=DEFAULT CPUs=20 RealMemory=48 NodeName=node1.ods.vuw.ac.nz NodeName=node2.ods.vuw.ac.nz ...

BTW: do your nodes only have 48 MB of memory? The unit in which "RealMemory" has to be specified is megabytes.

Regards, Hermann

On 12/4/24 01:47, Steven Jones via slurm-users wrote:

...

I guess I have the syntax wrong,

root@node1 slurm]# /usr/sbin/slurmd -D slurmd: fatal: Unable to create NodeAddr list from node[1-7].ods.vuw.ac.nz [root@node1 slurm]# tail /etc/slurm/slurm.conf #ResumeRate= #SuspendExcNodes= #SuspendExcParts= #SuspendRate= #SuspendTime= # # # COMPUTE NODES NodeName=node[1-7].ods.vuw.ac.nz CPUs=20 RealMemory=48 State=UNKNOWN PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP [root@node1 slurm]#

regards

Steven

*From:* Steven Jones via slurm-users slurm-users@lists.schedmd.com *Sent:* Wednesday, 4 December 2024 1:28 pm *To:* slurm-users@schedmd.com slurm-users@schedmd.com *Subject:* [slurm-users] Re: Slurm not running on a warewulf node Well that is a start, TY.

[root@node1 slurm]# /usr/sbin/slurmd -D slurmd: fatal: Unable to determine this slurmd's NodeName

Where is this set?

regards

Steven

*From:* Jeffrey R. Lang JRLang@uwyo.edu *Sent:* Wednesday, 4 December 2024 1:17 pm *To:* Steven Jones steven.jones@vuw.ac.nz; slurm-users@schedmd.com slurm-users@schedmd.com *Subject:* RE: Slurm not running on a warewulf node

You don't often get email from jrlang@uwyo.edu. Learn why this is important https://aka.ms/LearnAboutSenderIdentification

Steve

Trying running the failing process from the command line and use the -D option.

Per man page: Run slurmd in the foreground. Error and debug messages will be copied to stderr.

*Jeffrey R. Lang*

Advanced Research Computing Center

University of Wyoming, Information Technology Center

1000 E. University Ave

Laramie, WY 82071

Email: jrlang@uwyo.edu

Work: 307.766.3381

*From:* Steven Jones via slurm-users slurm-users@lists.schedmd.com *Sent:* Tuesday, December 3, 2024 5:39 PM *To:* slurm-users@schedmd.com *Subject:* [slurm-users] slurm not running on a warewulf node

◆ This message was sent from a non-UWYO address. Please exercise caution when clicking links or opening attachments from external sources.

Hi,

I have set a log creation/location in slurm.conf as,

SlurmdLogFile=/var/log/slurm/slurmd.log

But it is 0 length.

Slurm will not run, what else do I need to do to log why its failing pls?

regards

Steven

cgoll＠suse.de

3:35 p.m.

New subject: Slurm not running on a warewulf node

On 04.12.24 11:30, Hermann Schwärzler via slurm-users slurm-users@lists.schedmd.com wrote:

...

Hi Steven,

yes, you have the syntax a bit wrong. If you consult the documentation (or the man-page) of slurm.conf you find this in the "NODE CONFIGURATION" section (in the paragraph about "NodeName"):

Note that if the short form of the hostname is not used, it may prevent use of hostlist expressions (the numeric portion in brackets must be at the end of the string)

So the respective part in your slurm.conf should be

NodeName=node[1-7] ...

and you have to configure your name resolution (default domain?) such that these short names are resolvable to IP-addresses.

If that's not feasible you might have to use e.g. something like this

NodeName=DEFAULT CPUs=20 RealMemory=48 NodeName=node1.ods.vuw.ac.nz NodeName=node2.ods.vuw.ac.nz ...

BTW: do your nodes only have 48 MB of memory? The unit in which "RealMemory" has to be specified is megabytes.

Regards, Hermann

On 12/4/24 01:47, Steven Jones via slurm-users wrote:

...
I guess I have the syntax wrong,

root@node1 slurm]# /usr/sbin/slurmd -D slurmd: fatal: Unable to create NodeAddr list from node[1-7].ods.vuw.ac.nz [root@node1 slurm]# tail /etc/slurm/slurm.conf #ResumeRate= #SuspendExcNodes= #SuspendExcParts= #SuspendRate= #SuspendTime= # # # COMPUTE NODES NodeName=node[1-7].ods.vuw.ac.nz CPUs=20 RealMemory=48 State=UNKNOWN PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP [root@node1 slurm]#

regards

Steven

*From:* Steven Jones via slurm-users slurm-users@lists.schedmd.com *Sent:* Wednesday, 4 December 2024 1:28 pm *To:* slurm-users@schedmd.com slurm-users@schedmd.com *Subject:* [slurm-users] Re: Slurm not running on a warewulf node Well that is a start, TY.

[root@node1 slurm]# /usr/sbin/slurmd -D slurmd: fatal: Unable to determine this slurmd's NodeName

Where is this set?

regards

Steven

*From:* Jeffrey R. Lang JRLang@uwyo.edu *Sent:* Wednesday, 4 December 2024 1:17 pm *To:* Steven Jones steven.jones@vuw.ac.nz; slurm-users@schedmd.com slurm-users@schedmd.com *Subject:* RE: Slurm not running on a warewulf node

You don't often get email from jrlang@uwyo.edu. Learn why this is important https://aka.ms/LearnAboutSenderIdentification

Steve

Trying running the failing process from the command line and use the -D option.

Per man page: Run slurmd in the foreground. Error and debug messages will be copied to stderr.

*Jeffrey R. Lang*

Advanced Research Computing Center

University of Wyoming, Information Technology Center

1000 E. University Ave

Laramie, WY 82071

Email: jrlang@uwyo.edu

Work: 307.766.3381

*From:* Steven Jones via slurm-users slurm-users@lists.schedmd.com *Sent:* Tuesday, December 3, 2024 5:39 PM *To:* slurm-users@schedmd.com *Subject:* [slurm-users] slurm not running on a warewulf node

◆ This message was sent from a non-UWYO address. Please exercise caution when clicking links or opening attachments from external sources.

Hi,

I have set a log creation/location in slurm.conf as,

SlurmdLogFile=/var/log/slurm/slurmd.log

But it is 0 length.

Slurm will not run, what else do I need to do to log why its failing pls?

regards

Steven

Hello Steven, if its warewulf v4 you can have a look at https://github.com/warewulf/warewulf/blob/main/etc/examples/slurm.conf.ww which is a template for a slurm.conf created from the available nodes in warewulf.

kind regards, Christian

Steven Jones

7:16 p.m.

New subject: Slurm not running on a warewulf node

Hi,

Thanks, DNS is broken on my warewulf nodes it seems for some reason, but it cant find the controller's IP for some reason.

I had already swapped to,

NodeName=node[1-7]

and that works OK but not tested much yet.

TY for the MB headsup, I had assumed GB.

So I assume 48 x 1024?

This is a PoC (proof of concept) so some VM nodes with some old nodes thrown in just to test if I can build a HPC.

regards

Steven

________________________________ From: Christian Goll via slurm-users slurm-users@lists.schedmd.com Sent: Thursday, 5 December 2024 3:35 am To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [slurm-users] Re: Slurm not running on a warewulf node

On 04.12.24 11:30, Hermann Schwärzler via slurm-users slurm-users@lists.schedmd.com wrote:

...

Hi Steven,

yes, you have the syntax a bit wrong. If you consult the documentation (or the man-page) of slurm.conf you find this in the "NODE CONFIGURATION" section (in the paragraph about "NodeName"):

Note that if the short form of the hostname is not used, it may prevent use of hostlist expressions (the numeric portion in brackets must be at the end of the string)

So the respective part in your slurm.conf should be

NodeName=node[1-7] ...

and you have to configure your name resolution (default domain?) such that these short names are resolvable to IP-addresses.

If that's not feasible you might have to use e.g. something like this

NodeName=DEFAULT CPUs=20 RealMemory=48 NodeName=node1.ods.vuw.ac.nz NodeName=node2.ods.vuw.ac.nz ...

BTW: do your nodes only have 48 MB of memory? The unit in which "RealMemory" has to be specified is megabytes.

Regards, Hermann

On 12/4/24 01:47, Steven Jones via slurm-users wrote:

...
I guess I have the syntax wrong,

root@node1 slurm]# /usr/sbin/slurmd -D slurmd: fatal: Unable to create NodeAddr list from node[1-7].ods.vuw.ac.nz [root@node1 slurm]# tail /etc/slurm/slurm.conf #ResumeRate= #SuspendExcNodes= #SuspendExcParts= #SuspendRate= #SuspendTime= # # # COMPUTE NODES NodeName=node[1-7].ods.vuw.ac.nz CPUs=20 RealMemory=48 State=UNKNOWN PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP [root@node1 slurm]#

regards

Steven

*From:* Steven Jones via slurm-users slurm-users@lists.schedmd.com *Sent:* Wednesday, 4 December 2024 1:28 pm *To:* slurm-users@schedmd.com slurm-users@schedmd.com *Subject:* [slurm-users] Re: Slurm not running on a warewulf node Well that is a start, TY.

[root@node1 slurm]# /usr/sbin/slurmd -D slurmd: fatal: Unable to determine this slurmd's NodeName

Where is this set?

regards

Steven

*From:* Jeffrey R. Lang JRLang@uwyo.edu *Sent:* Wednesday, 4 December 2024 1:17 pm *To:* Steven Jones steven.jones@vuw.ac.nz; slurm-users@schedmd.com slurm-users@schedmd.com *Subject:* RE: Slurm not running on a warewulf node

You don't often get email from jrlang@uwyo.edu. Learn why this is important https://aka.ms/LearnAboutSenderIdentification

Steve

Trying running the failing process from the command line and use the -D option.

Per man page: Run slurmd in the foreground. Error and debug messages will be copied to stderr.

*Jeffrey R. Lang*

Advanced Research Computing Center

University of Wyoming, Information Technology Center

1000 E. University Ave

Laramie, WY 82071

Email: jrlang@uwyo.edu

Work: 307.766.3381

*From:* Steven Jones via slurm-users slurm-users@lists.schedmd.com *Sent:* Tuesday, December 3, 2024 5:39 PM *To:* slurm-users@schedmd.com *Subject:* [slurm-users] slurm not running on a warewulf node

◆ This message was sent from a non-UWYO address. Please exercise caution when clicking links or opening attachments from external sources.

Hi,

I have set a log creation/location in slurm.conf as,

SlurmdLogFile=/var/log/slurm/slurmd.log

But it is 0 length.

Slurm will not run, what else do I need to do to log why its failing pls?

regards

Steven

Hello Steven, if its warewulf v4 you can have a look at https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com...https://github.com/warewulf/warewulf/blob/main/etc/examples/slurm.conf.ww which is a template for a slurm.conf created from the available nodes in warewulf.

kind regards, Christian

-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com

Age (days ago)

Last active (days ago)

slurm-users@lists.schedmd.com

6 comments

4 participants

tags (0)

participants (4)

cgoll＠suse.de
Hermann Schwärzler
Jeffrey R. Lang
Steven Jones