[slurm-users] Re: CLOUD nodes with unknown IP addresses

Martin,

In a nutshell, when slurmd starts, it tells that info to slurmctld. That is the "registration" event mentioned.

Brian Andrus

On 7/19/2024 5:44 AM, Martin Lee via slurm-users wrote:

I've read the following in the slurm power saving docs: https://slurm.schedmd.com/power_save.html

...

*cloud_dns*

By default, Slurm expects that the network addresses for cloud
nodes won't be known until creation of the node and that Slurm
will be notified of the node's address upon registration. Since
Slurm communications rely on the node configuration found in the
slurm.conf, Slurm will tell the client command, after waiting for
all nodes to boot, each node's IP address. However, in
environments where the nodes are in DNS, this step can be avoided
by configuring this option.

I am creating the nodes on demand and don't know the IP ahead of
the instance start, so cloud_dns is not set.

I'm confused specifically by "Slurm will be notified of the node's
address upon registration." Who/what is expected to do this? If it
is expected to be performed by the ResumeProgram, does it need to
be done before slurmd starts on the node? Is it OK if the node
does it after slurmd has started with something like:

scontrol update nodename=$(hostname -s) nodeaddr=$(hostname -I)
nodehostname=$(hostname)

scontrol reconfigure

Thank you,

Martin