I've read the following in the slurm power saving docs: https://slurm.schedmd.com/power_save.html
*cloud_dns* By default, Slurm expects that the network addresses for cloud nodes won't be known until creation of the node and that Slurm will be notified of the node's address upon registration. Since Slurm communications rely on the node configuration found in the slurm.conf, Slurm will tell the client command, after waiting for all nodes to boot, each node's IP address. However, in environments where the nodes are in DNS, this step can be avoided by configuring this option.
I am creating the nodes on demand and don't know the IP ahead of the instance start, so cloud_dns is not set.
I'm confused specifically by "Slurm will be notified of the node's address upon registration." Who/what is expected to do this? If it is expected to be performed by the ResumeProgram, does it need to be done before slurmd starts on the node? Is it OK if the node does it after slurmd has started with something like:
scontrol update nodename=$(hostname -s) nodeaddr=$(hostname -I) nodehostname=$(hostname) scontrol reconfigure
Thank you,
Martin