[slurm-users] Re: How to power up all ~idle nodes and verify that they have started up without issue programmatically

That looks very promising. We will have to increase the timeout due to an extensive Ansible setup run on each node before it is ready, but the idea should work nonetheless. I will try it out.

Thank you!

Xaver

On 11/15/24 14:36, Schneider, Gerald wrote:

My approach would be something along this:

sinfo -t POWERED_DOWN -o %n -h | parallel -i -j 10 --timeout 900 srun -w {} hostname

sinfo lists all powered down nodes and the output gets piped into parallel. parallel will then run 10 (or how many you want) srun instances simultaneously, with a timout of 900 seconds to give the hosts enough time to power up. If everything works parallel exists with 0, otherwise it will sum up the exit codes.

works for me like charm, only downside is that parallel is usually needs to be installed for that. But it’s useful for other cases as well.

Regards,

Gerald Schneider

--

Gerald Schneider

tech. Mitarbeiter

IT-SC

Fraunhofer-Institut für Graphische Datenverarbeitung IGD

Joachim-Jungius-Str. 11 | 18059 Rostock | Germany

Telefon +49 6151 155-309 | +49 381 4024-193 | Fax +49 381 4024-199

gerald.schneider@igd-r.fraunhofer.de mailto:gerald.schneider@igd-r.fraunhofer.de| www.igd.fraunhofer.de http://www.igd.fraunhofer.de/

*From:*Xaver Stiensmeier via slurm-users slurm-users@lists.schedmd.com *Sent:* Freitag, 15. November 2024 14:03 *To:* slurm-users@lists.schedmd.com *Subject:* [slurm-users] Re: How to power up all ~idle nodes and verify that they have started up without issue programmatically

Maybe to expand on this even further:

I would like to run something that waits and comes back with a 0 when all workers have been powered up (resume script ran without an issue) and comes back with =/= 0 (for example 1) otherwise. Then I could start other routines to complete the integration test.

And my personal idea was to use something like:

scontrol show nodes | awk '/NodeName=/ {print $1}' | sed
's/NodeName=//' | sort -u | xargs -Inodenames srun -w nodenames
hostname

to execute the hostname command on all instances which forces them to power up. However, that feels a bit clunky and the output is definitely not perfect as it needs parsing.

Best regards, Xaver

On 11/14/24 14:36, Xaver Stiensmeier wrote:

Hi Ole,

thank you for your answer!

I apologize for the unclear wording. We have already implemented
the on demand scheduling.
However, we have not provided a HealthCheckProgram yet (which
simply means that the node starts without health check). I will
look into it regardless of my question.

Back to the question: I am aware that sinfo contains the
information, but I am basically looking for a method like sinfo
that produces a more machine friendly output as I want to verify
the correct start of all nodes programmatically. ClusterShell is
also on my list of software to try out in general.

*More Context*

We are maintaining a tool that creates Slurm clusters in OpenStack
from configuration files and for that we would like to write
integration tests. Therefore, we would like to be able to test
(CI/CD) whether the slurm cluster behaves as expected given
certain configurations of our program. Of course this includes
checking whether the nodes power up.

Best regards,
Xaver

Am 14/11/2024 um 14:57 schrieb Ole Holm Nielsen via slurm-users:

    Hi Xaver,

    On 11/14/24 12:59, Xaver Stiensmeier via slurm-users wrote:

        I would like to startup all ~idle (idle and powered down)
        nodes and check programmatically if all came up as
        expected. For context: this is for a program that sets up
        slurm clusters with on demand cloud scheduling.

        In the most easiest fashion this could be executing a
        command like *srun FORALL hostname* which would return the
        names of the nodes if it succeeds and an error message
        otherwise. However, there's no such input value like
        FORALL as far as I am aware. One could use -N{total node
        number} as all nodes are ~idle when this executes, but I
        don't know an easy way to get the total number of nodes.


    There exists good documentation around this, and I recommend
    to start with the Slurm Power Saving Guide
    (https://slurm.schedmd.com/power_save.html)

    When you have developed a method to power up your cloud nodes,
    the slurmd's will register with slurmctld when they are
    started.  Simply using the "sinfo" command will tell you which
    nodes are up (idle) and which are still in a powered-down
    state (idle~).

    When slurmd starts up, it calls the HealthCheckProgram defined
    in slurm.conf to verify that the node is healthy - strongly
    recommended.  The slurmd won't start if HealthCheckProgram
    gives a faulty status, and you'll need to check such nodes
    manually.

    So there should not be any need to execute commands on the nodes.

    If you wish, you can stll run a command on all "idle" nodes,
    for example using ClusterShell[1]:

    $ clush -bw@slurmstate:idle uname -r

    Best regards,
    Ole

    [1] The Wiki page
    https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_operations/#clustershell
    shows example usage of ClusterShell