[slurm-users] monitoring and update regime for Power Saving nodes

Thu Feb 24 13:03:13 UTC 2022

Hi Tina,

Thanks, its not so much the config being fully up to date on compute nodes, its more, when we transition into the system wide maintenance day reservation, I anticipate some of the compute nodes will be down due to power saving (I'm expecting the reservation not to impact that, no job will be trying to start on the power saving node(s), therefore they will be powered down but reserved). So then they need a prod to come up, so that could be achieved by a short running job, then after that, though they are reserved, they might try to turn off, if their idle threshold is met. So I guess you then either need to clustershell off (or similar) the slurmd or roll out a temporary (maintenance period only) slurm.conf that won't mean idle time threshold is hit anywhere.

(I probably didn't explain the slurm config bit very well)

I am perhaps overthinking it, a dummy job to bring powered down nodes up.... then a clustershell slurmd stop is probably the answer

regards
David

-------------
David Simpson - Senior Systems Engineer
ARCCA, Redwood Building,
King Edward VII Avenue,
Cardiff, CF10 3NB                                                                               

David Simpson - peiriannydd uwch systemau
ARCCA, Adeilad Redwood,
King Edward VII Avenue,
Caerdydd, CF10 3NB

simpsond4 at cardiff.ac.uk
+44 29208 74657

-----Original Message-----
From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Tina Friedrich
Sent: 24 February 2022 09:43
To: slurm-users at lists.schedmd.com
Subject: Re: [slurm-users] monitoring and update regime for Power Saving nodes

Hi David,

it's also not actually a problem if the slurm.conf is not exactly the same immediately on boot - really. Unless there's changes that are very fundamental, nothing bad will happen if they pick up a new copy after, say, 5 or 10 minutes.

But it should be possible to - for example - force a run of your config management on startup (or before SLURM startup)?

(Not many ideas about the Nagios check, unless you change it to something that interrogates SLURM about node states, or keep some other record somewhere that it can interrogate about nodes meant to be down.)

Tina

On 24/02/2022 09:20, David Simpson wrote:
> Hi Brian,
> 
>  >>For monitoring, I use a combination of netdata+prometheus. Data is 
> gathered whenever the nodes are up and stored for history. Yes, when 
> the nodes are powered down, there are empty gaps, but that is 
> interpreted as the node is powered down.
> 
> Ah time-series will cope much better - at the moment our monitoring 
> system (for compute node health at least) is nagios-like, hence the 
> problem. Though there's potential the entire cluster's stack may 
> change at some point, so this problem will be more easy to deal with 
> (with a change of monitoring system for node health).
> 
>  >>For the config, I have no access to DNS for configless so I use a 
> symlink to the slurm.conf file a shared filesystem. This works great.
> Anytime there are changes, a simple 'scontrol reconfigure' brings all 
> running nodes up to speed and any down nodes will automatically read 
> the latest.
> 
> Yes, currently we use file based and config written to the compute 
> node's disks themselves via ansible. Perhaps we will consider moving 
> the file to a shared fs.
> 
> regards
> David
> 
> -------------
> 
> David Simpson - Senior Systems Engineer
> 
> ARCCA, Redwood Building,
> 
> King Edward VII Avenue,
> 
> Cardiff, CF10 3NB
> 
> David Simpson - peiriannydd uwch systemau
> 
> ARCCA, Adeilad Redwood,
> 
> King Edward VII Avenue,
> 
> Caerdydd, CF10 3NB
> 
> simpsond4 at cardiff.ac.uk <mailto:simpsond4 at cardiff.ac.uk>
> 
> +44 29208 74657
> 
> *From:*slurm-users <slurm-users-bounces at lists.schedmd.com> *On Behalf 
> Of *Brian Andrus
> *Sent:* 23 February 2022 15:27
> *To:* slurm-users at lists.schedmd.com
> *Subject:* Re: [slurm-users] monitoring and update regime for Power 
> Saving nodes
> 
> *External email to Cardiff University - *Take care when 
> replying/opening attachments or links.
> 
> *Nid ebost mewnol o Brifysgol Caerdydd yw hwn - *Cymerwch ofal wrth 
> ateb/agor atodiadau neu ddolenni.
> 
> David,
> 
> For monitoring, I use a combination of netdata+prometheus. Data is 
> gathered whenever the nodes are up and stored for history. Yes, when 
> the nodes are powered down, there are empty gaps, but that is 
> interpreted as the node is powered down.
> 
> For the config, I have no access to DNS for configless so I use a 
> symlink to the slurm.conf file a shared filesystem. This works great.
> Anytime there are changes, a simple 'scontrol reconfigure' brings all 
> running nodes up to speed and any down nodes will automatically read 
> the latest.
> 
> Brian Andrus
> 
> On 2/23/2022 2:31 AM, David Simpson wrote:
> 
>     Hi all,
> 
>     Interested to know what common approaches were to:
> 
> 
>      1. Monitoring of power saving nodes (e.g. health of the node), when
>         potentially the monitoring system will see it go up and down. Do
>         you limit to BMC only monitoring/health?
>      2. When you want to make changes to slurm.conf (or anything else)
>         to a node which is down due to power saving (during a
>         maintenance/reservation) what is your approach? Do you end up
>         with 2 slurm.confs (one for power saving and one that keeps
>         everything up, to work on during the maintenance)?
> 
> 
>     thanks
>     David
> 
> 
>     -------------
> 
>     David Simpson - Senior Systems Engineer
> 
>     ARCCA, Redwood Building,
> 
>     King Edward VII Avenue,
> 
>     Cardiff, CF10 3NB
> 
>     David Simpson - peiriannydd uwch systemau
> 
>     ARCCA, Adeilad Redwood,
> 
>     King Edward VII Avenue,
> 
>     Caerdydd, CF10 3NB
> 
>     simpsond4 at cardiff.ac.uk <mailto:simpsond4 at cardiff.ac.uk>
> 
>     +44 29208 74657
> 

--
Tina Friedrich, Advanced Research Computing Snr HPC Systems Administrator

Research Computing and Support Services
IT Services, University of Oxford
https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.arc.ox.ac.uk%2F&data=04%7C01%7Csimpsond4%40cardiff.ac.uk%7C0309504ae4784c46798008d9f77c2b60%7Cbdb74b3095684856bdbf06759778fcbc%7C1%7C0%7C637812934947426854%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=StOhnAdss5NbDB0gfTTc40ow43fYVpmZXY73xnwi%2Bec%3D&reserved=0 https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.it.ox.ac.uk%2F&data=04%7C01%7Csimpsond4%40cardiff.ac.uk%7C0309504ae4784c46798008d9f77c2b60%7Cbdb74b3095684856bdbf06759778fcbc%7C1%7C0%7C637812934947426854%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=7NOUTmoDVMQ4SutDOjRYBZQxxaTkP7U9CPkewAKul3w%3D&reserved=0