<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Hi,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
follow this guide: <a href="https://slurm.schedmd.com/power_save.html" id="LPlnk920013">https://slurm.schedmd.com/power_save.html</a></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
Create poweroff / poweron scripts and configure slurm to do the poweroff after X minutes. Works well for us. Make sure to set an appropriate time (ResumeTimeout) to allow the node to come back to service.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
Note that we did not achieve good power saving with suspending the nodes, powering them off and on saves way more power. The downside is it takes ~ 5 mins to resume (= power on) the nodes when needed.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
Cheers,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
Florian</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Analabha Roy <hariseldon99@gmail.com><br>
<b>Sent:</b> Monday, 6 February 2023 18:21<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [External] [slurm-users] Hibernating a whole cluster</font>
<div> </div>
</div>
<div>
<div dir="auto">Hi,
<div dir="auto"><br>
</div>
<div dir="auto">I've just finished setup of a single node "cluster" with slurm on ubuntu 20.04. Infrastructural limitations prevent me from running it 24/7, and it's only powered on during business hours.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Currently, I have a cron job running that hibernates that sole node before closing time.</div>
<div dir="auto"><br>
</div>
<div dir="auto">The hibernation is done with standard systemd, and hibernates to the swap partition.</div>
<div dir="auto"><br>
</div>
<div dir="auto"> I have not run any lengthy slurm jobs on it yet. Before I do, can I get some thoughts on a couple of things?</div>
<div dir="auto"><br>
</div>
<div dir="auto">If it hibernated when slurm still had jobs running/queued, would they resume properly when the machine powers back on? </div>
<div dir="auto"><br>
</div>
<div dir="auto">Note that my swap space is bigger than my RAM. </div>
<div dir="auto"><br>
</div>
<div dir="auto">Is it necessary to perhaps setup a pre-hibernate script for systemd to iterate scontrol to suspend all the jobs before hibernating and resume them post-resume? </div>
<div dir="auto"><br>
</div>
<div dir="auto">What about the wall times? I'm uessing that slurm will count the downtime as elapsed for each job. Is there a way to config this, or is the only alternative a post-hibernate script that iteratively updates the wall times of the running jobs
using scontrol again? </div>
<div dir="auto"><br>
</div>
<div dir="auto">Thanks for your attention. </div>
<div dir="auto">Regards </div>
<div dir="auto">AR</div>
</div>
</div>
</body>
</html>