<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
Schedmd has docs about how to do this at:<br>
<br>
<a href="https://slurm.schedmd.com/slurm.conf.html#SECTION_LOGGING" id="LPlnk757262">https://slurm.schedmd.com/slurm.conf.html#SECTION_LOGGING</a><br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<br>
Our config at HMS looks like this:<br>
<br>
<br>
<span style="font-family: "Courier New", monospace;">/var/log/slurm/slurmctld.log {</span>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> create 0640 slurm root</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> daily</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> dateext</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> nocompress</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> notifempty</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> rotate 10</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> sharedscripts</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> postrotate</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> /bin/pkill -x --signal SIGUSR2 slurmctld >/dev/null 2>&1</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> endscript</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> prerotate</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> /bin/sh /usr/local/bin/pre_rotate.sh >/dev/null 2>&1</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> endscript</span></div>
<div class="ContentPasted1">}</div>
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
Our pre_rotate scripts rsync's the logs to an NFS mount.</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
Cheers</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
--Mick</div>
<div class="elementToProof">
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<div class="BodyFragment"><font size="2">
<div class="PlainText">
<div class="gmail_signature" style="font-family: arial, sans-serif; font-size: small; color: rgb(34, 34, 34);">
<div dir="ltr"><span style="font-family:Helvetica"><br>
</span></div>
</div>
</div>
</font></div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Davide DelVento <davide.quantum@gmail.com><br>
<b>Sent:</b> Tuesday, October 17, 2023 10:26 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Correct way to do logrotation</font>
<div> </div>
</div>
<div>
<div dir="ltr">I'd be interested in this too, and I'm reposting only because the message was flagged as both "dangerous email" and "spam", so people may not have seen it (hopefully my reply will not suffer the same downfall...)</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Mon, Oct 16, 2023 at 3:26 AM Taras Shapovalov <<a href="mailto:tshapovalov@nvidia.com">tshapovalov@nvidia.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg1347053265667618659">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hello,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
In the past it was recommended to reconfigure slurm daemons in logrotate script, sending a signal I believe was also the way to go. But recently I retested manual logrotation and I see that a removal of log file (for slurmctld, slurmdbd or slurmd) does not
affect the logging of the daemons. The dameons just recreate the log files and continue to write logs there. What is the right way to go in case of the modern Slurm versions?</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Best regards,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Taras</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>