<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Yeah, a word of warning about going from 21.08 to 22.05, make
sure you have enough storage on the database host you are doing
the work on and budget a long enough time for the upgrade. We
just converted our 198 GB (compressed, 534 GB raw) database this
week. The initial attempt failed (after running for 8 hours)
because we ran out of disk space (part of the reason we had to
compress is that the server we use for our slurm master only has
800 GB of SSD on it). That meant we had to reimport our DB, which
took 8 hours, plus then we had to drop the job scripts and job
envs, which took another 5 hours, to then attempt the upgrade
which took 2 hours.</p>
<p><br>
</p>
<p>Moral of the story, make sure you have enough space and budget
sufficient time. You may want to consider nulling out the job
scripts and envs for the upgrade as they complete redo the way
those are stored in the database in 22.05 so that it is more
efficient but getting from here to there is the trick.</p>
<p><br>
</p>
<p>For details see the bug report we filed:
<a class="moz-txt-link-freetext" href="https://bugs.schedmd.com/show_bug.cgi?id=14514">https://bugs.schedmd.com/show_bug.cgi?id=14514</a></p>
<p><br>
</p>
<p>-Paul Edmon-</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 7/14/2022 2:34 PM, Timony, Mick
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:MN2PR07MB7248CD875000DD9CBE3569C3B2889@MN2PR07MB7248.namprd07.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Arial, Helvetica, sans-serif; font-size:
10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size:
10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<br>
</div>
<blockquote itemscope=""
itemtype="https://schemas.microsoft.com/QuotedText"
style="border-left: 3px solid rgb(200, 200, 200);
border-top-color: rgb(200, 200, 200); border-right-color:
rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200);
padding-left: 1ex; margin-left: 0.8ex;">
<div style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">
What I can tell you is that we have never had a problem
reimporting the data back in that was dumped from older
versions into a current version database. So the import using
sacctmgr must do the conversion from the older formats to the
newer formats and handle the schema changes.<br>
</div>
</blockquote>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);">That's the bit of info I was missing, I
didn't realise that it outputs the data in a format
that sacctmgr can read.</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);"><br>
</span></div>
<blockquote itemscope=""
itemtype="https://schemas.microsoft.com/QuotedText"
style="border-left: 3px solid rgb(200, 200, 200);
border-top-color: rgb(200, 200, 200); border-right-color:
rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200);
padding-left: 1ex; margin-left: 0.8ex;">
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);">I will note that if you are storing
job_scripts and envs those can eat up a ton of space in
21.08. It looks like they've solved that problem in 22.05
but the archive steps on 21.08 took forever due to those
scripts and envs.</span></div>
</blockquote>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);">Yes, we are storing job_scripts with:</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);"><br>
</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: "Courier New", monospace;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);">AccountingStoreFlags=job_script</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);"><br>
</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);">I think when we made that decision, we
decided that also saving the job_env would take up too much
room as our DB is pretty big at the moment, at approx. 300GB
with the o2_step_table and the o2_job_table taking up the most
space for obvious reasons:<br>
<br>
</span><span style="font-family: "Courier New",
monospace; font-size: 10pt; color: rgb(0, 0, 0);
background-color: rgb(255, 255, 255);">+----------------------------+-----------+</span><span
style="font-family: Arial, Helvetica, sans-serif; font-size:
10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);">
<div><span style="font-family: "Courier New",
monospace;">| Table | Size (GB) |</span></div>
<div><span style="font-family: "Courier New",
monospace;">+----------------------------+-----------+</span></div>
<div><span style="font-family: "Courier New",
monospace;">| o2_step_table | 183.83 |</span></div>
<div><span style="font-family: "Courier New",
monospace;">| o2_job_table | 128.18 |</span></div>
<div><br>
</div>
<div><br>
</div>
That's good advice Paul, much appreciated. <br>
<br>
>took forever and actually caused issues with the archive
process</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);" class="elementToProof">I think that
should be highlighted for other users!<br>
<br>
For those interested, to find the tables sizes I did this:<br>
<br>
</span><span style="font-family: "Courier New",
monospace; font-size: 10pt; color: rgb(0, 0, 0);
background-color: rgb(255, 255, 255);">SELECT table_name AS
"Table", ROUND(((data_length + index_length) / 1024 / 1024 /
1024), 2) AS "Size (GB)" FROM information_schema.TABLES WHERE
table_schema = "slurmdbd" ORDER BY (data_length +
index_length) DESC;</span><span style="font-family: Arial,
Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);
background-color: rgb(255, 255, 255);"><br>
</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);"><br>
</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);">Replace </span><span style="font-family:
"Courier New", monospace; font-size: 10pt; color:
rgb(0, 0, 0); background-color: rgb(255, 255, 255);">slurmdbd</span><span
style="font-family: Arial, Helvetica, sans-serif; font-size:
10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);"> with the name of your database.</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);"><br>
</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<font face="Arial, Helvetica, sans-serif"><span
style="font-size: 13.3333px;">Cheers</span></font></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<font face="Arial, Helvetica, sans-serif"><span
style="font-size: 13.3333px;">--Mick</span></font></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);"><br>
</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255,
255);" class="elementToProof">
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 10pt; color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);"><br>
</span></div>
</blockquote>
</body>
</html>