<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>No, all the archiving does is remove the pointer. What slurm
does right now is that it creates a hash of the job_script/job_env
and then checks and sees if that hash matches one on record. If
not then it adds it to the record, if it does match then it adds a
pointer to the appropriate record. So you can think of the
job_script/job_env as an internal database of all the various
scripts and envs that slurm has ever seen and then what ends up in
the Job record is a pointer to that database. This way slurm can
deduplicate scripts/envs that are the same. This works great for
job_scripts as they are functionally the same and thus you have
many jobs pointed to the same script, but less so for job_envs.</p>
<p>-Paul Edmon-<br>
</p>
<div class="moz-cite-prefix">On 9/28/2023 1:55 PM, Ryan Novosielski
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1EB1BF96-FAC6-4C8E-9E1D-410FBD67C0DE@rutgers.edu">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Thank you; we’ll put in a feature request for improvements in that
area, and also thanks for the warning? I thought of that in
passing, but the real world experience is really useful. I could
easily see wanting that stuff to be retained less often than the
main records, which is what I’d ask for.
<div><br>
</div>
<div>I assume that archiving, in general, would also remove this
stuff, since old jobs themselves will be removed?</div>
<div><br>
<div>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); color:
rgb(0, 0, 0); letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space:
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; overflow-wrap: break-word;
-webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color:
rgb(0, 0, 0); letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space:
normal; word-spacing: 0px; -webkit-text-stroke-width:
0px; text-decoration: none; overflow-wrap: break-word;
-webkit-nbsp-mode: space; line-break:
after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color:
rgb(0, 0, 0); letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;
overflow-wrap: break-word; -webkit-nbsp-mode: space;
line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0);
color: rgb(0, 0, 0); letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration:
none; overflow-wrap: break-word; -webkit-nbsp-mode:
space; line-break: after-white-space;">
<div style="color: rgb(0, 0, 0); letter-spacing:
normal; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
overflow-wrap: break-word; -webkit-nbsp-mode:
space; line-break: after-white-space;">
--<br>
#BlackLivesMatter</div>
<div style="color: rgb(0, 0, 0); letter-spacing:
normal; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
overflow-wrap: break-word; -webkit-nbsp-mode:
space; line-break: after-white-space;">
____<br>
|| \\UTGERS,
|---------------------------*O*---------------------------<br>
||_// the State<span class="Apple-tab-span" style="white-space: pre;"> </span> |
Ryan Novosielski - <a class="moz-txt-link-abbreviated" href="mailto:novosirj@rutgers.edu">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922
(2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ<span class="Apple-tab-span" style="white-space: pre;"> </span> |
Office of Advanced Research Computing - MSB
A555B, Newark<br>
`'</div>
</div>
</div>
</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Sep 28, 2023, at 13:48, Paul Edmon
<a class="moz-txt-link-rfc2396E" href="mailto:pedmon@cfa.harvard.edu"><pedmon@cfa.harvard.edu></a> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div>Slurm should take care of it when you add it.<br>
<br>
So far as horror stories, under previous versions our
database size ballooned to be so massive that it
actually prevented us from upgrading and we had to
drop the columns containing the job_script and
job_env. This was back before slurm started hashing
the scripts so that it would only store one copy of
duplicate scripts. After this point we found that the
job_script database stayed at a fairly reasonable size
as most users use functionally the same script each
time. However the job_env continued to grow like crazy
as there are variables in our environment that change
fairly consistently depending on where the user is.
Thus job_envs ended up being too massive to keep
around and so we had to drop them. Frankly we never
really used them for debugging. The job_scripts though
are super useful and not that much overhead.<br>
<br>
In summary my recommendation is to only store
job_scripts. job_envs add too much storage for little
gain, unless your job_envs are basically the same for
each user in each location.<br>
<br>
Also it should be noted that there is no way to prune
out job_scripts or job_envs right now. So the only way
to get rid of them if they get large is to 0 out the
column in the table. You can ask SchedMD for the mysql
command to do this as we had to do it here to our
job_envs.<br>
<br>
-Paul Edmon-<br>
<br>
On 9/28/2023 1:40 PM, Davide DelVento wrote:<br>
<blockquote type="cite">In my current slurm
installation, (recently upgraded to slurm v23.02.3),
I only have<br>
<br>
AccountingStoreFlags=job_comment<br>
<br>
I now intend to add both<br>
<br>
AccountingStoreFlags=job_script<br>
AccountingStoreFlags=job_env<br>
<br>
leaving the default 4MB value for max_script_size<br>
<br>
Do I need to do anything on the DB myself, or will
slurm take care of the additional tables if needed?<br>
<br>
Any
comments/suggestions/gotcha/pitfalls/horror_stories
to share? I know about the additional diskspace and
potentially load needed, and with our resources and
typical workload I should be okay with that.<br>
<br>
Thanks!<br>
</blockquote>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</body>
</html>